Teaching a 135M model to tell stories with LoRA

The setup

LoRA freezes the pretrained weights and injects small trainable low-rank matrices into the attention and MLP projections, so the model adapts with a fraction of the parameters of full fine-tuning. Hyperparameters weren't defaults: a 2e-4 learning rate trained faster but produced a spiky, unstable loss curve, so 1e-4 was kept; rank 8 underfit the format, so it was doubled to 16.

Base model

HuggingFaceTB/SmolLM-135M (135M params)

Dataset

maxmyn/wholesome_greentext_110k (100-sample subset)

LoRA

r=16, α=32, dropout 0.05 · q/k/v/o + up/down/gate projections

Training

lr 1e-4 · batch size 4 · checkpoints every 100 steps

Ablation

openai-community/gpt2 (117M) · matched lr, rank, α, dropout

Perplexity across checkpoints

SmolLM's perplexity on the target distribution fell from 3.94 to 1.31 across checkpoints. The GPT-2 ablation — same LoRA rank, alpha, dropout, and learning rate — never got below 12.

Before & after, same prompt

The clearest evidence is qualitative. Given a greentext opening, the base model loops; the LoRA-tuned model continues the format with an actual narrative arc.

> Be me
> Go to the grocery store to buy a single carton of milk
> Cashier says "Enjoy your milk"

> You get it
> You don't
> You go back to the grocery store
> You have to get it back
> You need it
> You buy it
> You go back to the grocery store…

> Realize it's expensive
> Buy a bottle of water
> Go back to the grocery store
> Learn the value of moderation

GPT-2 under the same fine-tuning never internalized the format — even trained 2.5× longer it degenerated into token soup ("Don Don Don Don do").

Training dynamics

Loss curves from the notebook: SmolLM descends smoothly from ~3.4 to ~0.7 in 200 steps, while GPT-2 starts higher, stays above 2.0, and is visibly noisier at the same learning rate.

What this shows

1. 1

   LoRA works remarkably well at small scale.

   A 135M model internalized a distinctive text format from only 100 examples, training just low-rank adapter matrices instead of full weights.

2. 2

   Architecture beats parameter count.

   GPT-2 (2019, 117M) and SmolLM (2024, 135M) are nearly the same size, yet under identical LoRA settings GPT-2 plateaued ~7× worse — and training it 2.5× longer didn't close the gap.

3. 3

   Loss curves are a practical tuning instrument.

   The final learning rate (1e-4 over 2e-4) and rank (16 over 8) were both chosen by reading stability and format adherence off the curves — a workflow that transfers to any PEFT task.