Giles Thomas
@gilesthomas.com
Building LLMs, learning in public / Founded and led @PythonAnywhere.com / PSF Fellow / Writing at gilesthomas.com
I thought I'd put together a quick post on how I use AI when writing my blog: AIs identify problems and then I fix them myself. www.gilesthomas.com/2026/07/ai-use
How I use AI on this blog
A point-in-time snapshot of how I'm currently using AI to help with my experiments, and with blogging about them
gilesthomas.com
The first in what will probably be an occasional series on my blog: Why do OpenAI's GPT-2 weights beat mine? www.gilesthomas.com/2026/07/why-...
Why do OpenAI's GPT-2 weights beat mine?
The original GPT-2 small weights are much better than my own models at a specific instruction-following task. Time to look into why!
gilesthomas.com
A friend asked me how many tps I could get with Qwen 3.6 35B MoE on my RTX 3090. I may have overthought the question, but maybe the results are useful for other people? www.gilesthomas.com/2026/07/benc...
Benchmarking Qwen 3.6 35B MoE (3B active) on an RTX 3090
A friend asked me how many tokens per second I could get on Qwen 3.6 35B MoE on an RTX 3090. I probably dug into it more than he expected.
gilesthomas.com
With no weight tying, the token embeddings and the output head alone make up almost half of a GPT-2-small sized model. That makes sense, but it's kind of unintuitive! I asked GPT 5.6 Sol to create a visualiser: www.gilesthomas.com/post-assets/...
GPT-2 Parameter Counter
Explore how embedding width and layer count shape a GPT-2 model's parameter count.
gilesthomas.com
I'm repurposing an old PC as a dedicated LLM training box, and blogging the process. Here's part 1, featuring an almost-melted CPU and an accidental 11-day LLM training run: www.gilesthomas.com/2026/07/popp...
poppy the training box, part 1: the beginnings
Repurposing an old SFF PC as a dedicated LLM training box: a second-hand RTX 3090, an accidental 11-day train on a GTX 1660, and one dead CPU fan.
gilesthomas.com
And that's it! After 18 months, the capstone in my LLM from scratch journey. To test my knowledge, I built GPT-2 small in JAX, using just my notes. It worked really well, and I found an interesting way to put it together bit-by-bit, watching the loss go down. www.gilesthomas.com/2026/07/llm-...
Writing an LLM from scratch, part 34b -- from bigrams to GPT-2, one component at a time (in JAX)
The capstone of my LLM from scratch journey: building up from a bigram-style model to GPT-2 small in JAX, watching the loss fall as each component goes in.
gilesthomas.com
Coming to the end of my LLM from scratch journey... I'm building a GPT-2-small in JAX, and decided to build the training loop first, then build the model step-by-step. Here's how I build the training loop. www.gilesthomas.com/2026/06/llm-...
Writing an LLM from scratch, part 34a -- building a JAX training loop for an LLM training run
I'm going to build an LLM from scratch in JAX, but I wanted to get a training loop working first so that I could put it together piece by piece.
gilesthomas.com
A bit of a debugging journey with JAX, Flax and NNX: www.gilesthomas.com/2026/06/hash...
Flax debugging: making a hash of things
A quick way to help with debugging plumbing issues in JAX training loops.
gilesthomas.com
Switching from a power-hungry Marvell-based SFP+ 10GBASE-T module to a cooler Broadcom-based one: some oddities, including a mendacious EEPROM. www.gilesthomas.com/2026/06/10g-...
10Gb/s Ethernet: switching to a Broadcom SFP+ module
As predicted by several people, one of my Marvell-based SFP+ modules had overheating problems. I've switched it over to a Broadcom-based one, and it looks better, though I can't be certain.
gilesthomas.com
Hit some interesting memory management weirdness in JAX: arrays can move around between devices when you don't expect it, if you don't commit them to one explicitly: www.gilesthomas.com/2026/06/jax-...
JAX: commitment issues
JAX memory management is tricky; data can be moved around unexpectedly if you don't make sure it's committed to a device.
gilesthomas.com
More notes on working with JAX: backends and devices. www.gilesthomas.com/2026/06/jax-...
JAX backends and devices
Some basic notes on how to get JAX to load data to a particular device.
gilesthomas.com
Getting JAX/Flax and Safetensors to play nicely was a bit fiddly -- wrote it up here: www.gilesthomas.com/2026/06/flax...
Using Safetensors with Flax
Using Safetensors to save Flax models isn't all that hard, so long as you know the trick.
gilesthomas.com
I've spent some time learning JAX over the last month, and I have Thoughts: www.gilesthomas.com/2026/05/on-f...
On first looking into JAX
Some (well, quite a lot of) initial thoughts about JAX, having spent a short time playing with it.
gilesthomas.com
...and here's part 2: what I actually did. Turns out you could use an SFP+ 10GBASE-T module to make a (very small) cup of tea: www.gilesthomas.com/2026/04/10g-...
10Gb/s Ethernet: what I actually did to get it working in my home
I already had 2.5Gb/s working. Here's how I upgraded the whole house to 10Gb/s Ethernet -- including real iperf3 numbers, MikroTik switches, scary thermals, and what actually worked.
gilesthomas.com
WiFi has made home wired networking boring for the last 20 years or so, I think, but 10Gb/s "fixes" that :-) I've had a bit of fun over the last two weeks doing an upgrade, here's part 1 of a two-part writeup: www.gilesthomas.com/2026/04/10g-...
10Gb Ethernet: what I had to (re)learn
Wired networking in the home and small offices has been pretty stagnant for ages, so upgrading my home from 2.5Gb ethernet to 10Gb meant I had to (re)learn a few things.
gilesthomas.com
The appendices in "Build an LLM (from Scratch)" have a lot of really useful stuff. Should I have read them before heading off on my own training runs? I think no -- it would have saved time, but by trying, failing, and trying again, I learned more. YMMV! www.gilesthomas.com/2026/04/llm-...
Writing an LLM from scratch, part 33 -- what I learned from finally getting round to the appendices
I've finished the main body of Raschka's book -- here are my thoughts on the appendices, and why I'm glad I read them after learning things the hard way.
gilesthomas.com
Some interesting results with instruction fine-tuning. In general you'd expect that a model with lower loss on your test set would fine-tune to do better instruction-following. But the correlation doesn't seem as close as I would have thought: www.gilesthomas.com/2026/04/llm-...
Writing an LLM from scratch, part 32l -- Interventions: updated instruction fine-tuning results
I wanted to revisit the instruction fine-tuning tests that I'd put on hold, and try them with my new models. I found that loss predicts real-world usefulness much less than I would have thought!
gilesthomas.com
I had 57 checkpoints from my last local LLM training run on my disk, and thought it would be interesting to use them to show how the model got more coherent over time. By 1/3 of the way through, it was surprisingly solid! www.gilesthomas.com/2026/04/how-...
How an LLM becomes more coherent as we train it
As an LLM is trained, it gradually learns how to generate increasingly coherent text. Here's an example.
gilesthomas.com
Training a new GPT-2-small-style base model on my RTX 3090 in 40 hours: gradient accumulation, along with some of the other interventions I've experimented with, got me almost (but not quite) all the way to the quality of the original OpenAI model! www.gilesthomas.com/2026/04/llm-...
Writing an LLM from scratch, part 32k -- Interventions: training a better model locally with gradient accumulation
Having worked out which combination of interventions into my model and training run improved loss the most, it was time to see how well that worked with a local run, which meant I had to learn about g...
gilesthomas.com
After spending two months trying out different interventions on my GPT-2-style model, it was time to try stacking them up. Interesting results! www.gilesthomas.com/2026/04/llm-...
Writing an LLM from scratch, part 32j -- Interventions: trying to train a better model in the cloud
Now that I've tried a number of interventions into my model and training run, and some of them seem to improve the model, how do we stack them together, and what are the results?
gilesthomas.com
I wanted to see whether my results when testing interventions to my GPT-2-style training loop were signal or noise. The results were promising! www.gilesthomas.com/2026/04/llm-...
Writing an LLM from scratch, part 32i -- Interventions: what is in the noise?
How much of the variation in my training runs is real signal, and how much is random noise? I trained seven more models to find out.
gilesthomas.com
My final intervention test, in which I discover that there is such a thing as a free lunch, and it's called AMP. www.gilesthomas.com/2026/04/llm-...
Writing an LLM from scratch, part 32h -- Interventions: full fat float32
I've been using PyTorch's Automated Mixed Precision (AMP) and lower-precision matrix multiplication logic for my training runs so far for larger batch sizes and faster training. Does doing that lead ...
gilesthomas.com
Weight tying, by contrast with weight decay, was actually really easy! It didn't help, though :-( www.gilesthomas.com/2026/03/llm-...
Writing an LLM from scratch, part 32g -- Interventions: weight tying
Weight tying is apparently not used in modern LLMs, and intuitively would worsen performance in general. Does it?
gilesthomas.com
Weight decay is conceptually simpler than I worried it might be, but still pretty fiddly to get right... www.gilesthomas.com/2026/03/llm-...
Writing an LLM from scratch, part 32f -- Interventions: weight decay
What is weight decay, and what is the right value to set it to in order to get the best possible training run for our model?
gilesthomas.com
Learning rates for LLMs turned out to be a deep topic, but making some tweaks certainly seems to help my base model train: www.gilesthomas.com/2026/03/llm-...
Writing an LLM from scratch, part 32e -- Interventions: the learning rate
The learning rate is an essential hyperparameter for training models with gradient descent, but what actual values should you use, and what does it mean to schedule it?
gilesthomas.com
Now this was a surprise. QKV bias is not meant to be useful -- but with my GPT-2 small model, it looks like it is! www.gilesthomas.com/2026/02/llm-...
Writing an LLM from scratch, part 32d -- Interventions: adding attention bias
Having bias terms for the query, key, and value attention weights is apparently no longer used because it doesn't help. Let's check that it really doesn't for our model!
gilesthomas.com
Does removing dropout improve our baseline model's test loss? Yes, absolutely, and much more than gradient clipping did. www.gilesthomas.com/2026/02/llm-...
Writing an LLM from scratch, part 32c -- Interventions: removing dropout
Does removing dropout improve our baseline model's test loss? Yes, absolutely, and even more than gradient clipping did.
gilesthomas.com
First "intervention" test: does adding gradient clipping improve our baseline model by lessening the loss spikes during training? It does, but it turned out to be more of a rabbit hole than I expected. www.gilesthomas.com/2026/02/llm-...
Writing an LLM from scratch, part 32b -- Interventions: gradient clipping
Does adding gradient clipping improve our baseline model by lessening the loss spikes during training? It does, but it turned out to be more of a rabbit hole than I expected.
gilesthomas.com
Back to my LLM from scratch series. I want to train the *best* GPT-2-style model that I can locally in two days, and there are various levers to pull. Working out which ones work means I need a baseline for comparison. www.gilesthomas.com/2026/02/llm-...
Writing an LLM from scratch, part 32a -- Interventions: training a baseline model
I want to try a bunch of interventions like gradient clipping and removing dropout to see if my models get better. I need a baseline train without them so that I can be sure of the results.
gilesthomas.com