IIidle intelligence

llm-life compare

Without going too far into optimizing the cpu/memory paths, we compare (naive) cpu, (naive) 512-entry lookup table, with a language model, two tiny from-scratch models and a stencil-attention model, all computing the same generation from the same starting board.
When Moore's law no longer holds up, we need to find creative ways to spend more compute to achieve the same results.
This started as a (stupid) experiment: What if we ask an LLM to play Game of Life, cell by cell?
In the end, we answer some interesting questions.

llm-life demo · github.com/idle-intelligence/llm-life

settings · run

board

status

booting...

output · ladder

method s / generation parameters cells correct

load the LLM above to measure the LLM rows, or "run every method" will load it.

output · methods

Game of Life (rule): The classical rule, as a for loop.

lookup table: the same rule as a 512-entry table indexed by the 9 neighbourhood bits. The fastest way to compute it, and the thing every model below is measured against.

LLM per cell: Give a language model the rules and current cell's state, *as text*, and ask if it lives or dies. Repeat for each and every cell. Of course, it's very slow, and an off the shelf model doesn't work. Fine tuning a small LoRa adapter gets us perfect results, but it's still slow... Does it speed up if we batch the inference?

LLM whole grid: Look, there are many optimisation we could do. That's not the game we are playing. Yet, cell by cell _is_ stupid. Instead, we ask the model for the whole grid in one prompt, one token per cell. It's much faster, but training a 64x64 adapter gets expensive, and 256x256 doesn't fit in this small model's context. Another LoRa adapter gets us to 100% accuracy.

BERT of Life: Alive or Dead is a classification. What would a BERT do? A BERT-shaped classifier, one-layer encoder over the 9 cells as tokens, 3,490 parameters trained from scratch; it works.

9 numbers to centre: Why do we even use a language model? Can we fit a "modern" model architecture that actually learns the rules? A 2-layer MLP over the 9 cell values, 1,442 parameters; We run it cell by cell, pretty fast, it works.

stencil (grid to grid): Cell by cell is nice, but we can do better: One attention layer over the whole grid, each cell masked to its 3x3 stencil. 3,329 parameters, trained at 16x16, it scales to whatever grid size.