Skip to content
← The Workbook

No. 62 · 2026Concept

Embryo

A toy language model that starts as a blank slate and grows only from the text I feed it. Random weights at birth, no pretraining, no foundation model, no internet. Vocabulary literally cannot contain a character it has never seen. Babbles noise at first, forms word-shapes within a paragraph, eventually starts echoing my voice.

Status
Concept
Year
2026
Stack
AI · Transformer · Python · Toy · Pedagogy

The Idea

Every commercial LLM is born already knowing everything. Embryo is the opposite. A character-level transformer that starts with random weights and zero vocabulary, and learns from nothing but the text you hand it. The first response is noise. After a paragraph it's word-shapes. After a few thousand words of your own writing, it starts to sound a little like you.

The point isn't capability. It's the inversion. Watching the model build a vocabulary one character at a time, and a style one paragraph at a time, makes "what is a language model" feel concrete instead of magical.

The Architecture

About as small as a transformer gets:

  • 4 layers, 4 heads, 96-dim embeddings, 64-char context window.
  • ~500K parameters (GPT-4 is ~1T+, so this is roughly two-millionths the size).
  • Designed to train on Apple Silicon MPS or CPU in seconds per feed.
  • Single Python file for the model, single file for the REPL.

The REPL

The whole interface is a command line. A few commands:

  • feed <text>, adds text to the corpus and trains briefly.
  • feedfile <path>, absorbs an entire file in one shot.
  • gen [N], generates N characters from the current model.
  • gen "<seed>", generates starting from a seed string.
  • status, shows corpus size, vocab size, parameter count, training stats.
  • save / load / reset, persistence and start-over.

Three files on disk hold everything the embryo has ever experienced: a plain-text corpus, a vocab JSON, and the model weights. Personal artifacts, gitignored.

What It's For

Mostly a teaching tool for understanding what a model is. A few side uses:

  • Style mirror. Feed it your own writing, watch it gradually find your cadence.
  • Constrained generator. When you want a model that knows nothing it wasn't told, this is the strictest possible version.
  • Pairs with DJ ADHDJ. Same instinct applied to audio: a generative model raised on nothing but my own drum & bass vinyl.

Status

A thought, still. The scaffold exists on disk, but the experiment has not honestly been run, so it sits in the notebook until it earns Built the way everything here has to. The interesting question waiting for it: not "can it get smart" (it cannot) but "how does the texture of its babble change with the texture of the corpus."