GPT gpt

Overview

The gpt topic implements a tiny GPT trained from scratch — a decoder-only transformer with token and positional embeddings, multi-head self-attention, and feed-forward blocks. Train it on any text corpus with train, then generate continuations of a prompt with complete. Unlike the llm topic, no API key or network access is required — everything runs locally on the CPU.

Training produces a checkpoint directory (default ./.gpt) containing config.json (model configuration and tokenizer state), weights.bin (the weight tensors), and optimizer.bin (AdamW optimizer state). Train once, then call complete as many times as needed against the same checkpoint — or continue training it later with --resume.

Two tokenizers are available via --tokenizer. The default bpe is a byte-level byte-pair encoding (à la GPT-2): it learns subword tokens by repeatedly merging the most frequent adjacent byte pair up to --vocab-size tokens, so a fixed --block-size covers several times more text and the model spends capacity on structure rather than spelling. The alternative char is the original one-token-per-character scheme. Because BPE works on raw UTF-8 bytes, any input is representable — there are no out-of-vocabulary characters.

Architecture

The gpt topic follows the toolkit's component-per-command pattern: a thin System.CommandLine layer (GptCommand) parses options and delegates to one component per action. The components orchestrate a small set of services that together form a self-contained training and inference platform — there are no external ML dependencies (no TorchSharp, no ONNX, no native libraries). The entire stack, from autograd to optimizer, is implemented in C# in the spirit of Karpathy's micrograd/nanoGPT.

Model

GptModel implements a nanoGPT-style decoder-only transformer. A GptConfig record (VocabSize, BlockSize, NEmbd, NHead, NLayer) fixes the architecture; the head dimension is NEmbd / NHead, and construction validates that --n-embd is divisible by --n-head.

Key architectural details:

With the default hyperparameters (n-embd 128, 4 heads, 3 layers, block size 64) and a typical corpus vocabulary, the model comes out at roughly 600K parameters.

Autograd Engine

All numerics run through Tensor, a minimal reverse-mode automatic differentiation engine over 2D float matrices. Each operation (e.g. MatMul, LayerNorm, SoftmaxRows, Gelu, CrossEntropy) computes its result eagerly, records its parent tensors, and attaches a closure that knows how to propagate gradients backwards. Calling Backward() on the loss performs a topological sort of the computation graph, seeds the root gradient with 1, and invokes each node's backward function in reverse order, accumulating into every parameter's Grad buffer.

The op set is exactly what the transformer needs:

Performance and determinism are balanced deliberately: hot kernels use System.Numerics.Tensors.TensorPrimitives (SIMD) for dot products and elementwise math, and forward and backward passes parallelize with Parallel.For across independent work units — matrix rows, (sequence, head) attention units, normalization rows — falling back to sequential loops when a tensor is too small to justify thread-pool overhead. Parallelism is only applied where each unit writes disjoint state, and reductions inside a unit stay sequential, so results are bit-identical regardless of thread count — which is what makes --seed fully reproducible.

Training Pipeline

GptTrainComponent.TrainAsync wires the services together into a standard minibatch gradient-descent loop:

1. Tokenize — Tokenizers.Build builds the requested tokenizer (--tokenizer bpe learns byte-pair merges up to --vocab-size; char takes the sorted distinct characters), then encodes the whole corpus into one integer array. The corpus must encode to at least block-size + 1 tokens.
2. Construct — a GptModel is built from the config and initialized from the seeded RNG; an AdamW instance is bound to the model's parameter list (learning rate from --lr; betas 0.9/0.95, weight decay 0.1). With --resume, the model, tokenizer, and config are instead loaded from the checkpoint in --out (architecture and tokenizer options are ignored), and saved AdamW moments are restored so training continues without a loss spike.
3. Sample batches — each step draws --batch-size random windows of --block-size tokens; the target sequence is the same window shifted one token right, so every position learns next-token prediction simultaneously. Each window's start offset is returned with the batch so the component can mark which corpus positions and which vocabulary entries have been visited — random sampling means coverage is probabilistic, and a run with too few steps will never see parts of the corpus.
4. Step — ZeroGrad → Forward (logits + loss) → loss.Backward() → optimizer.Step(). The command layer prints step/loss at step 1, every 100th step, and the final step, while the component records every step's loss in full.
5. Persist — after the final step the checkpoint (config, tokenizer state, and all weight tensors in construction order) plus the optimizer state is written via CheckpointService, the loss and coverage series (with the packed visited sets and run boundaries) are written to history.bin so a later --resume can continue them, and a summary (parameter count, vocab size, final loss, corpus token count and chars/token, path, plus corpus and vocabulary coverage) is reported. When --loss-graph is set, LossGraphService then writes the whole per-step loss and coverage series out as a standalone HTML report.

Inference Pipeline

GptCompleteComponent.CompleteAsync performs autoregressive decoding — the same forward pass as training, but without targets, and repeated once per generated character:

Because the context window is capped at block-size tokens, generation slides the window forward once the sequence outgrows it — earlier tokens simply fall out of context. Each iteration is a full forward pass over the current window (there is no KV caching in this implementation). Sampling divides logits by --temperature before a max-subtracted softmax; --temperature 0 switches to pure argmax, and any sampled run is reproducible via --seed. --max-tokens counts tokens, not characters, so a BPE checkpoint emits more text per token than a char checkpoint. With the default bpe tokenizer any prompt is valid (it works on raw bytes); a char checkpoint still requires prompt characters that were present in the training corpus, since its vocabulary is fixed at training time.

Checkpoint Format

A checkpoint is a directory (default ./.gpt) written by CheckpointService containing four files:

On load, GptModel.LoadWeights validates that the tensor count and every tensor length match what the config-constructed model expects, so a corrupted or mismatched checkpoint fails fast with a clear error rather than producing garbage output. Because the config fully determines the parameter shapes and ordering, the checkpoint is portable across machines — inference only needs config.json and weights.bin.

train

Trains a tiny GPT on a text corpus and saves a checkpoint. The tokenizer is selected with --tokenizer: the default bpe learns byte-pair merges up to --vocab-size tokens, while char uses one token per distinct character. Training progress (step and loss) is reported periodically, and on completion the parameter count, vocabulary size, final loss, corpus token count (with chars/token), and checkpoint path are printed, along with how much of the corpus and vocabulary the randomly sampled batches actually visited. --source accepts either a path to a corpus file or literal text.

With --resume, training continues from the checkpoint already in --out for a further --steps steps: weights, tokenizer, and architecture come from the checkpoint (so --block-size, --n-embd, --n-head, --n-layer, --tokenizer, and --vocab-size are ignored), and saved optimizer state is restored when present. For a char checkpoint the corpus may only contain characters that were in the original training corpus; a bpe checkpoint accepts any text.

--loss-graph writes a standalone HTML report of the run to the given path (parent directories are created as needed). Every step's loss is recorded — not just the ones printed to the console — and embedded in the page, which draws the raw series plus a smoothed trend line on a canvas with hover readouts. A second chart tracks corpus coverage: because batches are drawn from random offsets, the report shows the cumulative share of the corpus's token positions that have been sampled at least once, so it is easy to see whether a run visits most of the corpus or keeps re-treading the same windows. The run's configuration, corpus coverage, and vocabulary coverage are tabulated below the charts. The file is self-contained (no external scripts or styles), so it can be opened directly or served with web serve-html. Because the series and visited sets are stored in the checkpoint's history.bin, a --resume run graphs the whole history — every run against that checkpoint, not just the latest — with a dashed divider at each resume point and coverage carried on from the positions already visited. If the resumed corpus has a different token count the recorded positions no longer apply, so coverage restarts from that run while the loss curve is still kept.

complete

Generates text from a trained GPT checkpoint by continuing the given prompt one character at a time. The checkpoint must have been produced with gpt train first. Sampling is controlled by --temperature — higher values are more random, and 0 is greedy/deterministic — and --seed makes sampled runs reproducible.

Command Reference