Markov markov

Word-level Markov chain text generation — train a model from a corpus, then generate completions
Pixelbadger Toolkit — pbtk markov← all topics

Overview

The markov topic provides word-level Markov chain text generation. A Markov chain model learns transition probabilities between words from a source corpus, then uses those probabilities to generate plausible continuations of a given input text.

The trained model is persisted to a .Markov subfolder relative to the current working directory. Train the model once from a corpus, then call complete as many times as needed.

flowchart LR CORPUS["corpus.txt"] --> TRAIN["pbtk markov train"] TRAIN --> MODEL[".Markov/\nmodel files"] MODEL --> COMPLETE["pbtk markov complete"] INPUT["--text\n(seed text)"] --> COMPLETE COMPLETE --> OUT(["stdout: generated text"])
The model directory is always .Markov/ relative to the current working directory. Run both train and complete from the same directory to use the same model.

train

Trains a Markov chain model from a source text file. The model records word-to-word transition probabilities across the entire corpus. The trained model is saved to the .Markov/ subfolder in the current working directory, replacing any previously trained model.

$ pbtk markov train --source corpus.txt

On success, the number of unique transition words in the model is reported to stdout.

OptionRequiredDescription
--sourceYesPath to the source text file to train from

complete

Generates a text completion by extending the provided seed text using the trained Markov chain model. The model must have been trained with markov train first. The seed text is used to find a starting point in the model; generation then follows learned transition probabilities word by word.

$ pbtk markov complete --text "The quick brown"
$ pbtk markov complete --text "Once upon a time" --count 100
flowchart LR TEXT["--text\n(seed)"] --> LOOKUP["Find seed in\nmodel transitions"] LOOKUP --> GEN["Generate next word\nfrom transition table"] GEN -->|"repeat --count times"| GEN GEN --> OUT(["stdout: seed + generated words"])
OptionRequiredDefaultDescription
--textYesSeed text to start the completion from
--countNo50Number of words to generate

Command Reference

Command Required options Optional options Output
markov train --source Success message with unique word count to stdout
markov complete --text --count Generated text to stdout