Skip to content

🧠 Strands MicroGPT

The entire GPT algorithm in pure Python. As a Strands model provider.

Based on @karpathy's atomic GPT gist: "The most atomic way to train and run inference for a GPT in pure, dependency-free Python. This file is the complete algorithm. Everything else is just efficiency."

What is this?

A complete GPT implementation — autograd engine, transformer, tokenizer, Adam optimizer, training, and inference — in pure Python. No PyTorch, no NumPy, no CUDA.

graph LR
    A["🗣️ Strands Agent"] --> B{"MicroGPTModel"}
    B -->|Train| C["📚 Any Text Dataset"]
    B -->|Generate| D["✨ New Text"]
    B -->|Checkpoint| E["💾 Save / Load"]

    style B fill:#e65100,color:#fff
    style A fill:#264653,color:#fff

Get Started in 3 Lines

pip install strands-microgpt

from strands_microgpt import MicroGPT

model, tokenizer, docs = MicroGPT.from_dataset()
model.train_on_docs(docs, tokenizer, num_steps=1000)

for name in model.generate(tokenizer, num_samples=10):
    print(name)

→ Full Quickstart | Installation

Three Ways to Use

from strands_microgpt import MicroGPT

model, tok, docs = MicroGPT.from_dataset()
model.train_on_docs(docs, tok, num_steps=1000)
samples = model.generate(tok, num_samples=10)

from strands import Agent
from strands_microgpt import MicroGPTModel

model = MicroGPTModel(num_steps=1000)
agent = Agent(model=model)
agent("Generate some names")

from strands import Agent
from strands_microgpt import microgpt_train, microgpt_generate

agent = Agent(tools=[microgpt_train, microgpt_generate])
agent("Train a GPT on names, then generate 10")

What's Inside

  • ⚡ Autograd Engine

    Value class with full backpropagation. Build computation graphs, compute gradients automatically.

    → Autograd Guide

  • 🧠 GPT Transformer

    Multi-head attention, MLP, RMSNorm, Adam optimizer. The full algorithm in ~300 lines.

    → Architecture

  • 🔧 Strands Tools

    Train and generate as tool calls from any Strands agent (Bedrock, OpenAI, etc.)

    → Tool Usage

  • 📚 Custom Datasets

    Train on anything — names, poems, code, molecules, DNA sequences.

    → Custom Datasets

Installation →

Quickstart →

Autograd Engine →

Training →

Tool Usage →

Architecture →

API Reference →

Examples →

Resources