Learn Tin · Part 2: Agents · Chapter 23
Record and replay
Save every model answer to a tape and run it again offline.
A program that talks to a model gives a different answer every run, costs money and needs the network. That makes it hard to test. Tin can record a run once and replay it as often as you like, offline, with the same output.
Record once, replay forever
tin run file.tin --record tape.json saves everything the run took from outside: each model
response, each fetch, each line of input and each random number.
--replay tape.json runs the program from that tape instead:
$ tin run calc.tin --record calc.json # talks to Claude, saves its answers
$ tin run calc.tin --replay calc.json # offline, in milliseconds, same output This program's tape was recorded with Claude. The docs build replays it every time this page is built:
tin run calc.tin --replay tests/tapes/agents-05-claude.json/// Add two integers and return the sum.
tool fn add(a: int, b: int): int => a + b
/// Multiply two integers and return the product.
tool fn multiply(a: int, b: int): int => a * b
agent Calculator {
system: "You are a calculator. Use the tools for all arithmetic. Reply with just the number."
tools: [add, multiply]
on toolCall(name, input) => print(" call {name} {input}")
}
fn main() {
let answer: int = Calculator("What is (17 + 25) * 3?")
print("(17 + 25) * 3 = {answer}")
} call add {"a":17,"b":25}
call multiply {"a":42,"b":3}
(17 + 25) * 3 = 126 Random numbers are recorded too, so even this dice program replays exactly:
tin run dice.tin --replay examples/learn/23-replay/dice.tape.jsonfn main() {
let rolls = range(0, 5).map(_ => randomInt(1, 7))
print(rolls)
print("total {rolls.sum()}")
} [6, 6, 5, 1, 4]
total 22 A changed program won't replay
Each recorded model answer is matched to the exact request that produced it. Change a prompt, a tool or a
schema, and the replay stops instead of handing back an answer to a different question. Here only the
3 became a 4:
tin run changed.tin --replay tests/tapes/agents-05-claude.json/// Add two integers and return the sum.
tool fn add(a: int, b: int): int => a + b
/// Multiply two integers and return the product.
tool fn multiply(a: int, b: int): int => a * b
agent Calculator {
system: "You are a calculator. Use the tools for all arithmetic. Reply with just the number."
tools: [add, multiply]
on toolCall(name, input) => print(" call {name} {input}")
}
fn main() {
let answer: int = Calculator("What is (17 + 25) * 4?")
print("(17 + 25) * 4 = {answer}")
} panic: replay: Calculator asked the model something that is not on the tape. The program, its prompts or its tools changed since the tape was recorded — record it again with --record. That's on purpose: a stale answer that looks right is worse than an error. Record again, and check the new tape in.
Model runs as tests
tin test file.tin --replay tape.json runs every test from the tape. That turns a real
conversation into a regression test that runs in CI without a key:
tin test regression.tin --replay tests/tapes/agents-05-claude.json/// Add two integers and return the sum.
tool fn add(a: int, b: int): int => a + b
/// Multiply two integers and return the product.
tool fn multiply(a: int, b: int): int => a * b
agent Calculator {
system: "You are a calculator. Use the tools for all arithmetic. Reply with just the number."
tools: [add, multiply]
on toolCall(name, input) => print(" call {name} {input}")
}
test "the calculator still uses its tools" {
let answer: int = Calculator("What is (17 + 25) * 3?")
assertEq(answer, 126)
assert(Calculator.turns() >= 1)
} ✔ the calculator still uses its tools
1 passed, 0 failed Tin's own examples that talk to Claude are tested this way, and the playground replays the same tapes.
Not recorded: the clock. A program that puts now() in a prompt asks something new
every run, so it won't replay.
Tasks
●○○ Task 1: Repeatable coins
Write a program that flips three coins with random(). Record it, then replay it and check you
get the same flips.
Solution
tin run task-1.solution.tin --replay examples/learn/23-replay/coins.tape.jsonfn flip(): string => if random() < 0.5 { "heads" } else { "tails" }
fn main() {
print([flip(), flip(), flip()].join(" "))
} heads heads tails ●●○ Task 2: Test the historian
Using the tape tests/tapes/agents-05-claude.json, write a test that asks the
Historian from examples/agents/05-claude.tin for three languages and checks that all
of them are from before 1975.
Solution
tin test task-2.solution.tin --replay tests/tapes/agents-05-claude.jsontype Language = {
name: string
/// The year it first appeared
year: int
paradigm: Paradigm
/// One sentence on what it is known for
knownFor: string
}
type Paradigm = | Functional | ObjectOriented | Procedural | Logic | Multi
agent Historian {
system: "You are a historian of programming languages. Be accurate and brief."
}
test "three languages, all from before 1975" {
let languages: Language[] = Historian("Name three influential programming languages from before 1975.")
assertEq(languages.len, 3)
assert(languages.all(l => l.year < 1975))
} ✔ three languages, all from before 1975
1 passed, 0 failed The Historian, its types and its prompt must match the recorded program exactly.
Change any of them and the test stops with "not on the tape".
What's next
Next: Real models. Swap the mock for a real model. Nothing else changes.
In the reference: Record and replay