Sn Tin

Learn Tin · Part 2: Agents · Chapter 23

Record and replay

Save every model answer to a tape and run it again offline.

A program that talks to a model gives a different answer every run, costs money and needs the network. That makes it hard to test. Tin can record a run once and replay it as often as you like, offline, with the same output.

Record once, replay forever

tin run file.tin --record tape.json saves everything the run took from outside: each model response, each fetch, each line of input and each random number. --replay tape.json runs the program from that tape instead:

$ tin run calc.tin --record calc.json      # talks to Claude, saves its answers
$ tin run calc.tin --replay calc.json      # offline, in milliseconds, same output

This program's tape was recorded with Claude. The docs build replays it every time this page is built:

calc.tintin run calc.tin --replay tests/tapes/agents-05-claude.json
/// Add two integers and return the sum.
tool fn add(a: int, b: int): int => a + b

/// Multiply two integers and return the product.
tool fn multiply(a: int, b: int): int => a * b

agent Calculator {
  system: "You are a calculator. Use the tools for all arithmetic. Reply with just the number."
  tools: [add, multiply]
  on toolCall(name, input) => print("  call {name} {input}")
}

fn main() {
  let answer: int = Calculator("What is (17 + 25) * 3?")
  print("(17 + 25) * 3 = {answer}")
}
  call add {"a":17,"b":25}
  call multiply {"a":42,"b":3}
(17 + 25) * 3 = 126

Random numbers are recorded too, so even this dice program replays exactly:

dice.tintin run dice.tin --replay examples/learn/23-replay/dice.tape.json
fn main() {
  let rolls = range(0, 5).map(_ => randomInt(1, 7))
  print(rolls)
  print("total {rolls.sum()}")
}
[6, 6, 5, 1, 4]
total 22

A changed program won't replay

Each recorded model answer is matched to the exact request that produced it. Change a prompt, a tool or a schema, and the replay stops instead of handing back an answer to a different question. Here only the 3 became a 4:

changed.tintin run changed.tin --replay tests/tapes/agents-05-claude.json
/// Add two integers and return the sum.
tool fn add(a: int, b: int): int => a + b

/// Multiply two integers and return the product.
tool fn multiply(a: int, b: int): int => a * b

agent Calculator {
  system: "You are a calculator. Use the tools for all arithmetic. Reply with just the number."
  tools: [add, multiply]
  on toolCall(name, input) => print("  call {name} {input}")
}

fn main() {
  let answer: int = Calculator("What is (17 + 25) * 4?")
  print("(17 + 25) * 4 = {answer}")
}
panic: replay: Calculator asked the model something that is not on the tape. The program, its prompts or its tools changed since the tape was recorded — record it again with --record.

That's on purpose: a stale answer that looks right is worse than an error. Record again, and check the new tape in.

Model runs as tests

tin test file.tin --replay tape.json runs every test from the tape. That turns a real conversation into a regression test that runs in CI without a key:

regression.tintin test regression.tin --replay tests/tapes/agents-05-claude.json
/// Add two integers and return the sum.
tool fn add(a: int, b: int): int => a + b

/// Multiply two integers and return the product.
tool fn multiply(a: int, b: int): int => a * b

agent Calculator {
  system: "You are a calculator. Use the tools for all arithmetic. Reply with just the number."
  tools: [add, multiply]
  on toolCall(name, input) => print("  call {name} {input}")
}

test "the calculator still uses its tools" {
  let answer: int = Calculator("What is (17 + 25) * 3?")
  assertEq(answer, 126)
  assert(Calculator.turns() >= 1)
}
✔ the calculator still uses its tools

1 passed, 0 failed

Tin's own examples that talk to Claude are tested this way, and the playground replays the same tapes.

Not recorded: the clock. A program that puts now() in a prompt asks something new every run, so it won't replay.

Tasks

●○○ Task 1: Repeatable coins

Write a program that flips three coins with random(). Record it, then replay it and check you get the same flips.

Solution
task-1.solution.tintin run task-1.solution.tin --replay examples/learn/23-replay/coins.tape.json
fn flip(): string => if random() < 0.5 { "heads" } else { "tails" }

fn main() {
  print([flip(), flip(), flip()].join(" "))
}
heads heads tails

●●○ Task 2: Test the historian

Using the tape tests/tapes/agents-05-claude.json, write a test that asks the Historian from examples/agents/05-claude.tin for three languages and checks that all of them are from before 1975.

Solution
task-2.solution.tintin test task-2.solution.tin --replay tests/tapes/agents-05-claude.json
type Language = {
  name: string
  /// The year it first appeared
  year: int
  paradigm: Paradigm
  /// One sentence on what it is known for
  knownFor: string
}

type Paradigm = | Functional | ObjectOriented | Procedural | Logic | Multi

agent Historian {
  system: "You are a historian of programming languages. Be accurate and brief."
}

test "three languages, all from before 1975" {
  let languages: Language[] = Historian("Name three influential programming languages from before 1975.")
  assertEq(languages.len, 3)
  assert(languages.all(l => l.year < 1975))
}
✔ three languages, all from before 1975

1 passed, 0 failed

The Historian, its types and its prompt must match the recorded program exactly. Change any of them and the test stops with "not on the tape".

What's next

Next: Real models. Swap the mock for a real model. Nothing else changes.

In the reference: Record and replay