What a railroad gametaught a model about finance

So when we wanted to improve a model's financial capabilities, we reached for a game with a market inside it: 1830, a railroad game1830A 1986 board game designed by Francis Tresham. Players float railroad companies, trade their shares, and run trains, and the share market is unforgiving enough that a bad round can end a player's game.× from the era of the robber barons. A full game runs three to six hours at a real table where players float companies, trade shares, pay dividends, and occasionally bankrupt each other.

We taught a 30B30B / PARAMETERSParameters are the learned numbers inside a model. Thirty billion of them is small next to a frontier system, and small enough to train and inspect on our own hardware.× model, NemotronNEMOTRONNVIDIA's family of openly released language models. The one we trained sits in the 30 billion parameter class.×, to play more intelligently with reinforcement learningREINFORCEMENT LEARNINGTraining a model by letting it act, scoring the outcome, and nudging it toward the choices that scored well. The model learns from consequences rather than from worked examples.×. Then we tested it on a subset of the Finance-Agent benchmarkFINANCE-AGENT BENCHMARKAn agentic benchmark from vals.ai built out of real research questions that analysts answer by digging through company filings. The model has to go find the evidence, not recall it.×, full of real SEC-filingsSEC FILINGS / 10-KReports public companies must file with the Securities and Exchange Commission. The 10-K is the annual one, and all of them are public on the SEC's EDGAR database.× research questions, and it beat its own base model. Surprisingly, it wasn't because it learned more about finance, it got better because it stopped confidently asserting things that weren't true.

We initially trained on 1830 as single-turn question answering: here's a game state, what's the right move? The model got dramatically better at the game, going from unable to play to genuinely competent, including in situations it had never seen. On the finance benchmark, it transferred nothing.

So we rebuilt the training as the thing itself: a multi-turn terminal agent that uses tools to explore its environment, plan a strategy, and adapt in real time. That version transferred.

Evidently, how you design the environment changes what the model learns from it. Ask for answers and you train answering. Ask for work and you train working.

To find out what traveled, we did what's sometimes called behavioral interpretabilityBEHAVIORAL INTERPRETABILITYWorking out what a model learned by studying what it does, run by run, rather than by reading its internal weights.×: carefully examining how the model acts by walking through and visualizing its trajectoriesTRAJECTORYThe full record of one run: what the model saw, what it decided, which tools it called, and how each step turned out.×.

Across training, the rate at which the model confidently asserted false claims fell steadily. The untrained base model, pressed for an answer it didn't have, would invent one. Sometimes citing dozens of plausible-looking URLs to articles that don't exist. The trained model, pressed the same way, went and looked, or held its tongue.

After the initial success on our small test we were in the midst of scaling the experiment up when we got early access to Goodfire's SilicoSILICOGoodfire's autonomous research agent for interpretability. It designs experiments, runs them on its own infrastructure, and reports what it found.×, an autonomous AI research agent optimized for interpretability. So we handed it two early checkpointsCHECKPOINTA saved copy of the model's weights at one point during training. Comparing checkpoints shows what changed and when.×, our thesis, and asked, "what did this training actually change?"

It designed its own test for citation-groundingCITATION-GROUNDINGWhether a claim is actually supported by the source attached to it, and whether that source exists at all.× in a style the training had never seen, generated the data, audited its own scorer's blind spots, and came back cautiously optimistic. On a small sample, it found the same fingerprint: the trained model cites more precisely and asserts less recklessly. The details didn't perfectly align with ours but the direction did. You can read its full report here.

The whole engagement cost about $2,000–3,000 in credits and 229 GPU hoursGPU HOURSOne hour of one graphics processor running. It is the standard unit for pricing and comparing the compute an experiment consumed.×. For comparison, our usual route to a second opinion is to spend more time ourselves or commission interpretability work from outside researchers, sometimes taking weeks. This one ran while we slept.

Games are dense with exactly the skills we must instill into models to trust them: grounded citations, calibrated confidenceCALIBRATED CONFIDENCESounding as certain as the evidence warrants: firm when the answer is verified, and openly unsure when it is not.×, planning, and strategic thinking. But a game only trains what its design demands.

As we continue to design environments, and turn games into arenas for knowledge, reliable tools for understanding how models learn help us be more intentional with what we teach. We're looking forward to having Silico as a tool in our arsenal, and if you are interested in joining us on this journey, please reach out!