Where an Agent’s Intelligence Lives
OpenAI tripled one benchmark score with two harness settings. In our game agents, the strongest results came when the model and its skill system learned together.
Research by Good Start Labs with collaborators from the University of Maryland, the University of Southern California, and Mohamed bin Zayed University of Artificial Intelligence. Accepted as a poster at Agent Skills ’26. The listed research affiliations also include an independent researcher, who remains credited in the linked paper.
The same model can look three times smarter without changing a single WEIGHTSThe learned numerical values inside a model. Changing the harness can change behavior without changing any of these values..
OpenAI recently tripled GPT-5.6 Sol’s score on ARC-AGI-3A benchmark of unfamiliar 2D games in which agents have to infer how each world works through interaction. by turning on two API settings. One RETAINED REASONINGA Responses API setting that carries the model’s reasoning context from one action to the next instead of discarding it each turn. between actions. The other COMPACTIONA way to condense older context when a run gets long, preserving the important information instead of simply dropping the oldest actions. instead of dropping it. The benchmark stayed the same. The weights stayed the same. What changed was how the agent carried its work from one move to the next.
A harness is how a model meets a goal
A harness is the layer that turns a model into an agent. It determines what the model can see, what it can remember, which skills and tools it can use, what actions it can take, and what feedback comes back afterward.
The goal sets the harness. Time, context, compute, permitted actions, and feedback determine how much an agent can try—and when it should persist, revise its plan, or stop.
Capability is a property of the whole agent.
Reasoning starts over; old context drops.
One plan continues; history becomes a summary.
In our COS-PLAY research, games let us test that idea one move at a time.
The player and the playbook improve together
One agent plays the game; another studies the replay. The decision agent reads the current state, chooses an intention, retrieves a relevant skill, and acts.
After the episode, the SKILL BANKAn external library of reusable behavior protocols the agent can retrieve instead of solving the same subproblem from scratch. turns useful stretches of behavior into reusable skills with a compact SKILL CONTRACTA compact specification of when a skill applies, what steps it follows, what effect it should have, and when it should succeed or stop.: when the skill applies, what to do, what success looks like, and when to stop. It refines useful skills, merges duplicates, splits overly broad ones, and retires what no longer helps.
The revised bank guides the next game, which produces new TRAJECTORYThe sequence of observations, decisions, actions, and outcomes produced during one run through an environment.. The player changes the playbook; the playbook changes what the player can do.
How COS-PLAY closes the loop
GAME
Start with the game: a changing state, a set of legal actions, and feedback after every move.
Under the hood
The main loop is simple to describe. The implementation separates its jobs so each part can learn without overwriting the others.
- BASE MODEL
- Qwen3-8B
- COLD START
- 60 GPT-5.4 seed trajectories per game
- SFTSupervised fine-tuning. The model learns from example trajectories before it begins the co-evolution loop.
- Shared initialization for both agents
- GRPOA reinforcement-learning method that compares groups of sampled behaviors and updates the model toward the better-performing ones.
- Updates the decision agent and skill-bank agent
- 5 LoRAA small trainable adapter that changes a model’s behavior without updating every weight in the base model.
- Two for decision making, three for skill-bank work
- UP TO 25 ITERATIONS
- Per game in the reported experiments
Better play becomes better intelligence
Across four single-player games, COS-PLAY’s EIGHT-BILLION-PARAMETER MODELQwen3-8B contains roughly eight billion learned parameters. Frontier baselines do not publish directly comparable parameter counts. averaged a 25.1 percent within-game reward gain over GPT-5.4. The study also reports Gemini-3.1-Pro, Claude-4.6-Sonnet, and GPT-OSS-120B as frontier baselines. This compares complete agent systems, not model size or parameter efficiency. See the reported results.
Performance, game by game
Plan beyond the next merge.
Single-player games use different reward scales. Compare models within a game, not raw scores across games.
| Game | Metric | GPT-5.4 | GPT-5.4 CI | Gemini-3.1-Pro | Gemini-3.1-Pro CI | Claude-4.6-Sonnet | Claude-4.6-Sonnet CI | GPT-OSS-120B | GPT-OSS-120B CI | COS-PLAY | COS-PLAY CI | Relative gain |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2048 | mean-reward | 1126.6 | 150.2 | 813.3 | 143.6 | 945 | 134.5 | 1029.5 | 122 | 1589 | 192.4 | +41.0% |
| TETRIS | mean-reward | 458.2 | 203.5 | 372.7 | 157.7 | 444.2 | 182.6 | 358.1 | 139.7 | 510.9 | 199.5 | +11.5% |
| CANDY CRUSH | mean-reward | 532.6 | 24.8 | 334.3 | 59.4 | 328.6 | 23.8 | 334.4 | 40.5 | 648.8 | 38.8 | +21.8% |
| SUPER MARIO | mean-reward | 752 | 35.7 | 436.8 | 86.1 | 399.5 | 53.4 | 968.5 | 175 | 948.9 | 153.2 | +26.2% |
| AVALON | win-rate | 65 | 14.2 | 42 | 13.2 | 40 | 13.1 | 40 | 13.1 | 39 | 9.4 | Not claimed |
| DIPLOMACY | supply-centers | 4.7 | 0.35 | 2.72 | 0.26 | 3.16 | 0.19 | 2.46 | 0.25 | 2.96 | 0.2 | Not claimed |
The score was only the easiest part to measure. In the paper’s qualitative examples, the agent also held skills over longer sequences, switched more deliberately as the state changed, and built a broader strategic repertoire.
The strongest result came when the model and skill bank evolved together. A later skill bank could make an earlier model worse because its skills no longer matched the situations that model created. Better components still had to stay aligned.
That coupling turns stronger behavior into better evidence for evaluation, skill learning, and training. By adding environments that pressure different abilities, we can test which improvements stay inside one game and which begin to travel.
How the improvement loop scales
The next challenge is to create, run, and observe these improvement loops at the pace of AI.
No one can inspect every action at that speed. We need OBSERVABILITYThe tools and views that show what an agent tried, which skills it used, where it failed, and whether a new version is actually better.: a clear view of what the agent tried, which skills it used, where it failed, and whether the next version is actually better. That evidence makes the system trustworthy enough to scale.
The human role moves up a level: set the goal, read the signals, and plan the next improvement. At Good Start Labs, we are building the environments, harnesses, training systems, and observability required to close that loop.
CLOSE THE LOOP
Design the system your agents learn inside
We turn games and capabilities into environments, evaluations, and improvement loops.