TL;DR: Elm Wealth ran eight language models through a real historical trading simulation with leverage. The models were credible at picking direction. They were catastrophic at sizing. The loss, in every case, came from the multiplier, not the view. This is a structural finding, not a statistical one.

• Language models can form directional market views that beat human benchmarks.
• They consistently over-leverage: average gross leverage near 30x against a defensible 7.5–15% volatility target.
• The models know the Kelly Criterion. They apply it in coin-flip games. They do not apply it in market scenarios. This is a transfer failure, not an ignorance failure.
• Prompt-based fixes (telling the model to read a sizing book) improved results but remain fragile: one context reset and the fix is gone.
• Human supervisors exposed to AI opponents got more aggressive, not less. Approval is a permission. It is not a size check.

Eight language models were handed tomorrow's newspaper and a million dollars. They were credible at deciding what to bet on, and catastrophic at deciding how much.

Elm Wealth ran an experiment I've been thinking about for a week. It landed directly on a decision I'd already made in code, for reasons I thought were mine, and I want to write down why.

The setup is their Crystal Ball Challenge, adapted for machines. A player gets the front page of tomorrow's Wall Street Journal, market moves redacted, and can trade the S&P 500 and 30-year Treasuries with leverage. Fifteen trading days drawn from 2008 to 2022, a million dollars, and instructions to treat that million as their entire financial wealth. Elm ran eight model configurations through it: base and reasoning versions of Claude, ChatGPT, Gemini, and Grok.

The headline is easy to misread. Claude's base model finished at $2.59M and beat 76% of the humans who'd played; the reasoning configuration finished at $3.34M with a hit ratio near 68%. It would be simple to write the post that says the models can trade now.

That's not the result worth examining. The result worth examining is that the models were good at one half of the job and disastrous at the other, and the half they were bad at is the half nobody talks about.

Two decisions, not one

Every position is two decisions. What to bet on, and how much. They feel like one decision because they arrive together, but they're governed by completely different things.

Decision one: The call

A claim about the world. Testable against the tape. This is the part the models did well.

Decision two: The size

A claim about your own tolerance for ruin. Not inferable from the news. This is the part they got wrong, every time.

On Elm's numbers, base models ran an average gross leverage around 30x when playing solo, and 7–12x in stocks specifically. That produced daily return standard deviations of 20% to 40%. Elm's benchmark is the Merton share: the optimal volatility to carry is your Sharpe ratio divided by your risk aversion. For a typical risk aversion of 2 to 4, that would have justified something in the range of 7.5% to 15%. The models were running somewhere near double to quadruple a defensible number, on edges that didn't remotely support it.

The cleanest illustration is Gemini's base run. It finished at $492,357, half the stake gone, while posting a positive Sharpe ratio of 0.02. Elm's counterfactual: had it made the same calls at one-tenth the size, it would have ended around $1,005,000. Same information. Same directional views. Same everything except the multiplier. Half the book, or slightly up.

The sign of the edge and the sign of the outcome came apart. Nothing in the model's reasoning about markets was responsible for the loss. The size was.

Key point: Directional accuracy is necessary but not sufficient. A correct view plus the wrong size still ruins the account.

They know the theory. They can't reach it.

Elm asked the same four models to play a coin-flip game. All four gave correct answers grounded in the Kelly Criterion. They know the maths. They can explain fractional Kelly, the geometric-growth argument, the whole apparatus.

And then, in Elm's phrase, they "couldn't apply it when the problem was dressed up in market rather than coin-flip clothing."

This is not an ignorance failure. It's a transfer failure. Transfer failures are worse news because you cannot fix them by telling the model more. The model already had the thing. It was sitting right there, correctly recalled, unconnected to the decision it was supposed to govern.

Elm did find something that helped. When they told the models to read their book on position sizing before playing, every one of the four improved: Claude to $1.36M, Gemini to $1.30M, ChatGPT to $1.20M, Grok to $1.12M. An externally supplied prior about size worked where the model's own knowledge of the same material didn't.

Note the shape of that fix. It's a prompt. It survives exactly as long as the prompt does. One context reset, one model swap, one adversarial input, and it's gone. The right instinct, implemented in the most fragile possible medium.

Key point: The model's failure is not about knowledge. It's about context-dependent application. You cannot train your way out of this with more theory.

Where I'd push back on the study

The part I don't believe

I don't think the model ranking in this study means much, and I'd rather say so than quote the table approvingly.

Fifteen opportunities across two instruments is roughly 30 observations. A Sharpe ratio estimated on 30 observations carries a standard error somewhere near 0.19; a reported 0.31 doesn't separate from zero at any conventional threshold. A 60.7% hit ratio on that sample sits about one standard error from a coin. And Gemini's reasoning configuration sized worse than its base configuration, 14.6x against 11.7x, which is the kind of inversion you get from noise rather than from a property of the model.

I hold the sizing finding at high confidence and the leaderboard at low. The sizing finding doesn't depend on the statistics: it's unanimous across eight configurations and two separate problem framings, and it's structural. The ordering is a 30-sample horse race.

The leaderboard is the part that will get screenshotted. That's the distinction worth flagging.

Key point: The sizing failure is structural and unanimous. The model rankings are statistically unreliable. Treat them accordingly.

What I'd already built, and why I now trust it more

I keep a system called keel, the front-office layer of the platform I'm building, run on my own book. It runs on paper. It has never sent an order to a real venue, never spent real money, and its language-model challenger has never been pointed at a live model API.

keel's central architectural decision is that agents get seams and never rails. A model can produce a signal, propose a trade, or challenge one. It cannot size. There is no seam for that. The limits live in afo/risk/limits.py, the gate lives in afo/agents/guardrail.py, and neither is reachable from anything a model touches. When a proposal exceeds a cap, the guardrail returns a REPAIR verdict: cut to the cap, and record that it was cut.

I wrote that rule for a governance reason: every decision replayable against a fixed set of controls. Elm has now given me a behavioral reason for the same rule, from real historical tape. It's the better reason.

There's a second finding I didn't expect. Elm found that humans playing head-to-head against the AIs got more aggressive. Average leverage rose from 25x to 30x, and players maxed out twice as often. The human supervisor is running the same failure mode. That's an uncomfortable finding for anyone whose control story is "a person approves each order," because approval is a permission, not a size check. keel already separates those two. Every approved item is re-checked against current state at execution. I'd written that rule to defend against stale prices. Not against the approver.

Key point: Architectural enforcement matters more than instructed behavior. If the model can touch the sizing, the sizing is not controlled.

The gap it opened

Honest cost of reading the piece properly: it names something keel doesn't have.

My position cap is 25% of net asset value. That's a notional cap. Elm's entire argument is that correct size scales with volatility, not with notional. A quarter of the book in a placid instrument and a quarter in a volatile one are identical numbers to my guardrail and wildly different risks to the account. Nothing in keel's money path is volatility-scaled. The sizing models that are volatility-aware live in a research harness deliberately forbidden from touching the guardrail.

That's now a real piece of work with a real argument behind it, which is more than it had a fortnight ago.

Key point: Notional caps are not risk caps. Volatility-scaling is the missing piece in most guardrail architectures, including this one.

The thing underneath

If you are putting a model anywhere near a decision that consumes capital, the question worth asking is not whether it can form a view. On this evidence it can form a decent one.

The question is whether the size of the consequence is under its control. The study shows the model will recite the correct sizing discipline, in detail, on request, and then not apply it. If the only thing standing between that and your balance sheet is an instruction in a prompt, you don't have a control. You have a preference that the model is currently honouring.

Anyone who has to evidence a control to a regulator already knows the difference.

Key point: A preference is not a control. The architecture has to enforce sizing, not request it.

Frequently asked questions

What is the Crystal Ball Challenge?
An experiment where players receive the front page of tomorrow's Wall Street Journal with market moves redacted, then trade the S&P 500 and 30-year Treasuries with leverage. Elm Wealth adapted it for language models across 15 historical trading days drawn from 2008 to 2022.

What is a transfer failure?
A transfer failure occurs when a model possesses correct theoretical knowledge but cannot apply it in a different framing of the same problem. In this case: the models knew and correctly applied Kelly Criterion in a coin-flip game, then failed to apply it when the same problem was framed as a market trade.

What is the Merton share?
The Merton share is a formula for the optimal amount of volatility to carry, calculated as your Sharpe ratio divided by your risk aversion. For a risk aversion of 2 to 4, this produces an optimal daily volatility target of roughly 7.5% to 15%. The models in Elm's study were operating at 20% to 40%.

Why doesn't a prompt fix the sizing problem?
Prompts are fragile. They survive only as long as the context window, the model version, and the input remain consistent. One context reset, one model swap, or one adversarial input removes the fix. A prompt is a preference, not an enforced control.

What does it mean that human supervisors got more aggressive around AI?
Elm found that humans playing head-to-head against the models increased their average leverage from 25x to 30x and maxed out their positions twice as often. The implication: human oversight that consists only of approving a model's proposed trade does not control sizing. The person approving the order is not immune to the same bias the model exhibits.

What is keel?
keel is a front-office system Reece Frazier is building under NOMARK, run on paper against his own book. Its architectural rule is that language models can propose trades but cannot set position sizes. Sizing is enforced in afo/risk/limits.py and afo/agents/guardrail.py, neither of which is reachable from any model input. This is not investment advice, and the system has not traded real capital.

What is the difference between a notional cap and a volatility-scaled cap?
A notional cap limits how much of the book can go into a single position by face value. A volatility-scaled cap adjusts that limit based on how much the instrument moves. A 25% notional cap applied to a low-volatility bond and a high-volatility equity produces identical numbers on paper and very different risk exposures in practice.

Why does this matter for regulated financial firms?
Regulators require evidenceable controls. A prompt is not evidenceable as a control because it is not enforced; it is requested. An architectural limit that fires a recorded REPAIR verdict when a threshold is breached is evidenceable. The study gives practitioners a behavioral basis for demanding the architectural version.

Key takeaways

• Language models beat human benchmarks on directional market calls. That finding is real but incomplete.
• The same models ran 20% to 40% daily return volatility against a defensible target of 7.5% to 15%. Sizing failure wiped accounts that had correct views.
• This is a transfer failure. The models know Kelly Criterion. They do not apply it when the problem looks like a market trade instead of a coin flip.
• Prompt-based sizing corrections work temporarily. One context reset removes them. They are preferences, not controls.
• Human supervisors became more aggressive in the presence of AI opponents. Approval is not a size check, regardless of who is approving.
• Notional caps and volatility-scaled caps are not the same thing. Most guardrail architectures enforce the former and ignore the latter.
• Architectural enforcement that prevents a model from touching sizing at all is the structural answer. Instructed behavior is not.

keel is an engineering proof running on my own capital, on paper. Nothing here is investment advice, and none of it predicts profit. Elm Wealth's write-up: elmwealth.com/ai-trading. All figures attributed to Elm are theirs; the statistical caveats and every reading of keel are mine.