Research
A new benchmark, KTD-Fin, anonymises tickers, dates and market narratives before testing ten frontier LLM trading agents on China's CSI300, then uses performance attribution to show most of their returns come from passive market and style exposure rather than genuine stock-picking skill
The paper's core complaint about existing LLM trading benchmarks is that long backtests overlap with a model's training-data knowledge cutoff, letting memorised prices and narratives masquerade as investment reasoning. KTD-Fin's masking protocol strips that out, and the attribution results that follow are unflattering.
KTD-Fin, short for Knowing-to-Doing Financial Benchmark, targets a specific and fairly damning flaw in how LLM trading agents get evaluated: if a backtest period falls inside a model's training data, the model can recall what actually happened to a stock rather than reason about what should happen given the information available at the time. The paper's fix is a masking protocol that anonymises tickers, calendar dates and market narratives consistently across every prompt and tool call, so the agent can't lean on memorised outcomes even if it has seen them during training.
Run across ten frontier LLM agents trading China's CSI300 over a 2024-2026 window, the masking changes agent behaviour immediately, pushing reasoning toward anonymised factor-style logic instead of narrative-driven stock calls. The more consequential result comes from the paper's Barra-style attribution layer, which decomposes each agent's returns into market exposure, style exposure and genuine stock-selection alpha. Under leakage-controlled evaluation, the paper finds that agents' cumulative returns are mostly explained by market and style exposure, with limited evidence of persistent stock-picking skill once that's stripped out.
That result lands in the same place this site has now seen from several different angles: the Edinburgh/UCLA buy-and-hold study, the Backtrader-Bench near-random performance on out-of-sample questions, and now an attribution-based benchmark that's specifically designed to close the memorisation loophole the other two didn't fully address. Each paper uses a different method and still arrives at a version of the same finding, that headline LLM trading returns look a lot weaker once the evaluation controls for what the model could plausibly already know.
What KTD-Fin adds that the others didn't is a mechanism for separating skill from exposure, not just skill from memorisation. A trading agent that's really just running a geared momentum or growth factor under the hood will show attractive raw returns and a weak attribution decomposition, and this benchmark is built to catch exactly that pattern. For anyone evaluating a live or backtested AI trading strategy, that's the more useful question to ask than whether the raw Sharpe ratio looks good.
Read the original: arXiv - A new benchmark, KTD-Fin, anonymises tickers, dates and market narratives before testing ten frontier LLM trading agents on China's CSI300, then uses performance attribution to show most of their returns come from passive market and style exposure rather than genuine stock-picking skill. Commentary is the independent editorial view of Share Trading; the original article is credited to its publisher.