Research
Researchers Bo Qu and Mingguang Chen publish CLQT, a benchmark showing that once look-ahead leakage and transaction costs are controlled, the apparent top-performing LLM portfolio-management agent in existing tests is often just a reliability artefact from a handful of lucky rounds
The paper, submitted in June and revised 2 August 2026, reframes evaluation of LLM trading agents as diagnosis rather than ranking, running a five-stage gather-synthesise-allocate-execute-reflect cycle with verifiable audit trails and a five-axis capability scorecard. Its central finding is that the agent with the best headline Sharpe ratio differs from the agent with the best underlying capability profile, with the nominal winner in one test traced to what the authors call a five-of-26-round reliability artefact.
This paper is aimed squarely at a problem every LLM-trading benchmark to date has had, which is that ranking agents by returns over a fixed backtest period conflates two very different things: how good the agent's decision-making actually is, and how favourable the market path happened to be during the window it was tested in. CLQT's authors state it plainly, the market path dominates a period's return, which means most existing leaderboards are measuring luck as much as skill and calling it a ranking.
The five-of-26-round finding is the paper's sharpest result and the one worth remembering next time a vendor or fund cites a backtest-topping LLM trading agent. If a model's headline win traces back to strong performance in five rounds out of twenty-six, with the other twenty-one contributing little or nothing distinctive, that is not evidence of a superior strategy, it is evidence of variance that happened to land in the model's favour during the test window. Swap the test window and the ranking plausibly inverts.
The methodological contribution that matters beyond this one paper is the shift from ranking to diagnosis: a five-axis scorecard, coherence, acuity, composure, discipline, reliability, that can register genuine differences between agents even when their financial returns are statistically indistinguishable. That is a more honest and more useful output for anyone actually trying to select or improve a trading agent, because it tells you where a model is weak rather than just telling you it lost money in one particular quarter.
The practical warning for the industry is the finding that signal-action agreement consistently exceeds coherence judgments, by 0.30 in backtest and 0.23 live. In plain terms, agents are more consistent about which action they take than about whether their own stated reasoning actually supports that action, which is a specific and testable form of the black-box problem regulators keep flagging in supervisory language without much empirical backing. CLQT gives that concern a number. Anyone deploying an LLM trading agent and treating its chain-of-thought as a genuine audit trail should treat this finding as reason to verify that assumption rather than take it on faith.
Read the original: arXiv - Researchers Bo Qu and Mingguang Chen publish CLQT, a benchmark showing that once look-ahead leakage and transaction costs are controlled, the apparent top-performing LLM portfolio-management agent in existing tests is often just a reliability artefact from a handful of lucky rounds. Commentary is the independent editorial view of Share Trading; the original article is credited to its publisher.