Research
A systematic review of 77 studies on LLM trading agents found only 2 reported a proper experimental protocol, meaning most of the field currently cannot be independently checked
Researchers auditing the LLM-agent-trading literature found 15 of 19 primary empirical studies sat at the lowest reproducibility tier, with almost none disclosing transaction-cost assumptions or addressing survivorship bias, a gap that matters now that agentic tools have moved from papers into live brokerage accounts.
The review, Agentic Trading: When LLM Agents Meet Financial Markets, took an audit-oriented approach rather than the usual survey-paper summary of who did what. The authors screened 77 studies on how large language models perceive market information, retrieve context, reason about decisions, emit tradable actions and adapt to feedback, then narrowed to a primary subset of 19 papers that actually produced action outputs and closed-loop evaluation, the kind of study that claims to show an LLM agent trading and generating a result, not just discussing the idea.
What they found inside that subset of 19 is the headline. Only two reported an extractable, time-consistent experimental protocol that another researcher could actually follow to reproduce the result. Only one documented an explicit transaction-cost model, meaning eighteen of nineteen studies claiming trading performance did not clearly account for the cost of actually executing the trades they describe. Only one addressed survivorship bias. Fifteen of the nineteen were coded at the lowest reproducibility tier the authors defined, and none reached the highest. This is not a field with a few weak papers dragging down an otherwise solid average, it is a field where rigor is the exception rather than the norm.
The timing is what makes this more than an academic complaint about publication standards. This review is landing in the same stretch of 2026 where agentic trading tools have gone from research demos to products with tens of thousands of live retail accounts attached, and where institutional desks are being surveyed on how fast they plan to deploy AI into execution workflows. The architectures generating the backtested performance claims that presumably inform some of that commercial confidence are, per this audit, mostly untestable by outside researchers. A trading system that cannot be independently reproduced is not automatically wrong, but it means nobody outside the team that built it can currently tell you whether it is right for the reasons claimed.
The authors' ask is for comparable evaluation protocols, clearer execution semantics and reproducible artifacts, the unglamorous infrastructure work that turns a promising architecture into a verifiable one. Whether platforms shipping agentic trading tools to real customers adopt anything resembling that discipline before the next serious drawdown is the thing worth watching. Right now the gap between what gets claimed in a paper and what survives contact with live capital and real transaction costs is, by this review's own measure, close to a coin flip on whether anyone could even check.
Read the original: arXiv - A systematic review of 77 studies on LLM trading agents found only 2 reported a proper experimental protocol, meaning most of the field currently cannot be independently checked. Commentary is the independent editorial view of Share Trading; the original article is credited to its publisher.