Research
Researchers Junyi Yao and Zihao Zheng audited 30 LLM-based trading studies across data provenance, execution timing, turnover treatment and transaction-cost modeling, finding that agent architecture is well-documented while evaluation methodology is not, and that realistic frictions materially compress reported returns
The paper scores each study on seven dimensions, point-in-time controls, split transparency, held-out evaluation, cost and turnover treatment, execution semantics, universe definition and artifact availability, building a coded evidence matrix rather than a narrative literature review. A worked 10-equity example shows that adding explicit transaction costs and realistic execution timing can substantially shrink the outperformance claimed by an active LLM strategy versus a naive backtest.
This is the third reproducibility-focused paper on LLM trading agents to surface on this site in the last three weeks, after the 77-study reproducibility review in July and the CLQT look-ahead-leakage benchmark on August 12. That clustering is itself the story: enough people are now trying to build on top of the initial wave of LLM-trading papers that the gaps in how those papers reported their results have become a bottleneck, not an academic footnote.
The specific finding, that architecture descriptions are well-documented while evaluation methodology is not, is exactly the failure mode you'd expect from a fast-moving research area chasing headline results. It is much easier to describe a novel agent architecture in a paper than to rigorously document point-in-time data controls, held-out evaluation splits and realistic transaction-cost assumptions, and reviewers and readers have historically rewarded the former over the latter.
The 10-equity worked example is the part that should worry anyone using published LLM-trading backtests to inform real capital allocation. If explicit friction and timing choices can 'materially compress' an active strategy's reported edge, then a meaningful share of the eye-catching outperformance numbers circulating in this literature may not survive contact with a realistic execution model, let alone real market impact.
None of this means LLM trading agents don't work, it means the field currently has no reliable way to tell which ones do. A coded evidence matrix across 30 studies is a useful diagnostic, but the actual fix, standardized reporting requirements for execution realism and reproducibility, is a coordination problem the field hasn't solved yet, and won't solve until enough allocators start asking these seven questions before deploying capital against a paper's backtest.
Read the original: arXiv - Researchers Junyi Yao and Zihao Zheng audited 30 LLM-based trading studies across data provenance, execution timing, turnover treatment and transaction-cost modeling, finding that agent architecture is well-documented while evaluation methodology is not, and that realistic frictions materially compress reported returns. Commentary is the independent editorial view of Share Trading; the original article is credited to its publisher.