Research

A new SSRN working paper builds the first structured taxonomy of LLM and multi-agent trading systems and uses it to name four open research gaps, the most consequential being that nobody has yet built a rigorous, cost-adjusted benchmark for comparing LLM-generated trading signals against each other

The taxonomy sorts existing work along five dimensions, the LLM's functional role, information modality, learning paradigm, single-agent versus coordinated multi-agent architecture, and evaluation standard, and finds that most published results sit at the weakest end of that last dimension, in-sample backtests rather than live-market evaluation.

· Source: SSRN


The most useful thing a taxonomy paper can do is not propose a new model, it is make visible how much of a field's published output is actually comparable to itself. This one does that job well. Sorting the existing LLM-trading literature along five dimensions, functional role, data modality, learning paradigm, architecture, and evaluation standard, exposes how differently researchers have been defining success. A paper claiming a trading LLM works based on an in-sample backtest and a paper claiming the same thing based on live-market results are not making comparable claims, even when both papers use the phrase state-of-the-art.

The evaluation-standard gap is the one worth sharing outside academic circles. If most of the published literature on LLM trading agents clusters at in-sample backtesting rather than live or even rigorous out-of-sample testing, that is a direct explanation for why the gap between reported academic performance and real deployed performance keeps surprising people. In-sample backtests are notoriously easy to overfit, intentionally or not, and a field where that is the dominant evaluation method will keep producing headline results that do not survive contact with live markets.

The missing cost-adjusted benchmark is the more actionable gap for anyone actually trying to deploy these systems. A trading signal that looks profitable before transaction costs, slippage, and API inference costs can be worthless after them, and LLM-based signals carry a cost structure, per-call inference pricing, latency, that traditional quant signals never had to account for. Without a standard benchmark that bakes those costs in, every claimed result requires the reader to independently reconstruct what the real, deployable return would have been, which most readers do not do.

The multi-agent failure-mode gap points toward where the next wave of interesting negative results is likely to come from. Coordinated multi-agent trading systems, several LLM agents in different roles debating a position before executing, are the architecture generating the most attention right now. This paper's read is that almost nobody has stress-tested what those systems do under adversarial conditions or in illiquid names, which are exactly the conditions where a coordination failure between agents would do the most damage.


Read the original: SSRN - A new SSRN working paper builds the first structured taxonomy of LLM and multi-agent trading systems and uses it to name four open research gaps, the most consequential being that nobody has yet built a rigorous, cost-adjusted benchmark for comparing LLM-generated trading signals against each other. Commentary is the independent editorial view of Share Trading; the original article is credited to its publisher.