Ask a language model for a strategy that makes money and it will give you one: an entry rule, a reason it should work, perhaps a number for how well it did. The answer reads well. It is almost always made up, because the model has never run the rule.
The lab's assistant answers the same request another way. It builds the test you describe on your canvas, runs it on twenty years of daily history and shows you what happened, including when the rule did worse than simply holding the stock. This essay is about why we built it like that.
Feynman (1974), in his "Cargo Cult Science" address, put the first principle plainly: you must not fool yourself, and you are the easiest person to fool. A fluent assistant makes that danger larger, because a confident answer feels like evidence. An assistant in a research lab should make fooling yourself harder.
Why "find me alpha" is the wrong question for a language model
A language model writes the text most likely to follow its prompt. Asked for a profitable strategy, it writes something that sounds profitable, because that is the shape of the many confident write-ups it has read. It cannot see whether the rule worked on years it was not tuned on, how many rules were tried before this one, or how few trades the result stands on.
Careful quantitative research is organised against exactly this. Harvey and Liu (2015) showed that once you count how many factors have been tested, most published "discoveries" fail a reasonable bar: with enough tries, a spurious result is the default. Bailey, Borwein, López de Prado and Zhu (2014) made the same point in their "Pseudo-Mathematics" essay: with enough trials, a beautiful in-sample Sharpe ratio can be produced for almost any series. A model asked for an edge is a tireless trial-runner with a gift for stories.
What the assistant does with your idea
You say the rule in your own words, for example "buy Microsoft when its 2-day RSI falls under 10 and sell above 70". The assistant puts the lab's own cards on your canvas while you watch: the stock's prices, the rule, a walk-forward that tunes it on past years only, and a forward test that trades it on the years after, which the tuning never saw. You can undo every step it takes, and every card is one the lab has: it reads them from the lab's own catalog, with their settings and ranges, so it cannot describe a tool that does not exist.
Where your words leave a setting open, it chooses one and says so in a line ("I used SPY as the benchmark and tested the last 20 years"). Then it runs the circuit, and the lab writes the reading under it: whether the rule beat simply holding, its worst fall beside holding's, and what it did in 2008 and 2020 when the run covers them. A loss is said as plainly as a gain.
- It explains. Ask what a card does or how a number was worked out, and it goes through the math in the lab's own terms.
- It questions a result. A high Sharpe ratio on a handful of trades, one lucky setting, no years out of sample: ask it where the result is weakest and it tells you.
- It reads your own work. Your saved projects and their runs, when you ask, and only yours.
- It changes one setting and runs the test again, so you see what the setting does.
What it will not do
It will not tell you what to trade. It reports what your own test did. It never recommends a trade and never names a future return. When you run a strategy live on paper, the note you get each morning describes your strategy's day and asks nothing of you.
It will not start anything behind your back. Before a strategy goes live on paper, the lab asks first, and nothing is added or registered before your yes.
It will not pretend. When the lab cannot build what you asked (a covered call, say: the lab has no options), it says so in one line and offers what the lab can do.
None of this makes the lab's assistant a source of truth. It can misread what you meant, choose a default you would not have chosen, or say something with more confidence than it has earned. Treat it the way you would treat a sharp colleague who has not seen your data before: quick at building the test and finding holes, never the final word. The evidence is in the run: the years the rule never saw, and the line for simply holding beside it.
The useful inversion
When you like an idea, you look for confirmation, and an assistant that gives it feels wonderful until the live results disagree. Aronson (2006) wrote a whole book on putting technical-analysis claims through real statistical tests, because the field had so often skipped them; what survives is a small part of what felt convincing. Popper (1963) put the attitude in one line: a claim earns its standing by how hard we try to break it.
So use the assistant the other way round. Build the test, run it on the years it never saw, then ask where it is weakest. It cannot make your strategy true. It can show you, in a minute, whether the evidence is there.
- A language model asked to "find alpha" writes a fluent story: the exact failure careful research exists to prevent.
- The lab's assistant builds the test you describe with the lab's own cards, on your canvas, and runs it on twenty years of daily history.
- Every result is read against simply holding, with the worst fall and the hard years beside it.
- It never tells you what to trade, never names a future return, and asks before anything goes live on paper.
- Ask it to break your idea: the evidence is in the years the rule never saw.
Further Reading
Foundational papers
- Feynman, R. P. (1974). Cargo Cult Science (Caltech commencement address). Engineering and Science, 37(7), 10–13.
- Popper, K. R. (1963). Conjectures and Refutations: The Growth of Scientific Knowledge. Routledge & Kegan Paul.
- Harvey, C. R. & Liu, Y. (2015). Backtesting. Journal of Portfolio Management, 42(1), 13–28.
- Bailey, D. H., Borwein, J. M., López de Prado, M. & Zhu, Q. J. (2014). Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance. Notices of the AMS, 61(5), 458–471.
Textbook references
- Aronson, D. R. (2006). Evidence-Based Technical Analysis: Applying the Scientific Method and Statistical Inference to Trading Signals. Wiley.
Related QuanterLab articles
Try it in QuanterLab
Try it in QuanterLab. Open the lab and say a rule in your own words, for example buying a stock after a sharp fall. Watch the assistant build it card by card and run it, then ask where the result is weakest. It shows you what your rule did; what to trade stays your call.