70% Right, Zero Edge: A Toolchain Built to Kill My Own Ideas
This year I built a five-instrument toolchain to answer one question about my own trading ideas: is there anything here, or am I fooling myself?
The toolchain's verdict on my most convincing strategy: it was right 70.8% of the time, and it had zero edge. Every part of that sentence is the point of this case study.
The Auditor's Reflex
I spent my first professional years auditing critical infrastructure for New York City — fifty-plus OT/ICS systems, where the job is never "does this look fine?" but "show me the evidence it is fine." When I started looking seriously at trading strategies, that reflex kicked in before any strategy did: before asking whether a backtest wins, ask whether it lies.
Most published trading strategies lie. Not maliciously — structurally. Backtests peek at future data through lookahead flags, repaint signals after the fact, fill limit orders at prices no live order would get, and default to a world with zero commissions and zero slippage. A strategy can compound 87% a year in that world and die instantly in this one.
So instead of hunting for an edge, I built instruments. Five of them, each designed to say no as cheaply as possible.
Instrument One: A Static Auditor for Backtest Lies
The first tool is a static analyzer for Pine Script (TradingView's strategy language) with 21 deterministic rules covering the ways backtests deceive: lookahead bias, repainting, optimistic fills, zero-cost assumptions, overfit parameter smells.
One design rule governed the whole build: if a check can be expressed as a parse or a regex, it is a rule — never a prompt. LLMs review what static analysis cannot reach; everything mechanical stays mechanical, because an auditor whose findings vary run-to-run is not an auditor.
Validated against eight real public strategies, it produced 88 findings with zero false positives. It is open source: github.com/lymnal/pine-auditor.
Instrument Two: A Journal That Cannot Be Argued With
Live signals need a record that is immune to selective memory. I built a hardened webhook receiver that journals every alert my charts emit — append-only, fsync-per-write, with one invariant: every request produces exactly one journal record, including the rejected ones.
The constraints made it interesting: TradingView gives a webhook three seconds, never retries, and signs nothing. So the service is zero-dependency TypeScript with constant-time secret comparison, source-IP allowlisting, and risk caps that are derived from the chart's timeframe rather than configured by hand — because a hand-configured cap is a cap someone will hand-loosen.
It records signals. It places no orders. That is an architectural commitment, not a feature gap.
Instrument Three: A Grader That Separates Accuracy From Edge
Here is the trap the whole toolchain exists to expose: a signal can be right almost every time and still be worthless, because the move it predicts is already in the price.
The grader scores journaled signals against realized forward returns — but the headline number is not the hit rate. It is edge against a baseline: the mean forward return of every bar in the same window. Being right in a market that rises is not skill; beating a random entry into that same market is. Hit rate prints last, because it is the number that misleads.
The Bug That Almost Lied to Me
The first real grading run reported a spectacular result: significantly negative edge, t = −4.48. A tradeable anti-signal! I did not believe it — results that interesting are usually bugs — and dug in.
They were. The strategy was long-only, and the grader was scoring its exits as short bets, marking every routine exit in a rising market as a failed prediction. The fix was semantic, not numeric: an exit is not a bet that price falls, and a sell without position-size context is excluded and counted, never guessed. Corrected, the edge was +0.209% at t = 0.66 — statistically nothing.
I keep this bug in the case study deliberately. An instrument you have not tried to disbelieve is an instrument you cannot trust.
What the Instruments Found
Run end to end on ten years of real index data, the toolchain killed everything I fed it, each in a different, instructive way:
- The 87%-return strategy died at 21.4 basis points of round-trip cost — transaction friction alone erased it. Its lower-returning sibling survived costs and still lost to doing nothing.
- Buy-and-hold returned 307% over the same window and beat every active variant tested.
- My live strategy — the convincing one, the one that felt right on the chart — graded at 70.8% accuracy with zero edge. The direction was right; the price already knew.
Instruments, Not Opinions
Nothing in this project produced a trading edge, and that was the design goal honestly stated: the deliverable is a cheap, fast, trustworthy no. Every idea killed in seconds by a static rule is an idea that never costs real money to disprove.
Security auditing and this are the same discipline pointed at different targets. You do not ask a system to testify about itself; you build instruments, you try to break your own findings, and what survives — if anything survives — is the only thing worth acting on.