Why You Shouldn’t Judge by PnL Alone

Data Science Tech Brief By HackerNoon

This story was originally published on HackerNoon at: https://hackernoon.com/why-you-shouldnt-judge-by-pnl-alone.
PnL can lie. This hands-on guide shows traders how hypothesis testing separate luck from edge, with a Python example and tips on how not to fool yourself.
Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #quantitative-research, #trading, #algorithmic-trading, #pnl, #udge-pnl, #profit-and-loss, #judge-profit-and-loss, #hackernoon-top-story, and more.
This story was written by: @ruslan4ezzz. Learn more about this writer by checking @ruslan4ezzz's about page, and for more stories, please visit hackernoon.com.
I’ve spent years building and evaluating systematic strategies across highly adversarial markets. When you iterate on a trading system, PnL is the goal but a terrible day-to-day signal. It’s too noisy, too path-dependent, and too easy to cherry-pick. A simple framework—form a hypothesis, measure a test statistic, translate it into a probability under a “no-effect” world (the p-value)—helps you avoid false wins, iterate faster, and ship changes that actually stick. Below I’ll show a concrete example where two strategies look very different in cumulative PnL charts, yet standard tests say there’s no meaningful difference in their average per-trade outcome. I’ll also demystify the t-test in plain language: difference of means, scaled by uncertainty.