Preventing Overfitting in Quantitative Finance
Recently, I gave an informal talk at Columbia University about how to prevent overfitting of algorithmic trading models. Personally, I think there is a general lack of statistical rigor in finance, with many traders not following recommended best practices in data science.
- Obtain good data that is of high granularity, up to date, and doesn’t contain survivorship bias.
- Make the problem easier by transforming inputs and outputs.
- Apply regularization to penalize the complexity of the model.
- Split data into test/training/validation sets for backtesting.
- Use walk-forward optimization and don’t only optimize for portfolio returns; I recommend optimizing for Sharpe ratio.
- Don’t ever assume financial data is generated by a Gaussian distribution.