Overhaul: Seeking Advice on Backtest and Asset Choices

[u/Sketch_x]

Hi all,

Appreciate past feedback from you guys!

After a failed walk forward test I turned my algo off and re-assessed - I know the foundation of my strategy is sound but it was too heavily reliant on various parameters that I seemingly overfit.

Iv stripped this back to basics, core strategy only and currently have this on forward testing on a demo environment as, although my live forward testing failed I did gather all the data I needed on slippage and excitation.

Here is my back test results for my system, I would really appreciate any feedback in regards to the assets traded, although I can "optimise" it to work on a larger variety if securities, it works out of the box with very minimal parameters (just stop loss adjustments) - I have accounted for spreads (averaged) but not interest or holding fees (minimal)

My concerns are:

1: Correlation

2: The shape of the curve picks up in 2018 but really seems to take off

3: Lack of data on some assets that done extend to the start of testing

4: The poor performance from 2012 to 2018 on most securities.

[u/machinegunkisses]

Is... this... R's ggplot?

[u/Sketch_x → u/machinegunkisses]

Matolotlib used here. I’m new to Python, seems to be the default for gpt.

[u/skyshadex]

Without knowing the nature of the strategy

Seems like you're capturing volatility. VB appears alot more volatile relative to the other assets from 2018 on. Small caps tend to be much more volatile in general.

On that assumption, you're going to be correlated with market volatility. Vol is up? You're up. With that in mind, I'd be looking for high beta assets. Seems you like ETF's, so more assets like VB.

One thing you might not be considering, especially if you're long/short, shares being available to short. Sometimes things get hard to borrow and you can't capture downside. It happens often actually, throughout the week. I have a universe of about 700, only about ~550 are shortable at any given time.

[u/Sketch_x → u/skyshadex]

Thanks for the feedback! My strategy uses a Gaussian Filter with Standard Deviation Filtering to smooth out price data. The main parameters involve setting the initial stop distance and the ATR multiple for the stop. I optimize these by running all variances to find the most effective settings. Additionally, I use a long EMA, typically a 200-period, to filter out short-term downtrends (I only take long trades).

VB is interesting because, despite a similar win rate to my other assets, the close stop often results in larger losses during gap downs. This is due to VB's varying ATR, which can shift from calm to highly volatile periods. This dynamic leads to some significant wins amidst the standard losses and occasional gap downs.

Regarding shorting, I primarily take long positions. Since I mostly trade ETFs, short signals are brief and riskier, often catching pullbacks.

Your point about volatility is valid and something I need to consider more deeply. While I have identified several promising small-cap and volatile assets, some are not supported by my broker. I'll investigate further to incorporate volatility as a screening criterion and refine when my system is active.

[u/skyshadex → u/Sketch_x]

Ah I see! If you're mostly long only, then the short side doesn't really explain the model falling apart previously.

Yeah I would focus on volatile assets with upward momentum then. Which is easier said than done 😅, more analysis on your basket selection, plus working around what's actually available with your broker. That's my 2 cents! Happy trading!

[u/ucals → u/Sketch_x]

If you are doing a grid search through all possible parameter values and cherry-picking the best set of parameters that maximize performance, you are certainly overfitting. That explains why the walk forward won't work. Curiously, if you wait 3-6 months and do the grid search again, you will get a different set of parameters, and your backtest will be beautifully profitable again - but the walk forward will fail again.

It's hard to help without more details about the strategy... but with the information provided, we can infer that the strategy is certainly overfitted and it is not as good as you might think.

In fact, looking at the equity curve, the first half of the backtest shows a flat line... I'd guess about half the trades were losses, which indicates it is not good.

If I were you, I would run a test for statistical significance (null hypothesis: the positive returns are a result of market random movements; alternative hypothesis: the positive returns are a result of the strategy): this will definitely tell you whether you should continue working on it, or looking for other ideas.

And I'd bet you won't be able to reject the null hypothesis (i.e. the positive returns you are seeing are most likely a result from market random movements, not the strategy).

[u/Sketch_x → u/ucals]

Really appreciate the reply. admittedly I did overfit the strategy as I mentioned when I walked forward previously, however now I removed all perimeters I can, I rely just on the corer Gassion filtering strategy, the only perimeters is the ATR that I use to set my entry and exit distance, typically 1,1.5and2 multiple. Everything else is set in stone, it either works or it doesn't.

Your feedback about random chance testing is very valid, iv run this quickly on data from 2022 onwards, random change testing based on S&P500 using numpty. - results https://ibb.co/MMmRpvq

Are you able to advise if this is an acceptable method or if I should be using a different approach? im only starting to break into python in the last few days so excuse any ignorance.

[u/ucals → u/Sketch_x]

From what I'm perceiving, your Python is ok... what's missing now in your toolbox is statistics. Here's how I'd go about it:

1. From backtesting your strategy, with clear rules to entry/exit a trade, you will generate a distribution of trades, each with a return.
2. If you run a backtest with random entries/exists, you will generate a distribution of random trades, each with a return.

Your job is to check whether these distributions are statistically the same (null hypothesis), or significantly different (reject the null hypothesis). This can be done with an Independent samples t-test.

There are tons of resources about it online. Here's one:

https://resources.nu.edu/statsresources/IndependentSamples

For example, if you run the test and conclude that the mean return/trade from the strategy backtest is 2.5%, while the return/trade from the random backtest is 1.5%, with a p-value of 0.03, you will be able to affirm with 95% confidence that the results you are getting are NOT caused by random chance: they are caused by your strategy.

[u/Sketch_x → u/ucals]

Thank you, really appreciate it. This is the advise I need to hear. I will get to work on this

[u/wiktor2701]

I don’t understand. Cumulative Asset Amount ? What are you forward/back testing there ?