Options Backtesting with OptionStack
A few weeks ago some readers and I decided to explore doing our own options backtesting in order to challenge / validate some research by Tastytrade and explore our own ideas. Among the details, the question of how to test is on the floor for decision. Do we buy a dataset and manually crunch numbers, build a rudimentary automation solution or do we invest in an off-the-shelf automation tool such as OptionStack or ORATS?
The time investment to manually backtest complex strategies is high, as is building a custom automation solution. Thus, an off-the-shelf solution is of interest.
OptionStack seemed to be above the crowd and could be run in a browser so I started there. Using a demo account, let’s see how this tool performs. By the way, this is NOT a sponsored post.
Contents
First Impression
Upon launching the platform I was greeted with a quick-start dialog which contained a few helpful videos. After watching the two brief clips I had a general feel for how the platform worked and was ready to start exploring.
After the welcome screen is dismissed the first thing I noticed is an already-configured backtest ready for action, followed by several more pre-built backtests in the left-hand column.
The creators of this tool are familiar with Tastytrade. The opening configuration is called “TastyCoveredCall” and there is a dedicated folder of preconfigured backtests covering some of their strategies.
Backtesting
From Idea to GUI
It took a bit of finagling to get my idea into the tool’s GUI. For example, I had to figure out how to describe “open a trade every trading day” using the available logic constructs. I initially started with “Days Past Since Last Trade >= 1”, but that didn’t work because closing trades reset the timer and prevented new positions from being opened the same day other positions are closed. The solution was to use “Day of the Week is between 1 and 7.”
Another example, which I haven’t been able to solve yet, is to replicate trading every Monday, Wednesday and Friday. The logic to open the trades works but the backtester doesn’t seem to be capable of opening positions on Friday that expire Monday. It thinks no such positions exist.
draft of ERN’s trade logic, debugging inability to sell 3 DTE positions on Fridays
Backtest Idea
With the recent elevation of VIX I’ve been able to consistently sell puts 10% OTM or more with deltas in the 9-12 range, well above my typical target of 5. This got me thinking, should I place trades based on delta or % OTM?
We know that at order entry delta is an approximation of probability of profit (POP). Would a better approximation of POP be % OTM at order entry?
A 5 delta 45 DTE SPY PUT @ VIX 11.95 ~10% OTM
A 12 delta 45 DTE SPY PUT @ VIX 20.14 ~10% OTM
Asked another way, is 10% OTM a better approximation of 95% POP when accounting for VIX dynamics?
Let’s run a backtest and find out!
Setting up the Backtest
I populated the backtester with my trade logic and enabled / disabled the highlighted sections for easy toggling between tests. The transparent elements are disabled.
Why do I use a range for OTM and delta instead of a fixed single value?
If a position doesn’t match the criteria EXACTLY the backtester doesn’t place the trade. Hence, I needed a small range to work around the issue. The width of my range was chosen after empirical testing; there’s nothing special about the selection other than “it works.”
Configurations
Open 1 SPY put closest to 45 DTE daily with a max of 21 open positions:
- Delta 4.6-5.2, close at max profit 25% or 21 DTE
- OTM 9.8-10.5%, close at max profit 25% or 21 DTE
- Delta 4.6-5.2, close at max profit 40% or 21 DTE
- OTM 9.8-10.5%, close at max profit 40% or 21 DTE
Results
Here’s what the tool provided as “summary” data:
The date range is capped at ~1 year or less, within the dates above, due to my account being a trial account. Paid accounts are able to use the full date range, which goes back to 2011.
I found it curious the tool didn’t auto calculate the win / loss percentages and avg trade P/L.
The output files include:
- Log: informational messages such as “no trades met your criteria for trading day ‘x’”
- Statistics: daily values for: portfolio, net change, cumulative P/L
- Transactions: execution time, buy/sell, qty, fill price, instrument, comments
- Summary: the values in the screenshot above
- User Plots: values used to construct charts, in this case SPY P/L and portfolio return
Discrepancies
Using scenario 1 as an example, when I dug into the transaction log I found significant discrepancies and misleading interpretations compared to the summary file. In no particular order:
- Max Drawdown is not the same as max loss. Drawdown appears to be calculated based on unrealized P/L. Max position loss is nowhere to be found.
- Num. Winners is not accurate. With the exception of a single position, all were winners. I’m not sure how the depicted number is derived.
- Num. of Losers is not accurate; I’m not sure how this number is derived. As a hunch I suspected it was the count of times the portfolio encountered a negative combined unrealized+realized P/L, but that’s not the case as the statistics file shows 6 instances of negative values. A manual review of the transactions demonstrates there was only one losing trade for a total loss of $37.50 (ignoring txn fees).
- Average Gain and Average Loss are inaccurate based on the previous observations.
- Portfolio Return is therefore not accurate either.
- Start Date is accurate. I looked up the closing value of SPY on July 3 2017 and compared the strike price of that option. It checked out. That was the only spot check I performed on this attribute.
Not listed in the summary is the mechanic of plotting total risk throughout the life of the backtest. There is an explicit feature for this in the tool but it only seems to record values every couple of weeks as opposed to daily, which prevents doing risk management.
So, yeah…
Summary
The transaction log does seem to be a solid resource. Identifying the positions for use in running calculations is half the battle, which OptionStack does nicely (as long as it’s not opening a 3 DTE trade on Fridays). By using the transaction log one can calculate any number of [accurate] statistics themselves.
I pulled together my own “dashboard” after spending some time in Excel. Going forward I should be able to reuse the formulas and build dashboards (or more accurately, crunch the data behind the dashboard) much faster.
Based on available data, it appears using % OTM generates a greater return with less volatility than using delta to base trades. I’m curious how this would have played out during 2008.
For those interested in reviewing the data for each of the backtests, it’s available here. As usual, the file is read-only. You will need to make a copy to make any edits.
I know the options trading crowd is a small world; I don’t intend for this to be a bash session on a particular tool. If you’re involved with the OptionStack team and reading this, there are several other bits of feedback I’d like to provide ranging from UI elements and UX to additional data points to help improve the product. Feel free to reach out. I’m happy to be a [back]tester.
Comment References:
- Time to Reach FI Based on Savings Rate and Portfolio Allocation
- Using a Cash Buffer to Offset Sequence of Returns Risk
- Bonds Don’t Diversify a Portfolio
- Low SWR @ High Equity Allocation is Optimal over 60 Years
- Distribution of SWR Rates Across Asset Allocations @ 60-year Horizon
May 14, 2019 @ 10:26 am
Great work! That seems piss-poor at best… I can’t help but wonder if something global was overlooked that resulted in many of the discrepancies. OS should definitely have a look at this to eliminate that possibility. Regardless, it shouldn’t be overly difficult. It seems like you put a lot of time into this and I know you went at this objectively.
May 14, 2019 @ 8:44 pm
Thanks Mark. I didn’t do much homework on the company’s background, but I did hear they’re a newer player in the backtesting space. I applaud them on the GUI as it’s a reasonable implementation to overcome working with raw code. However, agree re: their results.
As a positive, I trust the transaction log is largely accurate based on the spot check I did on the start date position, so there’s still value there. If my expectations were set that they would only provide a transaction log and nothing more, I would have had a different opinion.
This is a prime example of “trust, but validate.”
May 16, 2019 @ 9:20 am
How do you plan to overcome a year like 2008.. All this option selling looks great in most years. I tried many different scenarios but almost all fail in 2008 type year. And there were some similar crashes before, so it is not like one isolated case.. Drawdowns in those years are huge.. So one has to either A – hedge with long put option, which east away profits or sell very far OTM which limits drawdown, but then also take 5 years to recover the loss.. Or if you play it safe and small drawdowns are not that high, but then also the returns are really not worth playing with.. If you can get buy and hold almost 7-8 % returns counting with dividends, is it worth it then to play with all this? I am wondering more and more myself..
Spintwig, I suggest you read Reel Ken articles. I think he came as close to the holly grail as possible. https://seekingalpha.com/author/reel-ken#regular_articles
He has many SPX ideas. One of them is to buy 1 or 2 years out ATM put option. Then divide this by say 50 weeks and try to get as much extrinsic value by seling IN THE MONEY weekly puts to recover put purchase. So if you pay 1.400 USD for 1 SPY ATM put, then just divide this by 50 = 28 USD per week. This means you participate in lik 95 % of the upside, only some small portion of the upside is sacrified for extrinsic collection to recover the put purchase value. Steadyoptions made a whole model out of this and their backtesting shows that this way you do not lag behind even in raging bull market and what is most important you have downside hedged! But not immediately, but after one year. This is not on / off strategy. You try to recover put purchase in one year not in one week or month. So some patience is needed. And to avoid a whipsaw after big sell-off and subsequent recover it is imperative that you hold level! Read some of his articles, they are a huge eye opener. They allow you to be fully invested and no try to guess and market time, be 100 % protected all the time and only give some modest upside potential from time to time while you can sleep like a baby. What is more you can lever this 50 %, this is what Steadyoptions did and get even higher returns..
Try to read those three at least and tell us what you think:
https://seekingalpha.com/article/4205227-risk-risk-cake-eat
https://seekingalpha.com/article/4210320-selling-puts-good-bad-ugly
https://seekingalpha.com/article/4212622-selling-puts-options-options
May 16, 2019 @ 1:30 pm
Overall the years would be managed identically: small frequent trades far OTM. I’d like to backtest over that range to see how a fixed delta and a fixed OTM target perform. Specifically, opening all trades at 5 delta vs opening all trades 10% OTM. As VIX increases a 5 delta position becomes farther OTM. Inversely, when VIX was ~80, an ATM position was already 7-8% OTM. It would be interesting to see how these two approaches play out.
When selling 5 delta puts 45 DTE, there’s no value in buying protection aside form lowering margin requirements. The cost of hedging a >= 2SD move is inefficient and is an accepted risk.
A key principle in my options trading is I’m not using S&P500 as the measuring stick of performance. My collateral is held in SCHB so I’m already getting 100% of the S&P500 returns. Selling options is done to generate income / lower my withdrawal rate (and for fun as I find the mechanics interesting and the overall approach an optimization problem to be solved). Any options trading I do in this account, as long as it’s a positive number, is a success in my book. Simplicity and low time commitment are also requirements.
You’re spot on that one either exposes themself to significant drawdowns or they collect a nominal return.
Consider the examples in my last visual: all of them made money despite the sharp drop in Feb 2018, and the losses were nominal (the trade log behind each strategy is also available for review in the linked spreadsheet).
The annual P/L relative to the portfolio was about 1% (I used a virtual 450k account so 1 SPY 5 delta position was ~15% of my 20-30% capital allocation target). Suppose I’m using a 3% SWR, now I can live off option income of 1% + the 2% dividends and never have to sell a share of my portfolio again. Alternatively, perhaps my lifestyle requires a 4.5% SWR. Generating an extra 1% makes the SWR a sustainable 3.5%.
Reel Ken is describing strategies to have put writing outperform SPY. It’s a completely different approach than what my strategy is aimed to accomplish. That being said, there are two considerations to his strategy I’d like to highlight:
1) using VIX as in indicator of “high volatility” is good in concept but poor from a systematic perspective. Why? We don’t know how high VIX can go at any given time. Only by looking at historical data can we identify what “high volatility” means; it’s not an absolute or “bounded” scale / variable.
2) Taxes. Reel mentions this, and it’s a significant factor. Using the buy/hold approach one can defer taxes perpetually by never selling or using margin loans.
May 17, 2019 @ 5:57 am
Thanks for reply. Totally agree with you. As for myself, I am searching for something similar. I managed to save about 150.000 EUR of capital which is substantial for me, I live in Slovenia. I just like you am able to live comfortably on about 1000 EUR monthly and thinking of relocation to Asia. But at the same time I am very risk averse. I could not sleep knowing that I could loose 40-50 % of my capital in a few months during bear market, because then I would need to work much longer.. So I am searching for strategy that could get me 5 % to 10 % return (10 % is pushing it, I know) with shallow drawdowns.. I know in the long run one can just invest in SPY and get about 7 % return which would be ok for me. It is sequence of those returns that I do not like.. Specially now after 10 years of bull market I think there is much higher chance of a big down year than big up year..
May I ask you why you do not sell calls at the same time? I know puts are much more profitable, but I think tastytrade did a lot of research and found out that selling 10 delta calls is profitable too. Delta is overstated and premiums are higher. It also smooths returns, specially during down years. This is what I am playing with now. I would go about 40 days out and sell 7 delta put and about 10 to 15 delta call. This would bring the same premium in (7 delta puts have about the same premium currently as 15 delta calls). Then I would just roll untested side. So if ES starts moving up, my calls are picking up deltas and increasing, but I would close the 7 delta call now something like 5 delta and open new 7 delta call. So start out would be somehow delta neutral and then I would roll the untested side up or down a few times and then just close it and wait for reversal. This I know can be risky because a reversal might never happen.. So I have to think this through 🙂
So my question is if you ever plan to sell calls too or just staying with puts?
And also why you have principal invested without any protection? I mean if market tanks like 2008 you loose on both fronts. On your buy and hold and on short put at the same time? In your case I think it would make more sense to hedge your long portfolio with 2 years out put and just sell short time puts to finance this protection. Not sure if I understand correctly but you can loose 50 % of your principal in a bear market which I think is substantial amount of money? Much more than you can get from selling 5 delta puts in years..
May 17, 2019 @ 10:43 am
You’re welcome, and that’s excellent! Great work getting to 150.000 and getting your living expenses down to 1,000.
I’m assuming you’re still in the accumulation phase and about halfway to FIRE using 1.000/mo as the goal. If you have a high savings rate it won’t matter what the market returns over the next few years as you work to finish building the portfolio. GoCurryCracker has a great post on exactly that topic with charts and all. https://www.gocurrycracker.com/financial-independence-how-long-will-it-take/. Trying to eke out additional performance over the SPY is a time-consuming and risky proposition. All the research and real-world work I’m doing is to see if I can safely get just 1%. The volatility incurred to get much higher increases rapidly and is unattractive for FIRE.
To offset the sequence risk I have a ~1.5 year cash buffer in a savings account earning 2.5% at Investor’s Bank (I just switched from Vio Bank per my last post on Financial Operations https://spintwig.com/financial-operations). Per part 25 of ERNs SWR series https://earlyretirementnow.com/2018/05/23/the-ultimate-guide-to-safe-withdrawal-rates-part-25-more-flexibility-myths/ highlights the cash buffer as a relatively inexpensive solution to hedge against steep downturns. I wrote about my implementation of this strategy. This is also the same mechanic Warren Buffett uses. In one of his news interviews he was asked what’s the purpose of 10% bonds in his 90/10 portfolio. His response: when equities take a beating, live out of the bonds until stock prices recover. Granted, ERN’s data suggests a 2.5x cash buffer and I’m at 1.5. I’m willing to take that risk but I’m also willing/able to hustle for additional cash flow if needed.
I don’t sell calls because 1) I’m depending on my portfolio’s upside and most SP500 performance comes from just a handful of very strong up days. Those will be lost by selling covered calls. December 2018 is a great example – in one day everything was up 6% and was part of the road to recovery from the bear market. Had I have sold calls I wouldn’t have made the upside and would still be underwater. In fact, most sharp declines have similarly brisk recoveries, historically speaking, hence my proposal in Options Trading Part 4 to BUY a 90 DTE call after a 10-20% drop. This leads me to 2) I haven’t found a long term strategy that’s profitable with calls. Yes, I can find a year or two where it works but long term it doesn’t. I know there’s a few strangles in my trade log but I intend to stick with only puts going forward.
There’s no need to protect the SCHB positions from downside. Downside is an accepted risk of holding equities. If I wasn’t comfortable with that I’d be holding a different asset class. As for the put selling, that’s why I’m interested in seeing if there’s an approach that allows for just a trickle of income and is so far OTM / low risk that any declines are nominal from a P/L perspective. Keeping in mind declines can be slow and steady, in which case I’m looking to see if theta decays faster than the market drops, and declines can be sharp, in which case I’m looking to see if using delta or % OTM is a better risk-adjusted method to capture elevated premiums when VIX is high. If the market tanks, I’ll live out of the savings buffer while I reinvest dividends and wait for the markets to recover.
May 17, 2019 @ 6:12 am
Also what I think you are forgeting is that withdrawal of 3 % is much different when you have 500.000 USD which makes is 15.000 USD, then bear market comes and halves your portfolio and now you only have 250.000 USD. So 3 % is now only 7.500 USD.. Dividends are 2 % and trading is 1 % but now that you have only half money you had before bear market all numbers are 50 % lower, or you must take more risk, sell 2x SPY where you before sold only one etc.. This is why I think your approach of having a colateral unhedged in stocks will fail catastrophically in bear market. Also this is the reason to have 50 % in bonds and 50 % in stocks if you must live out of the markets. Because 99 % of the time if stocks are down big, bonds are up big, keeping your portfolio value mostly with much shallower decline.. I think this CAN and WILL change in next bear market, because bonds are also high and we had like 30 years bull market in bonds, which makes it difficult to judge how it will be in the next bear market with rates so low. But if you look at recent big selloffs, TLT behaved like it always does, it shoots up..
If you sell puts only for additional income I think you really should have principal in bonds and stocks then to avoid big drawdown or to go Reel Ken style, purchase cheap 2 years out ATM put protection for your principal and get the money you spent on purchase protection back via selling weekly/monthly.. calls or puts.. And reel ken does not look at VIX, never found this in any of his articles. He just mechanically sells deep in the money puts with just enough extrinsic value to cover the long put protection. So over 50 weeks he gets all the premium he spent for protection back. This is perfect if you look at what he has done. He will not have any negative year, or maybe a few % top, but will participate in the 90 % of the upside.. If you average this out I think 10 % returns are quite achievable and you can sleep better assured your principal is protected. What do you think?
May 17, 2019 @ 4:27 pm
Sticking with the theme of a 50% equity decline (2008), bonds were down 10%. It turns out bonds are not a diversifier of portfolio risk, they’re a tool to sometimes allow non-equity assets to perform better than cash while having a somewhat similar risk profile of cash. The overwhelming reason why portfolios <100% equities are less risky is because <100% is allocated to a risky asset, not because bonds provide diversification. See: https://earlyretirementnow.com/2016/08/17/bond-diversification-is-a-myth/
As for withdrawal rates and asset allocations over a 60-year horizon, having a low withdrawal rate and 90-100% equity share is optimal. See: https://earlyretirementnow.com/2016/12/21/the-ultimate-guide-to-safe-withdrawal-rates-part-3-equity-valuation/
When factoring in sequence of return risk, see the section called "Distribution of Failsafe SWR" at https://earlyretirementnow.com/2017/09/13/the-ultimate-guide-to-safe-withdrawal-rates-part-19-equity-glidepaths/ It highlights different buy/hold and glide path approaches to the risk. We can see how different bond allocations perform and where the optimal allocation resides. In theory we could setup a 3.25% withdrawal rate on a 100% equity portfolio, never look at the market or earn money and have a 99% chance of the portfolio lasting for 60 years.
I can be persuaded if there's data that suggests otherwise on any/all points 🙂
None of this has any bearing on option selling. My put writing is an overlay to the portfolio.
I haven't dug too deep but a few of the elements in Reel's strategy raise eyebrows. For example: selling ITM puts as a way to "hold the strike". There's no mention of assignment risk, which is very real when selling a 70 delta (30% POP) put (his examples reference options on SPY). If using options on SPX then it's a non-issue due to the European-style options but a portfolio 10x larger will be needed to account for the 10x larger position sizing of SPX. 2008 would have required a substantial amount of capital while "holding the strike" and the cost of hedging over time (buying puts) would have had a serious impact on profits, possibly negating all of them.
Can a put writing be devised to match SPY performance? Probably, but it'll be far easier, cheaper and less time consuming to simply buy/hold SPY.
November 5, 2020 @ 9:11 pm
Hello,
THanks so much for this, I also just tested Optionstack and not really a fan.
What do you use for your options backtesting? I am currently in the market for that right now. Would really appreciate your input.
November 6, 2020 @ 8:26 am
Hi Peter —
I use a series of custom-built tools to perform backtesting. An alternative to OptionStack is ORATS (99/mo, no free trial), but it has similar limitations (read: unreproducible stats). For example, there is no specification of starting capital or margin utilization so it’s unclear how several of the stats are derived. Also, there’s no support for daily interest accrual on margin collateral, trade fee stats, premium capture, and other KPIs.