A look At Optimization

To the new systems developer one of the most exciting things to play with is optimization. Optimization is using the power of the computer to examine every possible sequence of parameters and rules to find those that have worked out best in the past. With enough computer crunching power its possible to find systems that perfectly “predicted” the past. We can run number crunching PC's on automated routines and have them analyze billions of bits of data while we are sleeping! Many traders do this long enough and eventually "discoverer" the holy grail of trading systems. They jump into the markets with their new super predictive algorithms only to find they fall apart in real trading!

“What happened?” they ask themselves. The answer is that what they created was likely a system that was a statistical coincidence (known as a "curve fit"). Curve fitting is where a system has been optimized to a unique set of historical data. The problem is that the markets will behave much differently in the future than the past, therefore, a “perfect” trading system in the past could be useless in the future. For example, your computer finds the perfect dates in the past to have bought and then sold the market. Obviously this data does not mean anything in the future. . This is a simple example but most curve fits are some complex form of this basic concept.

Lets look at another flawed example. Assume we wanted to optimize a set of nickels that were most likely to land on heads. What we could do is flip a million nickels and only select those that landed on heads. Then, we can take those remaining nickels and flip them again, once again only choosing those that land on heads. We could repeat this process over and over again each time only choosing those nickels that land on heads. At this point we might conclude that we had narrowed down our nickels to only a small handful that were “optimized” to land on heads. We could then go out and make large bets with those nickels putting all our money on heads. We would quickly make a fortune right? WRONG!

Unfortunately we would very quickly lose our money. These nickels were not optimized for heads; they always did and always will have 50/50 odds. What might have confused some is that they thought they had found a predictable set of nickels when in fact they had just found a statistical coincidence!

Because there is so much data and so much computing power available, these kinds of errors find there way into trading systems all of the time. One of the worst offenders of such flawed optimized systems can be neural networks. When developing a system its imperative that optimizing is avoided as much as possible. You need to find NON curve-fit robust systems. There can be a place for certain types of optimizing, but it must be handled correctly.

What Is A Robust Trading System?

With computer’s as powerful as they are today its easy to optimize a trading system and make it look exceptional. However, as we’ve already discussed, an optimized system is not a good system. Just simply because you're able to train your computer to have 20/20 hindsight does not mean it will perform anything like that in the future (neural networks etc.).

The primary problem with optimizing past performance is that markets change in the future. A low volatility market suddenly becomes a high volatility market. A market prone to trends becomes a choppy directionless market. A market that had high leverage has it margin changed and now it has low leverage. A regulated market suddenly becomes unregulated. The list is endless.

What tends to happen is that market X will tend to start acting like market Y and market Y will tend to start acting like market Z etc. If you have perfectly optimized the system to trade market Z then you will be in trouble when it starts to trade like market X! This is a problem with many systems, especially stock index systems that tend to be optimized to a single market or sector. In spite of their occasional awesome looking results there's a drop of poison in their mix.

Contrast the previous scenario with one in which the system has been designed to work well most all markets A thru Z. Now, it does not matter if market Z starts to act like market Y or market A starts to act like market P etc.. They can change as many times as they want because the system has been designed to be universally robust with most ALL the various markets! Once again, the market characteristics can reshuffle themselves countless times and your system is like a Swiss Army knife that has proven in historical testing it can deal well with most all of those scenarios.

There are a few tip offs to an optimized system.

1. Unrealistically good looking performance
   
2. Only trades one market or sector well
   
3. Uses different rules (algorithms) for each market
   
4. Uses different inputs for each market even if the rules are the same
   
5. Uses different rules or inputs for initiating buys vs. sells
   
6. Does not factor in realistic transaction costs (slippage & commission)
   
7. Uses money management methods that don’t include market normalization (like single contract performance only)
   
8. Uses static numbers for all markets like a $2000 stop or $5000 profit target (some markets could hit those in an hour and others could take weeks). This may seem to contradict #3 but it does not. Its ok if markets have different stops and targets etc. as long as they were all dynamically computed from the same algorithm and inputs (as opposed to a static predetermined number across the board).
   

An important feature of a robust system is that it should weight every market equally. The testing should have been done in such a way to “normalize” the difference between the markets. For example, natural gas changes an average of a few thousand dollars per day per contract, however, Eurodollars change an average of a few hundred dollars a day per contract. You need a way to balance and normalize this difference in testing.

The reason you need to do this is because what if the system meets most of the above non-optimized rules. BUT, its trading one natural gas market contract for every one Eurodollar contract? The system would look great if in the past it had a lot of natural gas winners. However, what it in the future natural gas starts to have a lot of losing trades and the Eurodollar starts to have a lot of winning trades? Do you think a number of hundred dollar winning trades in a single Eurodollar contract are going to be enough to offset a number of THOUSAND dollar losing trades in a single natural gas contract?

If your trading 20 markets its because you want diversification. However, if your trading them all on a single contract basis then your not really diversified. You might have 50% of your portfolio accounting for 90% of the profits and losses! The problem is that moving forward, you will be dependant on specific markets instead of just a certain percentage of the markets (regardless of which ones). Its far more robust not to be dependant on certain markets within the portfolio. No one market should be any more meaningful than any other one.

In summary a robust system should do the following

1. Trade a large portfolio of markets successfully
   
2. Trade that large portfolio successfully over a very long test period
   
3. Use the exact same rules for every market
   
4. Use the exact same input values for every market even if the rules are the same
   
5. Have the exact same logic and input values for initiating both buys and sells
   
6. Factors in realistic transaction costs (slippage & commissions)
   
7. Be tested in a way where the markets have been normalized for risk (not single contract)
   
8. Doesn't use Static preset exits for all markets IE $2000 stop or $5000 profit target for all markets, but rather dynamically computed ones.
   

After you have done all of this, the final step would be to do some walk-forward testing. This means, test and develop your system on data up until year 2000 (for example). Then after all the testing is done see how it would have done from year 2000 until now etc. This helps avoid a lot of the benefit of hindsight. All of these are things we have done in the development our systems.

Feel free to email or contact us with any questions or comments on this subject. dhoffman@traderstech.net

Worst Case Analysis

Somebody once asked me “if there was only one performance report you could look at to make a decision about a system what would it be?” My first reaction was that this was a silly question. There are lots of factors that must be considered when choosing a trading system. There are countless performance indicators and ratios. Things such as average yearly return, maximum drawdown, Sharpe ratio, margin requirements, robustness, the list is very long. However, there has indeed been one report that I have come to rely on more than any other report. It’s a report that has given me more comfort and confidence as a trader than any other report. In fact, if I knew a system was properly created I could almost use this report alone to make a decision about trading it! So what is this report you ask? It’s called a “Start Trade Report”.

In my opinion, the start trade report gives you the most robust three dimensional view of a system possible. It cuts through so much of the pitfalls in traditional analysis. It even cuts through the nonsense involved in looking at real time performance. You may be thinking “wait a minute, how can you argue with real time performance?” Let me give you an example with one of my systems Synergy. In May of 2003 Synergy initiated a trade in London Copper. The trade became the most successful trade of the year. As of this writing, (March 7th 2004) the trade has profits of over $25,000 per contract. If you were using position sizing you might have had on 2 or 3 (or more) of these contracts. However, had you started a week or even a day after this trade you would have missed it! Therefore, two investors trading the exact same system with the exact same amount of money and the exact same money management rules could have a $25,000 or $50,000 or $75,000 (or more) difference in their account! AND they might have only started one day apart. This can create tremendous confusion because one broker’s real time accounts could be far different than another brokers real time accounts both trading the same system.

This phenomenon can also be used for disingenuous purposes. It’s possible for a systems vendor to cherry picked an optimal historical starting date. He can choose a date right before a huge winner (or series of winners). This can make it look like the system needed very little original starting capital and that the return on invested funds was enormous. Basically you financed trading with your initial winners. However, what if you had started on a different date? What if you had started on a date that was right before a series of losers? You might have needed 2 or 3 or 4 times the starting capital than you would have starting on a different date. Therefore, the return on invested capital would have been much less, or, you might have lost all your investment before making the profits shown.

Even if a broker or vendor shows an average of a number of accounts this can still be a very limited view. They could still cherry pick the 3 or 4 accounts and their different starting dates. Or they could have so few accounts to average from that the data suffers from what statisticians call a small sample size. Basically this means not enough data to draw any valid conclusions.

The worst offender would be if a disingenuous brokerage or vendor was pushing a day trading system because of the high frequency of trades and commissions it generated and then used some cherry picked “real time” accounts to “prove” that it worked.

The point I am basically making is that thee are countless ways that start dates can impact performance both in hypothetical reports and also “actual” real time performances.

You need to have something far more robust.

What’s the answer you ask?

Well, in my opinion it is the start trade report. What the start trade report does is tests a system hundreds or thousands of times over the given period. Each test it starts on a new date that coincides with a date that you could have taken a new trade. If there were 2000 trades over a 10 year period then it will retest the system 2000 times starting on the date of each new trade every time. In addition, it resets the equity back to the original starting amount with each test. This is important because when using position sizing you may skip certain trades in the beginning when the equity is small. It’s not correct to look at the results of trades that you would not have taken. I have sometimes seen brokerage firms report on trades my system made that many of my clients would not have taken based on their account size. For example, a $3,500 losing trade in a system where most clients would have skipped any trades with risk above $2,000. The start trade report knows to skip trades at the right time based on your starting amount. This report can also let you evaluate performance based off of the margin required. What this allows you to do is see ALL the outcomes, instead of just one.

A few things a Start Trade Report can show you are:

1. What percentage of the first 12 months were profitable over 2000 different starting dates?
2. What was the average first year performance when averaged over 2000 different starting dates?
3. How much money did I need in my account if I started on the worst possible date?
4. What was the average account size I needed to trade the system over 2000 different starting periods?
5. What was the average and the most I ever went under my original starting amount? (This is different than maximum drawdown)

The list goes on and on. This report allows you to filter out so much of the garbage seen in typical performance reporting. It filters out so many of the errors in reporting “real time” performance based on a limited sample size or “cherry picked” starting sates and accounts.

Hopefully you can see that this report is invaluable. I honestly don’t know how you could ever trade any system without it. You can see how much comfort and confidence this can build when you have looked at a system in this much detail. When I first started trading for myself this is he report that gave me incredible peace of mind. It was the only report that comforted me when there were drawdowns. It allowed me to know whether or not we were in the normal ranges of the bell curve. It also gave me a very realistic range of outcomes to expect within the first year of trading.

We feel that providing you these reports gives you an incredible edge and builds confidence. You need this confidence when the inevitable drawdown comes. In my own personal case Im able to remain very calm during those times because of these reports. To get a copy of our start trade reports please email us.

Source: www.traderstech.net

Why Trade Systems?

I am often amazed at the unbelievable amount of worthless investment advice offered. The reason I say that is because the vast majority of advice offered seems to be nothing more than opinions based on little to no real scientific research. Its similar to the countless “cures” that you can find in health food stores. Nothing more than untested and unsubstantiated claims.

My opinion on this continues to get stronger and stronger over the years. I have tested countless methods that the authors purported to be “money makers” only to see the vast majority of these ideas being nearly worthless. Many of these authors are very respected and have well established followings. In some cases they have near celebrity status!

A good example of this is Chart Patterns. There is an entire industry built on chart patterns. Things such as triangles, pennants, rising wedges, trend lines etc etc. Unfortunately, if you ever try to code these patterns you will first find that patterns are extremely subjective. A hundred people looking at the same chart will see 100 different patterns. In addition, most “obvious” patterns are only obvious after they have confirmed themselves. This is nothing more than hindsight bias. When you put the rubber to the road and actually test many of these “chart patterns” they don’t hold up at all. I find the same true for things such as Gann Lines, Fibonacci numbers, Elliot wave, the list goes on and on.

Think about this, you wouldn’t put a powerful drug in your body that was not first rigorously tested in a scientific manner would you? Why would you expose your financial health to ideas that were not also rigorously tested in a scientific manner? It’s amazing the financial advice and "tips" people will take based on the flimsiest of evidence! This was hammered home recently with all the multimillion-dollar salaried Wall Street analysts who lost their investors billions. In some cases this was disingenuous advice, in other cases it was just stupid predictions based on untested theories about what “should” happen. "Looks like a cup and handle base"????

What I am suggesting is that the ideas you use should be ones that have gone through rigorous scientific analysis. Ideas that have been broken apart in hundreds if not thousands of different ways and then tested across thousands of different examples to see how they would have performed. Furthermore, trading good systems can help to reduce the psychological sabotage so prevalent when trading without a mechanized method. The demons of fear and greed and panic and euphoria can be kept in check better.

The following text was taken from a popular trading book, Decision Traps, by J. Edward Russo and Paul J.H. Schoemaker. In this book, nine different types of decisions were tested using three different decision methods. The accuracy of the decisions was then compared and analyzed for effectiveness in predicting final outcomes. The investigator looked at different types of decisions, predicting grades, predicting recovery from cancer, performance of life insurance salesmen, as well as predicting changes in stock prices. He used three different decision making processes: an Intuitive Prediction Model, a Subjective Linear Model, and an Objective Linear Model.

Decision Models

• Intuitive Model = Discretionary Trader
• Subjective Model = Technical Trader
• Objective Model = Systems Trader

Intuitive Prediction Model (Discretionary Trader)

Intuitive prediction is defined as making a decision without the use of any objective or quantifiable data. For instance, in trying to predict the academic performance of graduate students, the researches asked their advisors to do so without seeing their grades and just by talking to them. The decision-makers had to rely on their intuitive impressions and any other factors they thought relevant (how the student dressed, their language skills, grooming habits, etc.). This is the same way discretionary traders make trading decisions - using intuition and gut instinct. In predicting the stock prices, it is highly likely that the researcher engaged a discretionary trader to predict the future prices of stocks.

Subjective Linear Model (Technical Trader)

A Subjective Linear Model is a much more complex decision making process. It starts with the interviewing experts in a field and learning how they make decisions. The researcher literally asks the expert how he or she makes decisions and they respond by explaining how they make their predictions. Although these experts are not using quantifiable data, they have enough experience and knowledge in their field to be successful. This decision making process is then outlined by the researcher.

For instance, a physician, highly experienced in treating cancer, has probably become fairly adept at predicting the life expectancy of his patients, even without using any objective data.

The researcher interviewed the physician and attempted to determine exactly how the physician made this assessment. Then the researcher put this newly quantified data into a regression model and attempted to predict the life expectancy of cancer patients.

This is very similar to how a technical trader makes decisions. He goes to seminars and reads books to learn how the experts make decisions using technical indicators. He then takes what he learns and attempts to trade like he experts. In a sense, he does his own regression model of the expert's process to make trading decisions.

Objective Linear Model (System Trader)

For the Objective Linear Model, the researcher developed an objective model based on historical tests and observations to predict results. This is defining and using quantifiable data, running historical tests, and then using the results of the tests to predict future outcomes.

For instance, the researcher would look at reams of physical data from terminal ill patients, and correlate the data with how long the patient lived. After running the historical tests, the researcher would then obtain the physical data form cancer patient, and using the historical test data, attempt to predict how long that cancer patient will live.

This is exactly what a system trader does. He runs historical tests and then uses that data to take a position in the market. He uses objective quantifiable data tested historically to make his trading decisions. The following table shows the results of tests.

In every case, the Subjective Linear Model outperformed the Intuitive Prediction Model bit only by a small margin. If you look at predicting the changes in stock prices, the Subjective Linear Model only slightly outperformed the Intuitive Prediction Model.

The real insight from this study comes when we look at the results of the Objective Linear Model. In every case, the Objective Linear Model outperformed both the Intuitive Prediction model and the Subjective Linear Model. In some cases, the improvement was minor, and in others it was substantial. It is interesting to observe that the greatest improvement came when using the Objective Linear Model in predicting the changes in stock prices!

Hopefully I have caused you to think a little bit about using thoroughly researched systematic methods in your trading (if your not already). My nearly 12 years of computerized research has given me the opinion that roughly 90% of the methods in the public domain (books, seminars etc.) has little to no real value.

Unfortunately, many charlatans and snake oil salesmen have used the guise of computerized research to sell equally worthless creations. The real key is learning how to use the power of computerized research to develop robust trading systems without falling into the many possible traps including optimization.

All of our systems have been rigorously tested and researched in a manner that we feel would meet the standards of the most knowledgeable and successful system traders.

Feel free to email or contact us with any questions or comments on this subject. dhoffman@traderstech.net

A study by Harvard Business School professor John E. Lintner found that including futures in an investment portfolio "reduces volatility while enhancing return". And that such portfolios "have substantially less risk at every possible level of return than portfolios of only stocks or only stocks and bonds".


Source: www.traderstech.net