Chapter 4
PREDICTOR TESTING METHODS
The Predict Predictors ors
Parallax has been producing neural-network based financial predictors since 1990 and so it is an integral part of our business to validate these predictors using reliable statistical methods. With each predictor, we need to answer the following set of questions: 1. What price price behavior behavior is being being predicted? predicted? 2. What is the the effective effective duration of of the predictor? predictor? 3. How statistically significant is the predictor at each time step forward? 4. Is the predictor effective on all time scales? 5. Is the predictor effective on all financial series? 6. What combinations combinations of predictors predictors are the most effective? effective? The following following section section is an overview overview of a simple and reliable statistical statistical method and an 8-year analysis of our present stable of predictors. In order to carry out this analysis, it is necessary to measure the price action during the time period immediately following each prediction event. We call this the “post-predictor” or “outrun” period. 74
Post-Predictor Z Scores
We are interested interested in characterizing characterizing the post-predictor post-predictor time period using a scaleindependent method that allows comparisons between financial series. The “Z Score” is the most appropriate measure for this job. A Z-Score is the measure of how many standard deviations price has moved away from its price at the prediction event, assuming that the probability of either an up or down move is random at 50%. By measuring local volatility at the prediction event, a normal probability distribution can be drawn going forward in time that acts as a roadmap for subsequent price moves. prediction event.
The map is centered at the closing price of the
Each day the map widens according to normal diffusion
velocities, velocities, which are proportional to 1/√ 1/√ Time, representing representing the region where the future price is most likely to be found. An example of such a probability map is shown below for the stock Home Depot on Feb 3, 2005:
75
Viewed from the side at two time steps, diffusion acts to spread out the region where we might find find the stock stock price price as time elapses:
Diffusion causes expected prices to spread out over time
At each time step, there is a larger standard deviation and the same mean. If we represent the actual price achieved at each step in terms of that standard deviation, we produce a series of Z-Scores.
For example, if the price at a
prediction event is $5, and then moves to $7 on day ten with a standard deviation of $1.60, then the Z-Score on day ten would be (7-5)/1.6 = 1.25. This means that price moved 1.25 standard deviations above the price at the prediction event. Since the standard deviation continues to increase, in order to maintain the same Z, price would have to increase by the same relative amount.
76
If all post-predictor prices for all financial series are converted to Z-Scores, and if the predictors do not work, and if markets are random, then at every time step, a histogram of all the Z-Scores would be expected to result in a “standardized” normal distribution, with mean of zero and a standard deviation of one N(0,1) as shown below:
We of course hope that our predictors predictors actually predict non-random non-random price behavior, so the degree of deviation from the normal curve is critical.
Are Financi Financial al Series Series Random? Random?
We have assumed that price movement movement is best characterized characterized by a random walk model, but this might not be the case. There is gathering evidence evidence for other characteristic distributions such as a Biased Random Walk, Truncated Levy Flights, or Cauchy distribution.
Our solution is to produce a quantitative
background distribution based on randomly selected dates across our entire 3000 day test period, and for all of the 2500 stocks being considered. The figure below 77
shows this background distribution of Z-scores corresponding to random prediction dates, and a normal distribution based on a random walk assumption plotted together. It is clear from this figure that a strict random walk assumption is inappropriate during our 8 year test period.
Instead there appears to be a
positive return bias.
To illustrate this another another way, we could ask what percentage of buy predictions predictions are winners (Z>0) if the timing is random, and plot this percentage each day following the randomly selected purchase dates. Normally this would be 50%, but since the background distributions is positively biased, the figure below shows the percent winners for randomly selected buys climbs steadily over time. The reverse reverse is of course true true for randomly randomly selected selected sells sells (mirror image).
78
On a weekly scale the effect is even more pronounced, as shown in the next figure. I guess this could be called the dartboard effect, in that even a random selection of stocks during this period showed a 59% win rate at 30 weeks after purchase, and shorting was decidedly unprofitable.
79
We will use use these curves as as benchmarks benchmarks for our predictors predictors over over this test period and stock set. In order to have a non-random buy predictor then, we need to see a win rate in excess of 59% at 30 weeks for example, and sells would need a win rate better than 41%. So far we have examined the background distribution of Z-Scores and the percentage of winners. What we still need to know is how much gain is possible from randomly selected buys and sells, so that each predictor’s excess gain profile can be evaluated. The pictures below show the background performance for randomly selected buys and sells:
80
81
Parallax Predictors
Using the randomly selected background distribution we found in the last section, we will now statisticall statisticallyy evaluate evaluate our main stable stable of predictors predictors using a chi-squared chi-squared test. The Chi-squared test is designed for comparing distributions and a p<0.01 constitutes a rejection of our predictor distributions as random. We will also look at the excess %winners and excess %gain versus the random benchmarks shown in the last figure. The following is a list and a qualitative description of each predictor: Predictor ExtremeHurst™
Extension (high)
ExtremeHurst™
Extension (low)
ExtremeHurst™
Compression
Price Wizard™
(Undervalued)
™ Price Wizard ™
(Overvalued)
Precision Turn™
(high) Precision Turn™
(low)
Description
Duration
Measures the degree of positive feedback present in price advances and predicts when the critical point has been reached. Price behavior in the post-predictor period is expected to be flat to down. This is an end-oftrend predictor.
Estimated at 2030 bars – bars can be any scale provided sufficient liquidity Estimated at 2030 bars – bars can be any scale provided sufficient liquidity Estimated at 2030 bars – bars can be any scale provided sufficient liquidity Estimated at 6 months to a year
Measures the degree of positive feedback present in price declines and predicts when the critical point has been reached. Price behavior in the post-predictor period is expected to be flat to up. This is an end-oftrend predictor. Measures the degree of negative feedback present in price and predicts when the critical point point has been reached. reached. Price behavior behavior in the postpredictor period is expected to be characterized by a significant volatility volatility increase increase and high trend persistence. persistence. This is a new trend predictor. Individual stock prices are found using neural networks by simultaneously weighing all major stores of fundamental value in the context of industry, sector, and economy. If the determined price is less than the actual price, then the stock is undervalued and considered a buy Individual stock prices are found using neural networks by simultaneously weighing all major stores of fundamental value in the context of industry, sector, and economy. If the determined price is more than the actual price, then the stock is overvalued and considered a sell Individual trend change cycle dates are found using a method which is blind to high/low, but sensitive to the degree degree of trend change. Sells are determined using a local Hurst measure to confirm an uptrend into the turn date Individual trend change cycle dates are found using a method which is blind to high/low, but sensitive to the degree degree of trend change. Buys are determined using a local Hurst measure to confirm an downtrend into the turn date
82
Estimated at 6 months to a year
Estimated at 6 days
Estimated at 6 days
ExtremeHurst™ Extension Buy & Sell Predictors
Our ExtremeHurst extension predictor is Parallax’s most predictive tool. It measures the degree of positive feedback present in price and predicts when the “end-of-trend” critical point has been reached using the principle of discrete scale invariance. Price behavior in the post-predictor period is expected to be flat or opposite of the preceding trend. On a daily scale, we have accumulated Z-Scores for all 30 days following the signal. The distribution of these Z-Scores on day eleven, shown in the figure below, shows the strong positive bias, with 60% of the returns being positive by day eleven, with a p-value = 0.00000005. This means that these are not random signals, and that market behavior leading up to the prediction event “conditioned” what happened in the post-prediction period. ExtremeHurst is scale invariant by definition, so this same effect should be present on weekly, monthly, or intra-day periods as well.
83
The ExtremeHurst sell predictor is extremely fast, showing 56% winners at only 4 days from the signal, with a Chi squared test p-value = 6.7x10 -124. This means that these are not random signals, and that market behavior leading up to the prediction event significantly “conditioned” what happened in the postprediction period.
ExtremeHurst is scale invariant by definition, so this same
effect should be present on weekly, monthly, or intra-day periods as well.
84
Looking at the percentage of winning predictions for both ExtremeHurst™ predictors compared to the background rate yields the following picture:
The way to interpret this picture is that we gain a significant significant advantage advantage over random for both buy and sell predictors for at least 30 days. Extension buy signals yield the best percentage of winning predictions at 8% more than the random background after 22 days. This tells us that if we had a large sample of trades that were triggered by this predictor, we could expect that at 22 days, a total of 55%+8% 55%+8% = 63% 63% of the trades would be ahead. Remember that our random trade sample had 55% winners at 22 days, so we would beat it by 8%. This chart also tells us that we can hold trades for a relatively relatively long time. Despite the win rate staying constant, average portfolio returns would climb steadily. 85
The sell signal is different and faster. We immediately achieve a 7% edge above random at 3 days and that edge diminishes steadily over 30 days. This might make us design a faster sell trading strategy than what we would use for the long side. So far we have examined the overall distribution of Z-Scores for ExtremeHurst, and the percentage of times that a buy signal goes up or a sell goes goes down. The win rate is only part of the story however. What we still need to know is how much gain is possible from these buys and sells in excess of the background market performance. The pictures below show a gain analysis for ExtremeHurst buys and sells. Each plot shows the median percentage gain with error bars at the 40th and 60th percentiles and the average of all returns. Note that the average is not as good a measure of performance because of the overemphasis on outlier signals. The count of signals is shown in parenthesis.
86
87
On a weekly scale, the power of our ExtremeHurst extensions becomes even more evident. Remember that 0% on the graph is equivalent to a random win rate, so any excursion above zero stacks the deck in our favor. Weekly scale extension buy predictions reach their maximum effect after 8 weeks, with an overall win rate that is 15% over the random background. Again we see that the extension sell signals are faster and give only a 2.5% edge over random.
88
89
Price Wizard™ Wizard™ Fundamental Value Predictor
Price is often out of sync with value, being bid up or down based on future expectations and irrational trend persistence. Our model incorporates all major stores of corporate value, normalized simultaneously for sector, industry, and economic factors. The figure below shows the Z-Scores 30 weeks following a stock moving to undervalued from overvalued. positive bias.
..
90
Note the significant (p=0)
91
92
93
Precision Turn™ Cycle Predictor
…..
94
95
96
