Hkdfsa’epie/.

we’’

74987329klglgk;skd

Hjgfkjhlkj;fd;lkf

djglkjgl

the returns from the unpermuted and the permuted

markets is the Skill component, which cannot exist in permuted
data. In other
words, Equation 10 below represents the average total return
obtained from
training on permuted markets.
As a point of interest, look back at Figure 16 here. That figure is
based on profit
factors, while the equations just shown are more applicable to total
return.
Nonetheless, the principle is the same for either measure of
performance. The
bars in this histogram represent the values of Equation 10 above
(plus the single
observation for the unpermuted market, which is also used to
compute the
histogram). The red vertical bar is the value represented by
Equation 9 above.
We can thereby see the effect of Skill on performance relative to
the
performance of the trained trading systems when Skill is not
involved. If the
Skill component is significant, we would expect the vertical red bar
to be on the
far right side of the histogram, pushed there by the ability of the
trading system
to find authentic patterns in the market. Conversely, if the Skill
component of
performance is small, the red bar is more likely to be somewhere in
the interior
mass of the histogram.
Additional information can be gleaned from the permutation
training by
analyzing Equation 10 in more detail. When the trading system is
trained by
maximizing performance on a permuted dataset, the training
algorithm will
attempt to simultaneously maximize both components of Equation
10: it will try
to produce an unequal number of long and short trades (assuming
that the user
allows unbalanced systems) in order to take advantage of any long-
term trend in
the market(s), and it will try to take individual trades in such a way
as to
capitalize on any patterns it finds in the data. Of course, since it is
operating on
randomly permuted data, any patterns it finds are inauthentic,
which is why the
Skill component plays no role.
Now suppose we have a system that makes only long trades, and it
does so
randomly; individual trades are made with no intelligence
whatsoever. Equation
11 shows its expected return:
In this equation, BarsLong is the number of bars in which a long
position is
signaled, and TotalNumberOfBars is the number of bars in the entire
dataset
whose performance is being evaluated. This ratio is the fraction of
the number
of bars in which a long position is signaled. Looked at another way,
this ratio is
the probability that any given bar will signal a long position.
TotalTargets is the
sum of the target variable across all bars. This will be positive for
markets with
a long-term upward trend, negative for those with downward trend,
and zero for
a net flat market. So if our system is without intelligence, with
signals randomly
given, we will on average obtain this fraction (the fraction of time a
long signal
is given) of the total return possible.
A similar argument can be made for a short-only system, although
the sign
would be flipped because for short systems a positive target
implies a loss, and
a negative target implies a win. Combining these two situations
into a single
quantity gives Equation 12, the expected net return (the Trend
performance
component) from a random system that contains both long and
short signals.
When we train a trading system using a permuted market, we can
note how many
long and short trades it signaled and then use Equation 12 to
determine how
much of its total return is due to position imbalance interacting
with long-term
trend. Anything above and beyond this quantity is training bias due
to the system
learning patterns that happen to exist in the randomly shuffled
market. This
training bias can be estimated with Equation 13.
It would be risky to base a Bias estimate on a single training
session. However,
as long as we are repeating the permutation and training many
times to compute
the probability of the observed return happening by just good luck,
we might as
well average Equation 13 across all of those replications. If at least
100 or so
replications are used, we should get a fairly decent estimate of the
degree to
which the training process is able to exploit inauthentic patterns
and thereby
produce inflated performance results.
Two thoughts are worth consideration at this point:
• TSSB allows various BALANCED criteria which force an equal
number of
long and short positions in multiple markets. In this case, Trend will
be
zero.
• Powerful models will generally produce a large Bias, while weak
models
(not necessarily a bad thing) will produce small Bias.
Now that the Bias can be estimated, we can go one or two steps
further.
Equation 9 can be rearranged as shown in Equation 14 below.
This equation shows that if we subtract the Bias estimated with
Equation 13
from the total return of the system that was trained on the original,
unpermuted
data, we are left with the Skill plus Trend components of the total
return. Many
developers will wish to stop here. Their thought is that if a market
has a longterm
trend, any effective trading system should take advantage of this
trend by
favoring long or short positions accordingly. However, another
school of
thought says that since deliberately unbalancing positions to take
advantage of
trend does not involve actual trade-by-trade intelligent picking, the
trend
component should be removed from the total return. This can be
effected by
applying Equation 12 to the original trading system and subtracting
the Trend
from the UnbiasedReturn, as shown in Equation 15 below.
This difference, the Skill component of Equation 9, is often called
the
Benchmarked return because the NetExpectedReturn (Trend component)
defined by Equation 12 can be thought of as a benchmark against
which to judge
actual trading performance.
These figures are all reported in the audit log file, with p-values for
the profit
factor and total return printed first. Here is a sample:
Net profit factor p = 0.0600 return p = 0.0400
Training bias = 52.3346 (67.1255 permuted return minus 14.7909
permuted
benchmark)
Unbiased return = 55.4320 (107.7665 original return minus 52.3346
training
bias = skill + trend)
Benchmarked return = 39.8372 (55.4320 unbiased return minus
15.5947
original benchmark = skill)
The Training bias line is Equation 13, with the term benchmark
referring to
Trend in the equation, and these figures averaged over all
replications. This
figure (52.3346 here) is the approximate degree to which the
training process
unjustly elevates the system’s total return due to learning of
inauthentic patterns.
The Unbiased return line is Equation 14, the actual return that we
could expect
in the future after accounting for the training bias. This figure
includes both the
true skill of the trading system as well as the component due to
unbalanced
trades that take advantage of market trend.
The Benchmarked return line is Equation 15. This is the pure Skill
component
of the total return.