Report Group 05

Project Report

Bitcoin Price Prediction

Applied Statistics and Experimental Design

Author: Group 05
Tran Ngoc Khanh - 20200326
Nguyen The Minh Duc - 20204904
Nguyen Ngoc Toan - 20200544
Pham Thanh Nam - 20204921
Nguyen Hoang Tien - 20204927
Advisor: Assoc Prof. Nguyen Linh Giang
Academic year: 2022

Abstract
The year 2008 marked the birth of a completely new concept in the financial world - cryptocur-
rency. Cryptocurrency is a new type of digital currency so that all the transactions are verified and
maintained by a decentralized system, rather than by a centralized authority like the current financial
system. This development is done in conjunction with the blockchain concept and has had huge
potential growth in the future financial system. For this reason, the cryptocurrency market was born,
where blockchain projects can find investment, as well as users can put their trust in a decentralized
digital currency to perform transactions and investors can put their money to make a profit. Price
prediction systems that have been studied for a long time ago can be applied by investors and in-
vestment funds in this new market. Although this will be more difficult when the market is very
volatile and very erratic, cryptocurrency price prediction is still a very hot and interesting topic for
researchers. Our target is to implement some Statistical models, Machine Learning models and Deep
Learning models in the price movement prediction problem. Our study involves comparing all these
models and selecting the best model to make the final training strategy and evaluation. And the
result is: the best model is LightGBM and with this model, we obtained the average accuracy on the
test set of 55.262%, and the average AUC score on the test set of 0.56747. The corresponding values
of the validation sets are 54.658% and 0.55127.

Acknowledgment
We would like to express our sincere gratitude to Assoc. Prof. Nguyen Linh Giang for giving us
an opportunity to work on this wonderful project on Bitcoin price prediction. Throughout some
quick encounters with him, we wanted to say thank you for your valuable words and advice that
motivated us. We are also extremely grateful to Assoc. Prof. Than Quang Khoat who taught us
about Machine Learning this semester 20212. The knowledge that he taught us is indispensable to
completing this project. Preparing this project in collaboration with our teachers was a refreshing
experience.

1
Contents

1 Introduction 4

2 Methodology 5

3 Data Preparation 5
3.1 Data collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.2 Exploratory data analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.2.1 Label analysis (Close price) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.2.2 Feature analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.3 Feature engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.4 Feature scaling and normalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.4.1 Window normalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.4.2 Logarithmic scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

4 Models 9
4.1 Theoretical background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.1.1 Statistical models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.1.2 Machine Learning models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4.1.3 Ensemble methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.1.4 Deep learning models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.2 Practical Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.3 Evaluation and Error Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.4 Final Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

5 Conclusion 23

6 Scope for further research 23

A RAW DATA 26

B FEATURES AND INDICATORS 27

2
 Report Group 05

List of Figures
1 Global crypto market cap [7] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Top 10 Crypto Fund Manager Managers by Managed AUM (Swfinstitute) [8] . . . . . 4
3 Project pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
4 Bitcoin Close Price . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
5 Bitcoin close price 1st difference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
6 Close price trend seasonal component . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
7 Close price residual component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
8 First 100 close price seasonal component datapoints . . . . . . . . . . . . . . . . . . . . 7
9 Residual histogram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
10 Residual divided by trend histogram - 1 hour . . . . . . . . . . . . . . . . . . . . . . . 7
11 Residual divided by trend histogram - 1 day . . . . . . . . . . . . . . . . . . . . . . . . 8
12 Features correlation matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
13 Close price before difference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
14 The close price after first difference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
15 ACF and PACF plot of each difference of Close Price . . . . . . . . . . . . . . . . . . . 10
16 Returned series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
17 Summary of ARIMA model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
18 The random 300 data points of trainning set results . . . . . . . . . . . . . . . . . . . . 10
19 GARCH model summary of volatility . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
20 Confident interval on whole test set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
21 300 random continuous data points with confident interval . . . . . . . . . . . . . . . . 11
22 Odds and logit function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
23 The distance between the expectations and the sum of the variances affect the discrim-
inant degree of the data [13] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
24 Error with training rounds [17] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
25 Amount of say . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
26 Error with training rounds [19] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
27 The repeating module in a standard RNN contains a single layer [24] . . . . . . . . . . 17
28 The repeating module in an LSTM contains four interacting layers [24] . . . . . . . . . 17
29 LSTM notations [24] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
30 LSTM cell state line [24] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
31 LSTM gate [24] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
32 LSTM forget gate [24] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
33 LSTM input gate and new cell state [24] . . . . . . . . . . . . . . . . . . . . . . . . . . 18
34 Update cell state [24] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
35 LSTM output gate [24] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
36 GAN architecture [26] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
37 Generator architecture [26] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
38 Features importance barplot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
39 AUC-Iterations plot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
40 Train set score distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
41 Validation set score distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
42 Test set score distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
43 Validation set calibration curve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
44 Test set calibration curve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
45 Test score histogram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
46 Test set calibration curve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
47 Validation score histogram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
48 Validation set calibration curve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
49 Raw data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
50 Features and indicators - 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
51 Features and indicators - 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3
 Report Group 05

1. Introduction jumped into this cryptocurrency market. The

top 10 largest crypto funds have a total asset
The concept of cryptocurrency began in 2008,
under management (AUM) worth around $14.6
when Satoshi Nakamoto created the first cryp-
billion.
tocurrency named Bitcoin based on a peer-to-
peer version of electronic cash, which allows on-
line payments to be sent directly from one party
to another without going through a financial in-
stitution. He has provided a solution to the
double-spending problem of this technology [1]
and has been ushering in a New Era of Money
Technology [2].

This Bitcoin will be worth nothing if no one be-

lieves in it until someone realized how valuable
this technology could be when one person paid
10,000 Bitcoins for two pizzas delivered by Papa
John’s [3].

At this time, there were only two ways to ob-

Figure 2: Top 10 Crypto Fund Manager Man-
tain bitcoin — by mining it yourself or arrang-
agers by Managed AUM (Swfinstitute) [8]
ing a peer-to-peer (P2P) trade via a forum like
Bitcointalk, which Nakamoto founded to host
Bitcoin-related discussions [4]. When Bitcoin
The price movement prediction and price fore-
proven its usefulness, the need for exchanging
casting problems have been studied for a long
USD for Bitcoin increased, and the first Bitcoin
time ago, since the beginning of the finan-
exchange was born - Mt. Gox. This was a bit-
cial market quantitative analysis, which reaches
coin exchange based in Shibuya, Tokyo, Japan,
back to the early 20th century with the publica-
and it was handling over 70% of all Bitcoin
tion of the groundbreaking paper "The Theory
transactions worldwide by early 2014 [5]. Unfor-
of Speculation" by Louis Bachelier in 1900 [9].
tunately, this exchange filed for bankruptcy after
Quantitative analysts are the people who have
a series of hacks in 2014 that saw 850,000 Bit-
used the data to make investment decisions and
coins disappear – nearly 2% of the total amount
they have used Machine Learning techniques for
of bitcoin that will ever exist [6].
their decision-making processes. This quantita-
tive investment strategy has been developed pri-
marily for stock markets.

The birth of cryptocurrency market created

Figure 1: Global crypto market cap [7]
Despite the lack of security while trading and
using cryptocurrencies, this technology is still
evolving. Its total market capitalization even
peaked at around $2.6 trillion in October 2021
and along with it, a series of investment funds
something new and interesting: the high volatil-
ity in cryptocurrency prices and its potential big
profit have led many researchers to use machine
learning and deep learning algorithms for pre-
dicting its movement. They have used multiple
models and strategies to complete this task.
In this project, we are trying to predict the Bit-
coin price movement in the next hour.
This report contains 6 main sections. Section
1 is Introduction. Section 2 is Methodology,
where we explain the flow of this report and
further describe the methods used to complete
this project. Section 3 is Data preparation - the

4
 Report
most important part in this report, where we de-
scribe the Data collection task, the Exploratory
data analysis task (EDA), the Feature engineer-
ing task, and finally, the Feature scaling and nor-
malization task. Section 4 is Models, where we
explain the theoretical background of different
models and the practical results. We conclude
the report in section 5, and list some further re-
search directions in section 6.
2. Methodology
For this project, we have followed this pipeline:
Figure 3: Project pipeline
Before making any statistical and machine learn-
ing models, the first task to do is data prepara-
tion. This task contains 4 subtasks which are
data collection, exploratory data analysis, fea-
ture engineering, and feature scaling and nor-
malization.
Next, after preparing the data, we choose the
models to approach and solve the problem. We
first analyze the theoretical background of each
model and next, choose the baseline model,
then, we fit the models to the data statistic the
results for more in-depth evaluation. The data
collected from the previous phase is split with-
out shuffled into 3 parts: train set (70%), val-
idation set (15%), and test set (15%). The in-
depth evaluation mentioned above is based on
the test set. The best model is chosen to be
retrained and re-evaluated by using time-based
cross-validation.
Finally, we compare our results with previous
solutions of other researchers and make the con-
clusion.
5
Group 05
3. Data Preparation
3.1. Data collection
We have collected the Bitcoin data from 2
sources (the API provided by Binance [10] and
the crawled result from CryptoQuant [11]). Our
data is based on the cryptocurrency pair BTC
– USDT where USDT is a stable coin, and its
value is worth $1. The time period of the data
that we studied is from 15:00:00 (December 19,
2017) to 10:00:00 (June 22, 2022).
The initial data can be divided into 2 groups
(classic price-volume data and on-chain data).
Classic price-volume data is the data about
the price and the volume in the Binance ex-
change, while on-chain data is the third-party
data which is inferred by using blockchain data
analytics and centralized exchange data ana-
lytics. Blockchain data analytics involves the
block discovery of BTC to derive some met-
rics and indicators while centralized exchange
data analytics involves the use of data provided
from the centralized exchange such as the total
amount of BTC transferred to the centralized
exchange, the total amount of BTC transferred
from the centralized exchange, . . . Note that we
have used the concept of centralized exchange
to distinguish from the decentralized exchange,
which is a part of decentralized finance - a de-
rived finance technology based on blockchain.
Blockchain data analytics and centralized ex-
change data analytics can be used at the same
time, for example, the data provider uses the
blockchain wallet of a centralized exchange to
make a new indicator about the amount of BTC
flowing in this exchange.
For more details, check Appendix A
3.2. Exploratory data analysis
3.2.1. Label analysis (Close price)
3.2.1.1. Overall analysis (Close price)
The Bitcoin price is highly volatile. After the
last 3 months of 2017 when the price increased
more than 6 times from $3,000 to a peak of
$20,000, BTC witnessed a decline from the peak
to $6,000 in the first 2 months of 2018. Dur-
ing the period from May to November of 2018,
BTC always held the support of $6,000, but
on November 14, BTC broke this hard support,
 Report
causing its price to halve and fluctuate between
$3,000 and $4,000 until April 2019. From April
4 to the end of June this year, the price of BTC
increased sharply 3 times and then continued to
decline to the old price 9 months later. BTC
then moved sideways through October 2020 in
the $8000 to $12,000 price range. The first 10
months of this year are also the time when big
investors and sharks gather to prepare for a big
BTC wave in 2021. 1 year later, BTC peaked at
nearly $70000 and it has corrected to a third
of its peak price with a current value of just
$21,000.
Figure 4: Bitcoin Close Price
The figure 4 shows the difference between the
current day’s closing price and the previous
day’s price. It can be seen that this spread is
more volatile in the recent period from 2021
Figure 5: Bitcoin close price 1st difference
6
Group 05
3.2.1.2. Decomposition of Time Series
close price data
*Theoretical background
First of all, we need to understand the 3 pat-
terns that make up a time series data, which are
“trend”, “seasonal” and “residual”.
Trend (Tt )
A trend pattern shows the tendency of the data
to increase or decrease during a period of time.
A time series data can go up, go down instantly
but the trend must be upward, downward or sta-
ble in a period of time.
Seasonal (St )
A seasonal pattern represents the seasonal factor
that affect the time series data. The seasonality
is fixed and has a known frequency.
Residual (Rt )
A residual pattern is the result when we ex-
tract trend and seasonal components out of the
original time series data. This residual may be
named remainder by other researchers, but in
this report, we use the residual to name it along
with the library convention.
Since time series data contains a variety of pat-
terns, and it is often helpful to split a time
series into 3 components: a trend component,
a seasonal component, and a residual compo-
nent. In our project, we have decided to decom-
pose the original time series data into 2 compo-
nents: a trend seasonal component which com-
bines trend component with seasonal component
and a residual component.
There are 2 types of decomposition:
Additive decomposition
yt = Tt + St + Rt
Multiplicative decomposition:
yt = Tt ∗ St ∗ Rt
*Analysis
We decompose the close price data into 3 com-
ponents by using statsmodel library [12]:
 Report
Figure 6: Close price trend seasonal component
Figure 7: Close price residual component
For this time series data, the seasonal compo-
nent is too low as depicted in Figure 8, so we
include it in the trend component to make the
trend seasonal component.
Figure 8: First 100 close price seasonal compo-
nent datapoints
We decided to do more analysis for the resid-
ual component to create features (we don’t do
7
Group 05
it on trend component because this component
is highly correlated with MA features). The fol-
lowing is the residual distribution:
Figure 9: Residual histogram
We can see that the distribution is skewed to-
wards the mean (near 0) because during 2 years
before 2021, the close price is small so that
the margin of the residual component is small).
Therefore, in this period of time, statistical mod-
els and machine learning models can work well.
But after that, there exists the problem of drift
which will greatly affect the models, the price
fluctuates sharply and so does the residual. For
this reason, we take the rate of residual and
trend to remove the price unit and make it fair
for every timestamp.
Figure 10: Residual divided by trend histogram
- 1 hour
We have removed the effect of price but the dis-
tribution is still relatively skewed towards the
mean. This shows that with high frequency (1h)
the rate of variation will be small.
To reinforce the argument, if we continue to do
the same with lower frequency (1 day), the rate
 Report
variable of small fluctuation is significantly re-
duced compared to high frequency (1h).
Figure 11: Residual divided by trend histogram
- 1 day
We will use these analyses to do the same on
trading volume. Then we will create more fea-
tures based on the above insights that we found.
3.2.2. Feature analysis
Figure 12: Features correlation matrix
As we can see on the correlation matrix, the
price information is almost completely corre-
lated with each other, which is understandable
since we are observing over a period of relatively
high frequency.
We also can see the correlation between miner
reserve USD and exchange reserve USD with
BTC price, which is understandable since it is
calculated as BTC amount multiplied by BTC
price, so this correlation with the price is natu-
ral.
8
Group 05
3.3. Feature engineering
In fact, investors often use technical indicators
to make decisions about when and where to buy
and sell. That is the reason we generate new fea-
tures based on the definition of these technical
indicators. We also include the trend seasonal
component, residual component and the indica-
tors derived from it in our features.
For more details about the features, check Ap-
pendix B.
3.4. Feature scaling and normaliza-
tion
All the features can be divided into 2 groups:
continuous features (which are not scaled) and
scaled features (which are already scaled by its
definition, for example, the value of RSI indica-
tor is between 0 and 100).
As described in section 3.2.2, some features can
drift, that is the reason we should scale the fea-
tures carefully. Others, depending on the al-
gorithm we use, the scaling and normalization
methods may be different.
In this section, we describe only the scaling
method, for more details about the different
scaling methods applied to different features,
check Appendix B.
3.4.1. Window normalization
This is the a method that we use to normalize
the data and make it undrift:
data[i] − data[i − p : i].mean()
datanorm [i] =
data[i − p : i].std()
where datanorm is the data after being normal-
ized, data[i] is the original data at the row i,
data[i − p : i] contains all the rows from the last
p days through the row before row i.
3.4.2. Logarithmic scale
Logarithmic scale method is applied for all the
features having positive values, they are marked
using their logarithms instead of their actual val-
ues. This scaling method is used to deal with
the problem of skewness towards large values;
i.e., cases in which one or a few points is much
larger than the rest of the data.
 Report Group 05

4. Models pends only on the lagged forecast errors:

In our work, we have proposed several models Yt = c + θ1 ∗ ϵt−1 + θ2 ∗ ϵt−2 + . . . + θq ∗ ϵt−q
based on statistics and machine learning to pre-
• d: The minimal order differencing required
dict bitcoin price movements and forecast bit-
to get a near-stationary series which roams
coin prices. After applying all the models, we around a defined mean and the ACF plot
determine which one is the most accurate for our reaches to zero fairly quick.
future goals and choose the appropriate param-
eters to get better performance in this model. B ∗ Yt = Yt−1

where B is called a backshiff operator. Thus, a

4.1. Theoretical background first order difference is written as:
4.1.1. Statistical models ′
Yt = Yt − Yt−1 = (1 − B) ∗ Yt
Statistical modeling is the use of mathematical
So, in general, a d th_order difference can be
models and statistical assumptions to generate
written as :
sample data and make predictions about the real
′
world. Yt = (1 − B)d ∗ Yt

4.1.1.1. ARIMA model

ARIMA, abbreviated for ’Auto Regressive Inte-
grated Moving Average’, is a class of models that
’demonstrates’ a given time series based on its
previous values: its lags and the lagged errors in
forecasting, so that equation can be utilized in
An ARIMA model is one where the time se-
ries was differenced at least once to make it sta-
tionary and you combine the AR and the MA
terms. So the equation becomes:
′ ′ ′
Yt = c + ϕ1 ∗ Yt−1 + ϕ2 ∗ Yt−2 + . . . +
′

order to forecast future values. ϕp ∗ Yt−p + θ1 ∗ ϵt−1 + θ2 ∗ ϵt−2 + . . . +

There are three terms characterizing an θq ∗ ϵt−q + ϵt
ARIMA model
4.1.1.3. Find the order of differencing
• p: the order of the AR term.
(d) in ARIMA model
• d: the number of differences required to
make the time series stationary.
• q: the order of the MA term.

4.1.1.2. What are ’p’, ’q’, ’d’ mean

First, we need to make the time series station-
ary in order to build an ARIMA model. This is
because AR: auto regressive in ARIMA im- Figure 13: Close price before difference
plies a Linear regression model using its lags
as predictors. And we already know that Linear
The purpose of differencing it to make the time
regression models work well for independent
series stationary.
and non-correlated predictors.
The right order of differencing is the minimum
Therefore, the value of d is the minimum num-
differencing required to get a near-stationary
ber of subtractions. If the time series is already
series which roams around a defined mean and
stationary, then d = 0.
the ACF plot reaches to zero fairly quick.
• p is the order of the AR (Auto-
Regressive) term which means that Yt de-
pends on its own lags:
Yt = α + β1 ∗ Yt−1 + β2 ∗ Yt−2 +
β3 ∗ Yt−3 . . . + βp ∗ Yt−p + ϵ1

• q is the order of the MA (Moving-

Average) term which means that Yt de- Figure 14: The close price after first difference

9
 Report Group 05

remains in the differenced series. Of course, you

Figure 15: ACF and PACF plot of each differ-
ence of Close Price
could just try some different combinations of
terms and see what works best. But there is a
more systematic way to do this. By looking at
the auto-correlation function (ACF) and partial
auto-correlation function (PACF) plots of the
differenced series, you can tentatively identify
the number of AR and MA terms that are
needed
Note: See on figure 15

4.1.1.5. Find the order of the MA terms

We can see that the close price is similar to white (q) in ARIMA model
noise. We can check that if the series is sta-
tionary by Augmented Dickey Fuller Test (ADF In order to find q, we can count the number of
Test). significant lags (outside the blue band) on the
ACF graph.
Augmented Dickey-Fuller Test Results: Note: See on figure 15
ADF Test Statistic -1.507686e+01
P-Value 8.565591e-28 4.1.1.6. Visualizaion of ARIMA model
Lags Used 5.000000e+00 results
Observations Used 1.639000e+03 After tuning and training, we have the summary
Critical Value (1%) -3.434346e+00 of ARIMA model.
Critical Value (5%) -2.863305e+00
Critical Value (10%) -2.567710e+00

As we can see from the results when using ADF

test on the first diff of close price, p_value is
close to 0. So we can reject the null hypothesis
and assume that the series is stationary. We will
use return-measurement as input for model.

Return: The percent change in a stock

price over a given amount of time.
P rice(t) − P rice(t − 1)
Returns(t) = ∗ 100
P rice(t − 1)

Figure 17: Summary of ARIMA model

Figure 16: Returned series

4.1.1.4. Find the order of AR terms (p)

in ARIMA model
After the time series has been stationarized by
differencing, the next step in fitting an ARIMA
model is to determine whether AR or MA terms
are need to correct any auto-correlation that
Figure 18: The random 300 data points of train-
ning set results
We can see that, the predicted value is equal to
the first lag order of the true value.

10
 Report Group 05

4.1.1.7. GARCH Model GARCH model estimates the conditional vari-

The GARCH or Generalized Autore- ance present in the residuals of the ARIMA
gressive Conditional Heteroskedasticity estimation.
p q
method provides a way to model a change in X X
variance in a time series that is time dependent, yt = µ + ai ∗ yt−1 + bi ∗ ϵt−i + ϵt
i=1 i=1
such as increasing or decreasing volatility. If the
change in variance can be correlated over time, where
√
then it can be modeled using an autoregressive × zt
ϵt = σ t P
σ 2 = ω + pi=1 αt × ϵ2t−i + qi=1 βi × σt−i
2
P
process, such as ARCH.
ARCH models are a popular class of volatility
ARIMA model is not capable of forecasting a
models that use observed values of returns or
very long time, when the forecast period ap-
residuals as volatility shocks. A basic GARCH
proaches very large, the forecast results will be
model is specified as
almost constant. And ARIMA as we see the
rt = µ + ϵt model almost lags 1 with the actual price so we
want to improve it
ϵt = σt × et
q p
We wanted a margin measure to make the arima
X X more reliable by combining with the ARCH
σt2 = α0 + αi × ϵ2t−i + βj × 2
σt−j
i=1 j=1
model to get a more confident prediction inter-
val. By using ARCH’s forecast as the upper and
where et ∼ N (0, 1). lower bounds for ARIMA’s forecast results, we
get a confidence interval.
After tunning and trainnig on the residual out-
put of ARIMA model with the same process for
ARIMA, we get the summary of the GARCH
model.

Figure 20: Confident interval on whole test set

For ease of observation, we choose randomly 300

continuous data points to visualize.

Figure 21: 300 random continuous data points

with confident interval

And we compute the Probability of confidence

Figure 19: GARCH model summary of volatility interval bound actual value (Return value on
close price) and the result is 0.65329.
4.1.1.8. ARIMA_ARCH model 4.1.2. Machine Learning models
ARIMA and GARCH models are so of- The first concept that should be clear is odds, it
ten combined. An ARIMA model estimates is defined as:
the conditional mean, where subsequently a p
Odds(p) =
1−p

11
 Report
Odds is a function that varies between 0 and ∞.
We know that the good situation occurs when
odds > 1, it means that a bad odds happens
when odds function varies between 0 and 1 while
a good odds function happens when its value is
between 1 and ∞, and this is not fair. This
asymmetry makes it difficult to compare these 2
situations, that’s why we need logarithmic scale.
Figure 22: Odds and logit function
The logarithmic scale of the original odds func-
tion is logit function. This function makes the
odds assessment symmetrical and fair. So we
have:
logit(p) = logit(P (y = k|x))
= log(odds(p))
 
P (y = k|x)
= log
1 − P (y = k|x)
where p = P (y = k|x) is the beneficial probabil-
ity (when y = k) given x.
4.1.2.1. Logistic regression
Suppose that this logit function can be formed
by a linear relation with our data x. So:
 
P (y = k|x)
logit(p) = log = A0 + AT x
1 − P (y = k|x)
The decision boundary is a hyperplane which is
the set of points x for which the log-odds are
zero and defined by A0 + AT x = 0. There are
2 popular methods that result in linear logits:
linear logistic regression - which is discussed in
this section and linear discriminant analysis -
which is discussed in the next section.
e A0 +AT x
Now we have: P (y = k|x) = , s1 2 + s22
1 + eA0 +AT x
which can be seen as a sigmoid function of x.
12
Group 05
In order to find A0 and A, we will minimize a loss
function and the popular loss function for the
classification problem is binary cross-entropy.
4.1.2.2. Linear Discriminant Analysis
(LDA)
The approach of LDA is different from logistic
regression, instead of assuming about the linear
logits (which means that logit(P (y = k|x)) can
be formed by a linear relation A0 + AT x), we
assume that p = P (y = k|x) follows a normal
distribution.
Figure 23: The distance between the expecta-
tions and the sum of the variances affect the
discriminant degree of the data [13]
The problem of LDA is to find a projection on an
arbitrary axis through which we can determine
the separation of two classes. This method can
be considered as a dimension reduction method
where we reduce the original dimension into a
lower dimension where we can easily determine
the separation mentioned above.
Assume that there are N data points
x1 , x2 , . . . , xN which are divided into 2 classes:
C1 and C2 . The data projection onto a straight
line can be described by a coefficient vector w
and the corresponding value of each new data
point is given by: yi = wT xi , 1 ≤ i ≤ N .
The expected vector of each class is:
1 X
mk = xn , k = 1, 2
Nk
n∈Ck
According to Figure 23, the best solution is when
m1 , m2 are farthest away and s1 , s2 are the
same. That is the reason LDA tries to solve:
(m1 − m2 )2
maxJ(w) =
where
 Report
1 X 1 X
m1 −m2 = yi − yj = wT (m1 −m2 )
N1 N2
i∈C1 j∈C2
and
X ples from X, Y; call these Xb , Yb ;
s2k = (yn − mk )2 , k = 1, 2
n∈Ck
For more details, check [13]
4.1.3. Ensemble methods
Ensemble methods are techniques that create
multiple models and then combine them to pro-
duce improved results [14]. In this project, we
have used Random Forest, Adaboost and Light-
GBM.
4.1.3.1. Random Forest [15]
Decision trees are a popular method for various
machine learning tasks. Tree learning "come[s]
closest to meeting the requirements for serving
as an off-the-shelf procedure for data mining",
say Hastie et al., "because it is invariant un-
der scaling and various other transformations of
feature values, is robust to inclusion of irrele-
vant features, and produces inspectable models.
However, they are seldom accurate".
In particular, trees that are grown very deep
tend to learn highly irregular patterns: they
overfit their training sets. Random forests are
a way of averaging multiple deep decision trees,
trained on different parts of the same training
set, with the goal of reducing the variance. This
comes at the expense of a small increase in the
bias and some loss of interpretability, but gener-
ally greatly boosts the performance in the final
model.
Forests are like the pulling together of decision
tree algorithm efforts. Taking the teamwork of
many trees thus improving the performance of
a single random tree. Though not quite similar,
forests give the effects of a k-fold cross valida-
tion.
Bagging
The training algorithm for random forests ap-
plies the general technique of bootstrap aggre-
gating, or bagging, to tree learners. Given a
training set X = x1 , x2 , ..., xn with responses
Y = y1 , y2 , ..., yn , bagging repeatedly (B times)
13
Group 05
selects a random sample with replacement of the
training set and fits trees to these samples:
Algorithm 1 Random Forest training algorithm
for b = 1, 2, ..., B do
Sample, with replacement, n training exam-
Train a classification or regression tree fb on
Xb , Yb ;
end
After training, predictions for unseen samples x′
can be made by averaging the predictions from
all the individual regression trees on x′ :
B
1 X
fˆ = fb (x′ )
B
b=1
or by taking the majority vote in the case of
classification trees.
This bootstrapping procedure leads to better
model performance because it decreases the vari-
ance of the model, without increasing the bias.
This means that while the predictions of a single
tree are highly sensitive to noise in its training
set, the average of many trees is not, as long
as the trees are not correlated. Simply train-
ing many trees on a single training set would
give strongly correlated trees (or even the same
tree many times, if the training algorithm is de-
terministic); bootstrap sampling is a way of de-
correlating the trees by showing them different
training sets.
Additionally, an estimate of the uncertainty of
the prediction can be made as the standard de-
viation of the predictions from all the individual
regression trees on x′ :
s
PB ˆ2
′
b=1 (fb (x ) − f )
σ=
B−1
The number of samples/trees, B, is a free param-
eter. Typically, a few hundred to several thou-
sand trees are used, depending on the size and
nature of the training set. An optimal number
of trees B can be found using cross-validation,
or by observing the out-of-bag error: the mean
prediction error on each training sample xi , us-
ing only the trees that did not have xi in their
 Report Group 05

bootstrap sample. The training and test error low bias and low variance, or model with good
tend to level off after some number of trees have predictive results.
been fit.
The idea of aggregating decision trees of the
From bagging to random forests Random Forest algorithm is similar to the idea
The above procedure describes the original bag- of The Wisdom of Crowds proposed by James
ging algorithm for trees. Random forests also in- Surowiecki in 2004. The Wisdom of Crowds says
clude another type of bagging scheme: they use that usually aggregating information from one
a modified tree learning algorithm that selects, group is good than from an individual. In the
at each candidate split in the learning process, Random Forest algorithm, it also synthesizes in-
a random subset of the features. This process is formation from a group of decision trees and the
sometimes called "feature bagging". The reason results are better than the Decision Tree algo-
for doing this is the correlation of the trees in an rithm alone.
ordinary bootstrap sample: if one or a few fea-
tures are very strong predictors for the response
variable (target output), these features will be
selected in many of the B trees, causing them
to become correlated. An analysis of how bag-
ging and random subspace projection contribute
to accuracy gains under different conditions is
given by Ho.

Typically, for a classification problem with p fea-

√
tures, p (rounded down) features are used in
each split. For regression problems the inventors
recommend p/3 (rounded down) with a mini-
mum node size of 5 as the default. In practice Figure 24: Error with training rounds [17]
the best values for these parameters will depend
on the problem, and they should be treated as 4.1.3.2. AdaBoost
tuning parameters. AdaBoost uses an iterative approach to learn
from the mistakes of weak classifiers and turn
Why Random Forest is good? [16]
them into strong ones. What this algorithm does
In the Decision Tree algorithm, when building a
is that it builds a model and firstly gives equal
decision tree, if the depth is arbitrary, the tree
weights to all the data points.
will correctly classify all the data in the training
set, leading to the overfitting problem. The initial formula to calculate the sample
Random Forest algorithm consists of many de- weight is:
cision trees, each decision tree has random ele-
ments: 1
w(xi , yi ) =, i = 1, 2, ..., N
• Randomize data to build a decision tree. N
• Randomize the attributes to build a deci- Next, AdaBoost will create a decision stump
sion tree. having the lowest Gini index [18].
Since each decision tree in the Random Forest
algorithm does not use all the training data, nor After that, AdaBoost will calculate the “Amount
does it use all the attributes of the data to build of Say” for this classifier in classifying the data-
the tree, each tree may make a bad prediction. points using this formula:
The decision tree is not overfitting but can be 1 1 − T otalError
underfitting, in other words, the model has high log
2 T otalError
bias. However, the final result of the Random
Where the total error is the summation of all the
Forest algorithm is aggregated from many deci-
sample weights of misclassified data points.
sion trees, so the information from the trees will
complement each other, leading to a model with

14
 Report Group 05

It can be seen that the test error decreases even

after training error reached zero, but why do
more weak learners and more complex hypothe-
sis not lead to overfitting?

This can be explained by the Margin Theory,

which states that AdaBoost keeps increasing the
margins even after it has managed to classify all
training examples correctly [20] [21].

4.1.3.3. LightGBM
Figure 25: Amount of say First, we need to understand how the Gradient
Boosting Decision Tree works.
Note that the total error will always be between
0 and 1 and 0 indicates perfect stump and 1 For a given data set with n examples and m
indicates horrible stump. From Figure 10, it can features
be seen that if the error rate is 0.5 (the classifier
predicts half right and half wrong), the “amount
D = (xi , yi )(|D| = n, xi ∈ Rm ), yi ∈ R)
of say” will be 0. If the error rate is small, alpha
will be positive. If the error rate is large, alpha Starting with the predicted value obtained by
will be negative. the tree using K additive functions, we have:
K
New sample weight = Old weight × e−α
X
yˆi = ϕ(xi ) = fk (xi ), fk ∈ F
k=1
The new sample weights will be normalized, and
Where F is the space of regression trees, each fk
a new training round will be performed.
corresponds to an independent tree structure q
In conclusion, the individual learners can be and leaf weights w, yˆi is the predicted value.
weak, but if the performance of each one is
To learn the set of functions used in the model,
slightly better than random guessing, the final
we minimize the following regularized objective:
model can be proven to converge to a strong
learner. X X
L(ϕ) = l(yˆi , yi ) + Ω(fk )
i k
Adaboost often not overfit on practice
When using Adaboost, we observe it behaving 1
where Ω(f ) = γT + λ||w||2
like this: 2
The first term is the training loss and the second
term, which is the regularized term, represents
the complexity of all the trees. Here, T is the
number of leaves in the tree and w is the leaf
weight vector of each tree. γ and λ are the 2
regularization parameters that are pre-defined.
Our objective is to minimize the loss function
above.

But the model is trained in an additive manner,

we need to define the regularized objective at the
tth iteration, and we will minimize this objective.
Formally, let yˆi (t) be the prediction of the ith
instance at the tth iteration, we will have:
Figure 26: Error with training rounds [19]

15
 Report Group 05

That is the core idea of XGBoost algorithms.

n
Algorithm 2 Exact Greedy Algorithm for Split
X
L(t) = l(yi , yˆi (t−1) + ft (xi )) + Ω(ft )
i=1
Finding [22]
Input I, instance set of current node
Second-order approximation can be used to Input d, feature dimension
quickly optimize the objective. gain ←
P0 P
G ← i∈I gi , H ← i∈I hi
n
X 1 for k = 1 to m do
L(t) ≈ [l(yi , yˆi (t−1) )+gi ft (xi )+ hi ft2 (xi )]+Ω(ft ) GL ← 0, HL ← 0
i=1
2
for j in sorted(I, by xjk ) do
GL ← GL + gj , HL ← HL + hj
where gi = ∂ŷ(t−1) l(yi , ŷ (t−1) ) and hi =
GR ← G − GL , HR ← H − HL
∂ŷ2(t−1) l(yi , ŷ (t−1) ) are first and second order gra- G2L G2
score ← max(score, HL +λ + HRR+λ −
dient statistics on the loss function.
G2
H+λ )
Here, l(yi , yˆi (t−1) ) is L(t−1) and gi ft (xi ) +
1 end
hi f 2 (xi ) + Ω(ft ) is L̃(t) - the additional loss
2 t end
from iteration t. And since L(t−1) is a constant Output Split with max score
term, we remove it to obtain the simplified ob-
jective at step t.
Algorithm 3 Approximate Algorithm for Split
Define Ij = {i|q(xi ) = j} as the instance set of
leaf j. We have: Finding [22]
Input I, instance set of current node
n
X 1 Input d, feature dimension
L̃(t) = [gi ft (xi ) + hi ft2 (xi )] + Ω(ft )
i=1
2 for k = 1 to m do
n T Propose Sk = {sk1 , sk2 , ..., skl } by per-
X 1 1 X 2 centiles on feature k.
L̃(t) = [gi ft (xi ) + hi ft2 (xi )] + γT + λ w
i=1
2 2 j=1 j Propose can be done per tree (global), or
XT X 1 X per split (local).
= [( gi )wj + ( hi + λ)wj2 ] + γT
2
j=1 i∈Ij i∈Ij end
for k = 1 to
Pm do
Now we can compute the optimal weight wj∗ of Gkv ← j∈(j|sk,v ≥xjk >sk,v−1 ) gj
leaf j by: P
Hkv ← j∈(j|sk,v ≥xjk >sk,v−1 ) hj
P
i∈Ij gi
wj∗ = − P end
i∈Ij hi + λ Follow same step as in previous section to find
And its corresponding optimal value is: max score only among proposed splits.

T 2
P
(t) 1 X ( i∈Ij gi ) That is the basics of Gradient Boosting Deci-
L̃ (q) = − + γT
sion Tree. Now, we will deep dive into Light-
P
2 i∈Ij hi + λ
j=1
GBM. Indeed, LightGBM is a gradient boosting
The formula above is used to measure the quality framework based on decision trees to increase
of the tree structure q. And for evaluating the the efficiency of the model and reduce memory
split candidates, we have the following formula:
usage.
( i∈IL gi )2 ( i∈IR gi )2
" P P #
( i∈I gi )2
P
1
Lsplit = P +P −P −γ It uses two novel techniques: Gradient-based
2 i∈IL hi + λ i∈IR hi + λ i∈I hi + λ
One Side Sampling (GOSS) and Exclusive Fea-
where IL , IR , I are the instance set of left leaves, ture Bundling (EFB) which fulfills the limita-
right leaves and leaves before splitting, respec- tions of histogram-based algorithm that is pri-
tively. marily used in all Gradient Boosting Decision

16
 Report Group 05

Tree frameworks, as described by the authors in 4.1.4.1. LSTM [24]

[23].
Gradient-based One Side Sampling Tech-
nique
These authors above notice that the gradient for
each data instance in GBDT provides us with
useful information for data sampling. That is, if
an instance is associated with a small gradient,
the training error for this instance is small and it
is already well-trained. A straightforward idea is
to discard those data instances with small gra-
dients. However, the data distribution will be
changed by doing so, which will hurt the accu-
racy of the learned model. To avoid this prob-
lem, the authors propose a new method called
Gradient-based One-Side Sampling (GOSS).
Long Short Term Memory networks are a kind
of Recurrent Neural Network, capable of learn-
ing long-term dependencies and were introduced
by Hochreiter Schmidhuber (1997) [25]. This
model is widely used in time series problems be-
cause of its strength.
Firstly, note that all recurrent neural networks
have the form of a chain of repeating modules
of neural network. In a standard RNN model,
each repeating module has a very simple struc-
ture with a single tanh layer.

GOSS keeps all the instances with large gradi-

ents and performs random sampling on the in-
stances with small gradients. In order to com-
pensate the influence to the data distribution,
when computing the information gain, GOSS Figure 27: The repeating module in a standard
introduces a constant multiplier for the data in- RNN contains a single layer [24]
stances with small gradients. Specifically, GOSS
firstly sorts the data instances according to the LSTM inherits this structure, but in a different
absolute value of their gradients and selects the way. Instead of having a single neural network
top a × 100% instances. Then it randomly sam- layer, there are four layers that interact in a spe-
ples b×100% instances from the rest of the data. cial way.
After that, GOSS amplifies the sampled data
with small gradients by a constant 1−a
b when cal-
culating the information gain. By doing so, we
put more focus on the under-trained instances
without changing the original data distribution
by much.

Exclusive Feature Bundling Technique

High-dimensional data are usually very sparse
Figure 28: The repeating module in an LSTM
which provides us a possibility of designing a
contains four interacting layers [24]
nearly lossless approach to reduce the number of
features. Specifically, in a sparse feature space,
many features are mutually exclusive, i.e., they The notations can be understood as illustrated
never take nonzero values simultaneously. The below:
exclusive features can be safely bundled into
a single feature (called an Exclusive Feature
Bundle). Hence, the complexity of histogram
building changes from O(#data × #f eature)
to O(#data × #bundle), while #bundle << Figure 29: LSTM notations [24]
#f eature. Hence, the speed for training frame-
work is improved without hurting accuracy.
The key component of LSTM is the cell state,
4.1.4. Deep learning models denoted by Ct (which means the cell output
value at tth unit of LSTM). Its where LSTM

17
 Report
remember the information, it can remove or add
information to this cell state at each unit, care-
fully regulated by structures called gates.
Figure 30: LSTM cell state line [24]
Gates are a way to decide if information can pass
through. They are composed out of a sigmoid
layer and a pointwise multiplication operation.
This sigmoid layer outputs numbers between
zero and one, describing how much of each com-
ponent should be let through (from 0, which
means “let nothing through”, to 1, which means
“let everything through”)
An LSTM has three of these gates, to protect
and control the cell state.
Figure 31: LSTM gate [24]
Steps of LSTM
The first step in LSTM is to decide what infor-
mation it is going to throw away from the cell
state. This decision is made by a sigmoid layer
called the “forget gate layer.” It uses ht−1 and xt
(where ht−1 is the output result to the memory
cell at time t − 1 and xt is the input vector of
the memory cell at time t) to output a number
between 0 and 1 for each number in the cell state
Ct−1 .
Figure 32: LSTM forget gate [24]
The next step is to decide what new informa-
18
Group 05
tion we’re going to store in the cell state. This
has two parts. First, a sigmoid layer called the
“input gate layer” decides which values we’ll up-
date. Next, a tanh layer creates a vector of new
candidate values, C̃t , that could be added to the
state. LSTM will then combine these two to cre-
ate an update to the state.
Figure 33: LSTM input gate and new cell state
[24]
Next, LSTM will update the old cell state Ct−1
into the new cell state Ct . It multiplies the old
state by ft , forgetting the things we decided to
forget earlier and then adds C̃t (the new candi-
date value) × it (how much we want to update
this candidate state value).
Figure 34: Update cell state [24]
Finally, LSTM will decide the final output.
First, a sigmoid layer will decide what parts of
the cell state we’re going to output. Second, a
tanh layer is applied to the cell state Ct (to push
the values to be between 1 and 1) and multiplied
by the previous output of the sigmoid gate to get
the final output value.
Figure 35: LSTM output gate [24]
 Report Group 05

4.1.4.2. GAN [26] [27] This is our GAN architecture:

The generative network generates candidates
while the discriminative network evaluates
them. The contest operates in terms of data
distributions. Typically, the generative network
learns to map from a latent space to a data
distribution of interest, while the discriminative
network distinguishes candidates produced by
the generator from the true data distribution.
The generative network’s training objective is to
increase the error rate of the discriminative net-
work (i.e., "fool" the discriminator network by
producing novel candidates that the discrimina-
tor thinks are not synthesized (are part of the
true data distribution))

GANs often suffer from a "mode collapse" where

they fail to generalize properly, missing entire
Figure 36: GAN architecture [26]
modes from the input data. For example, a
GAN trained on the MNIST dataset containing
many samples of each digit, might nevertheless We use the LSTM to predict the next price, then
timidly omit a subset of the digits from its out- use this price to generate dummy data for Dis-
put. Some researchers perceive the root problem criminator to learn. And this is our generator
to be a weak discriminative network that fails to architecture:
notice the pattern of omission, while others as-
sign blame to a bad choice of objective function.
Many solutions have been proposed. Conver-
gence of GANs is an open problem.

GANs are implicit generative models, which

means that they do not explicitly model the
likelihood function nor provide means for find-
ing the latent variable corresponding to a given
sample, unlike alternatives such as Flow-based
generative model. About the algorithm, we will
find the solutions for the following function

minG maxD V (D, G) = Ex∼pdata (x)[log(D(x))] +

Ez∼px (z)[log(1−D(G(z)))]

We follow the same idea in [26]. We use the Gen-

erator as an LSTM network to predict the price
at time t of the time series and combine with
the forward value series so that the Discrimina- Figure 37: Generator architecture [26]
tor can act as a Deep Auto Regression network
to classify real and fake data. With the hope
that the Generator will generate data as close
to reality as possible. Through this, we hope to
reduce the overfitting that occurs in the LSTM
training phase as described above.

19
 Report Group 05

4.2. Practical Results

Train set Validation set Test set

MODEL
AUC ACC AUC ACC AUC ACC
ARIMA 0.4553 0.4982
Logistic Regression 0.52388 0.52551 0.51602 0.50449 0.52032 0.50129
SVM (using calibration to get probability) 0.52974 0.51398 0.50404 0.50638 0.50838 0.49353
Linear Discriminant Analysis 0.63507 0.59709 0.54451 0.52985 0.54438 0.53269
Random Forest 1 1 0.53859 0.52329 0.54506 0.52769
LightGBM 0.64555 0.5986 0.55127 0.54658 0.56747 0.55262
AdaBoost 0.58722 0.56401 0.53913 0.53838 0.56194 0.55068
LSTM 0.59734 0.5699 0.54085 0.53639 0.55233 0.54708
GAN 0.56301 0.55922 0.50911 0.50729 0.51642 0.51280
Rolling training (with LightGBM) 0.58469 0.56977 0.57309 0.55148

As we can see, the performance of the statistical and time-consuming. Therefore, we will choose
models is the lowest. This is relatively easy to LightGBM as a representative model for analy-
understand because in this method we only use sis in the following sections.
the close price for forecasting, the model uses
much less information than others, so low per- 4.3. Evaluation and Error Analysis
formance is inevitable. In section 4.2, we can see that the performance
on the LightGBM model gives the best results.
Next to the linear methods, the method that we Therefore, in this section, all the performance
see the best performance is LDA. We don’t have evaluation and error analysis will be based on
time to analyze specifically what the reason is the output of the LightGBM model. We will first
here, but we will make an assumption that be- look at the important features of this model:
cause our window normalization causes the fea-
ture distributions to move towards normal dis-
tributions. Most of the dimensions do not carry
many classification capabilities, so they satisfy
more applicable conditions than Logistic Regres-
sion or Support Vector Machine.

Regarding ensemble learning methods, we can

see the superiority of boosting, performance on
this set of methods is stronger than other mod-
els.

As for Deep Learning models, the performance is

at a good threshold. We think the performance
can be even better on GAN and LSTM but we
don’t have resources to tune the parameters nor
make the neural network more complicated to
see if the performance can be higher than boost-
ing.
In summary, we can see the superiority of boost-
ing over linear models (with very fast train-
ing times and very good performance). Deep
Learning has a good performance but the train-
ing time is really long and the tuning of the
parameters is also extremely resource-intensive
Figure 38: Features importance barplot
BOP (balance of power) is the most important
feature, it gives us the feeling that the current
candle will greatly influence whether the next
candle will be green or red. The price-related
features after being normalized also appear at
the top of the most powerful features. In ad-
dition to the basic features or technical indica-
tors created, among the most powerful features
above, there are also some features calculated by
us based on EDA and analysis from part 3 such

20
 Report Group 05

as log_volume_resid_divide_prev. Finally, it
can be seen that all the features can be grouped
into the following: Price, Volume, Exchange,
Coinbase Exchange.

Now, we will analyze the AUC-iterations graph

during the training phase.

Figure 41: Validation set score distribution

Figure 39: AUC-Iterations plot

The AUC on validation set barely increases af-

ter a few rounds and early stopping is at 1518
(although using a tree base has a max depth of
2).

Figure 42: Test set score distribution

The score distribution is almost centered around

0.5, which shows that the model did not learned
through many boost rounds but the valid loss
has shown signs of reversal (we tried increasing
the boost round and not using early stopping,
then the results only on the training set are more
polarized and on the validation, results are ex-
tremely bad).

Even though we used window normalization to

Figure 40: Train set score distribution reduce the effect of drift and difference in sig-
nificance at the same values, the performance
didn’t improve too much (increased from accu-
racy from around 0.54 to around 0.55) .

We can see in the calibration curve that even

though the same behavior at different time inter-
vals, the same behavior at a different time may
go against the statistics we did in the training
set. (even when we choose an early stropping

21
 Report
strategy for validation set, the curve on test set
and validation set are bad).
Figure 43: Validation set calibration curve
Figure 44: Test set calibration curve
We intend to use a logistic regression to calibrate
the output of the model to improve performance
but the calibration curve of the datasets is too
bad to improve it so we will only choose the best
1 threshold on the validation set and used for the
test set.
From all examples just mentioned above, it
shows that past behavior is not enough to be
used as a model for the future (both in terms of
width (attributes) times length (number of sam-
ples)). It is true because the market depends
on many other factors such as: Inflation, inter-
est rates, Covid, stocks, war, ... and one factor
that can be said to be the most important is the
large quantity. Bitcoin is in the hands of a few
mysterious people, and they can manipulate the
market without any rules at all.
22
Group 05
4.4. Final Pipeline
After doing feature engineering and testing on
various models, we choose LightGBM with the
highest performance as our best model. Af-
ter that, we do the features selection process.
Considering the trained LightGBM model, we
can see the features with their importance gain.
For faster training with cumulative assessment,
we only choose the topmost important features
that contribute to 99.7% of the total importance
gain of all features. The number 99.7% comes
from the probability that a normal random vari-
able will fall within 3 standard deviations of the
mean. By this way of feature selection, we ob-
tained only 260 features, which means we can
reduce about 660 features.
After that, we evaluate the LightGBM model by
training cumulatively as follow: At the first it-
eration, we consider the first 10,000 data points.
From the end of the dataset, we choose the last
500 data points to be the test set, the next 2,000
data points to be the validation set, and the rest
are for the training set. After each iteration, we
increase the size of the dataset by the next 500
data points, and we choose the validation set
and test set the same way as before. At each
iteration, if the AUC score of the validation set
is lower than 0.56, we retrain the model of that
iteration, but this time we consider only 10,000
latest data points. This time, we choose the test
set is still the last 500 data points, but the vali-
dation set is the next 1000 data points.
With this method of training the LightGBM
model, we obtained the average accuracy of the
test set of 55.148%, and the average AUC score
of the test set of 0.57309.
Report
Figure 45: Test score histogram
Figure 46: Test set calibration curve
Figure 47: Validation score histogram
23
Group 05
Figure 48: Validation set calibration curve
When using the above pipeline to evaluate the
test set and the valid set, the calibration curve
is also a bit better. When considering the ac-
curacies of the whole test sets, we notice that
the time period from Jan 2021 to May 2021 has
poor performance.
After using the above evaluation method, both
the distribution and calibration curves of test
set and validation set have become quite similar,
that makes our prediction results more reliable.
5. Conclusion
Bitcoin prediction is an extremely difficult topic
for any method, as mentioned in the Feature En-
gineering section, the problem depends on too
many factors and if only based on common ana-
lytical data types, it is difficult to give a highly
accurate result. In the above report, we have
applied all the methods we have learned in 2
subjects Applied Statistics and Machine Learn-
ing to analyze data and come up with a Bitcoin
price forecasting model. Although we used many
different analytical methods as well as many dif-
ferent models to find a better result, we only
saw performance inching little by little. Due to
the difficulty of the problem, we do not expect
a high result, so we always try to come up with
the most complete process possible. We see that
this is an open topic, and there is potential for
further exploration in the future.
6. Scope for further research
This topic still has a lot of interesting research
directions that we can do.
 Report
In terms of data, we can add many other on-
chain indicators as well as calculate other tech-
nical indicators to improve the model. In addi-
tion, we can add data related to news and in-
vestor sentiment - these indicators can also be
accurate predictors of the next direction of the
price.
In terms of model, we can consider adding CNN,
Transformers and Attention, Deep Forest, . . .
These robust models can be combined to im-
prove the prediction performance.
References
[1] Satoshi Nakamoto. “Bitcoin: A peer-to-
peer electronic cash system”. In: Decentral-
ized Business Review (2008), p. 21260.
[2] How Digital Currency Is Ushering in a
New Era of Money Technology. Avail-
able at https : / / www . firstcitizens .
com/commercial/insights/technology/
digital-currency. May 2021.
[3] Guardian Nigeria. The idea and a brief
history of cryptocurrencies. Available at
https : / / guardian . ng / technology /
tech/the-idea-and-a-brief-history-
of-cryptocurrencies. May 2022.
[4] Cryptopedia Staff. The Early Days of
Crypto Exchanges. Available at https://
www.gemini.com/cryptopedia/crypto-
exchanges - early - mt - gox - hack. Mar.
2022.
[5] Wikipedia. Mt. Gox. Available at https:
//en.wikipedia.org/wiki/Mt._Gox.
[6] Brian McGleenon. How a bitcoin court
case in Japan may create crypto mil-
lionaires. Available at https : / / uk .
finance . yahoo . com / news / britcoin -
millionaires - mt - gox - case - japan -
153624083-230116218.html. Oct. 2021.
[7] Trading View. CRYPTOCURRENCY
MARKET. Available at https : / /
www . tradingview . com / markets /
cryptocurrencies / global - charts/.
2022.
[8] Swfinstitute. Rankings by Total Managed
AUM. Available at https : / / www .
swfinstitute . org / fund - manager -
rankings/crypto-fund-manager. 2022.
24
Group 05
[9] W. Ballmann and T. Seng. Quantitative
Investing. Available at https : / / www .
eurekahedge.com/Research/News/1102/
Quantitative-Investing. Oct. 2005.
[10] Binance. Binance API Documentation.
Available at https : / / binance - docs .
github.io/apidocs/spot/en. Sept. 2019.
[11] CryptoQuant. Bitcoin: Summary, on-
chain data analytics. Available at https:
/ / cryptoquant . com / asset / btc /
summary.
[12] statsmodels.tsa.seasonal.seasonald ecompose.
Available at https://www.statsmodels.
org / stable / generated / statsmodels .
tsa . seasonal . seasonal _ decompose .
html.
[13] Linear Discriminant Analysis. How a bit-
coin court case in Japan may create
crypto millionaires. Available at https :
//machinelearningcoban.com/2017/06/
30/lda. June 2017.
[14] Necati Demir. Ensemble Methods: Elegant
Techniques to Produce Improved Machine
Learning Results. Available at https : / /
www . toptal . com / machine - learning /
ensemble-methods-machine-learning.
[15] Wikipedia. Random Forest. Available at
https : / / en . wikipedia . org / wiki /
Random_forest.
[16] Tuan Nguyen. Random Forest al-
gorithm. Available at https : / /
machinelearningcoban . com / tabml _
book/ch_model/random_forest.html.
[17] Chetan Prabhu. What Do Random Forests
And The Wisdom of Crowds Have In
Common? Available at https : / / www .
linkedin . com / pulse / random - forest -
wisdom - crowds - chetan - prabhu/. May
2017.
[18] Neelam Tyagi. Understanding the Gini In-
dex and Information Gain in Decision
Trees. Available at https://medium.com/
analytics- steps/understanding- the-
gini - index - and - information - gain -
in - decision - trees - ab4720518ba8.
Mar. 2020.
 Report Group 05

[19] Nikolaos Nikolaou. Introduction to Ad-

aBoost. Available at https://nnikolaou.
github . io / files / Introduction _ to _
AdaBoost.pdf. 2014.
[20] Peter Bartlett et al. “Boosting the margin:
A new explanation for the effectiveness of
voting methods”. In: The annals of statis-
tics 26.5 (1998), pp. 1651–1686.
[21] Wei Gao and Zhi-Hua Zhou. “On the
doubt about margin explanation of boost-
ing”. In: Artificial Intelligence 203 (2013),
pp. 1–18.
[22] Tianqi Chen and Carlos Guestrin. “Xg-
boost: A scalable tree boosting system”.
In: Proceedings of the 22nd acm sigkdd in-
ternational conference on knowledge dis-
covery and data mining. 2016, pp. 785–
794.
[23] Guolin Ke et al. “Lightgbm: A highly effi-
cient gradient boosting decision tree”. In:
Advances in neural information processing
systems 30 (2017).
[24] Christopher Olah. Understanding LSTM
Networks. Available at https : / / colah .
github . io / posts / 2015 - 08 -
Understanding-LSTMs/. Aug. 2015.
[25] Jürgen Schmidhuber, Sepp Hochreiter, et
al. “Long short-term memory”. In: Neural
Comput 9.8 (1997), pp. 1735–1780.
[26] Kang Zhang et al. “Stock market predic-
tion based on generative adversarial net-
work”. In: Procedia computer science 147
(2019), pp. 400–406.
[27] Wikipedia. Generative adversarial net-
work. Available at https : / / en .
wikipedia . org / wiki / Generative _
adversarial_network.

25
 Report Group 05

A. RAW DATA

Figure 49: Raw data

26
 Report Group 05

B. FEATURES AND INDICATORS

Figure 50: Features and indicators - 1

27
Report Group 05

Figure 51: Features and indicators - 2

28