See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/286380167

Computer Science Predicts the Stock Market

Article · December 2015

CITATIONS READS
0 4,054

1 author:

Avery Leider
Pace University
18 PUBLICATIONS 67 CITATIONS

SEE PROFILE

All content following this page was uploaded by Avery Leider on 09 December 2015.

The user has requested enhancement of the downloaded file.

 Computer Science Predicts the Stock Market
Avery Leider
Pace University
Department of Computer Science
Pleasantville, New York
al43110n@pace.edu
Abstract— Why is it that the greatest utilization of recent
advances in faster computing power and smarter algorithms is in
the stock market? How are these advances changing the stock
market and what is the future? The place that gives the greatest
financial rewards for new computing phenomena is the stock
market, and so it is in the innovations of the stock market that we
can see the greatest advances of computer science in use. Today
Big Data is bringing about the rise of robot brokers who will be
able to manage stock portfolios by software models. The advances
in hardware and software are co-designed to benefit each other.
The trend is of increasing complexity and inclusion of
unstructured text data. This is enabled by amazing advances in
computing that allow greater amounts of data to be processed by
supercomputers and parallel processing of distributed computers.
Keywords—stock market; big data; text mining; robot brokers
I. INTRODUCTION
Enabled by greater amounts of data, that can be processed
by incredibly smarter algorithms that could not have been
used earlier because the speed of processing them was, in the
past, not fast enough to deliver results in time for decision
making, but are fast enough now, the big data revolution can
be seen best by observing its impact on the stock markets.
President Obama at “the White House defines Big Data as
“data that is so large in volume, so diverse in variety or
moving with such velocity, that traditional modes of data
capture and analysis are insufficient—characteristics
colloquially referred to as the ‘3 Vs.’”. [1] The “3 Vs” are
Volume, Variety and Velocity, and in big data, this means
more, more volume, more variety, and more velocity, than
previously.
More than the vast amounts of data, the smarter algorithms
of today make processing that big data at near real-time speeds
possible. This makes them useful in microseconds of decision
making.
“For example, stock market data are a typical domain that
constantly generates a large quantity of information, such as
bids, buys, and puts, in every single second. The market
continuously evolves and is impacted by different factors,
such as domestic and international news, government reports,
and natural disasters, and so on. An appealing
Big Data mining task is to design a Big Data mining system to
predict the movement of the market in the next one or two
minutes. Such systems, even if the prediction accuracy is just
slightly better than random guess, will bring significant
business values to the developers” [2]
Several of the algorithms that are in use today in the stock
market were originally used in other fields of scientific
research, such as genetics. In return, refinements of these
algorithms as they are used in the stock market may lead to
new discoveries of their usefulness in other fields. Use of
models and algorithms in new areas that are different than
what they were designed for carries risk as well as reward.
Additionally, data that before now was not possible, was
not collected, nor was possible to process, such as
unstructured text data, can now be included in what we
account for when trying to predict the stock market.
This phenomenon is spreading quickly worldwide.
Already text mining the news is spreading from English-
speaking stock markets to Turkish-speaking stock markets.
The exciting new advances in big data processing allow
fresher new possibilities for predicting the stock market
prices, however, these innovations will spread to other fields
such as medicine, education and transportation – it is just that
we see these evidenced first in the stock market because it is
more richly rewarded there than in any other field.
These smarter algorithms are thought of in terms of models.
The U.S. Federal Reserve System (Fed) defines “the term
model [as] a quantitative method, system, or approach that
applies statistical, economic, financial, or mathematical
theories, techniques, and assumptions to process input data into
quantitative estimates.” [3]
The Federal Reserve System’s purpose is to manage banks
in the U.S. so that the risk of their failure is low. Models are
used widely. Other than the stock market, financial
institutions, such as banks and insurance companies,
increasingly use models built with algorithms to manage their
information. The Fed is worried about the risk of algorithm
driven models being used for automated financial decision
making. In 2011, the Federal Reserve System issued a
regulation on Model Risk Management called SR 11-7
designed to control for the use of smarter algorithms executed
using models by banks. The risks, according to this regulation,
are of the increasing sophistication and complexity of models,
to the point that the human users of the models are no longer
understanding how they work and use them incorrectly, and
also that a model may produce inaccurate outputs, delivering
errors that are not corrected. When an algorithm developed for
a different area, such as genetics, is used in a new way in
banks, the Fed considers it a risk that must be managed by
regulations.
In the future of the smarter algorithms that are being
developed for the stock market, similar laws and regulations
will be applied, as government catches up to the catastrophic
risk that could come with mistakes made in the super-fast
financial big data mining models and robot-managed
portfolios.
II. ALGORITHMS TODAY – RELATED WORKS
A. Self Organizing Maps and Support Vector Regression
Making predictions based only on price changes of stocks
is not enough to account for risk and volatility. Adding in the
standard deviation or variance is needed. But further than
that, the investor needs “a robust tool that can accurately
gauge the mood of the market” and match it against the “risk
averseness or risk empathy” of the investor. The model
presented by Choudhury in 2014 [4] first clusters all of the
stocks based on their risk and return profiles using Self
Organizing Maps (SOM) with k-means clustering. [5] Then
Support Vector Regression (SVR) [6] is used to predict, for
short trading cycles (in his paper, a short trading cycle is two
trading days), the future price and volatility of the stocks. The
output of the model gives a range of values that the investor
can select from according to his tolerance of risk.
The model shows the direction of the price movements, up
or down, and predicts their volatility. What is unique in this
paper is that it combines k-means clustering with the
predictive technique of Support Vector Regression to deliver a
strategy for what stocks to buy and sell. Choudhury makes the
case to show that Support Vector Machine (SVM) is more
popular today than Artificial Neural Networks (ANN), saying
that SVM outperforms many other predictive algorithms. The
output from SVM that he demonstrates shows both potential
prices and their volatility values.
However, Choudhury reveals frustration where he says
“Markets are generally difficult to study owing to strong
coupling between stocks, nonlinear investor response to news
and imperfect dissemination of news among the investors
rendering it highly inefficient, a marked deviation from
traditional assumptions of market efficiency.” [4] This
frustration reveals the advantage that a perfect response to
news in faster time than the competition would deliver better
results, as well as the advantage of software robots who would
respond instantly in a linear fashion to a linear change to the
market that the news brought, unlike humans responding in a
nonlinear way to the same news that they processed in
different amounts of time.
B. Weighted Random Forests and Seaonality
Random forests are used in Booth’s 2014 paper [7] to predict
not only the signals to buy and sell stocks, but also the size that
the transaction should be. His paper aims to help software make
the decisions with “Automated trading system based on
performance weighted ensembles of random forests …trading
seasonality events”. [7] Seasonality, or seasonal trends, means
accounting for the turn-of-the-month and exchange holiday bias
(upward trend in prices) and weekend bias (downward trend in
prices).
The performance weight of the random forests is done by
Booth with time, with the most recent historical price data
having greater weight than more distant transactions. An
ensemble of two algorithms that are reacting to the seasonality
and performance weight are trained in random forests to predict
prices so that software can autonomously trade in the financial
markets using those predictions to signal when to buy and sell
and sometimes to signal the size of the transaction.
Using data from the European Stock Market, Ballings [8]
evaluated multiple benchmark ensemble methods (combining
two or more algorithms in a model) such as Random Forest
against single algorithm models such as Neural Networks,
Regression, Support Vector Machines and K-Nearest
Neighbor. Ballings found that, in 2015, the Random Forest is
the top algorithm, followed by Support Vector Machines, then
Neural Networks, K-Nearest Neighbor, and Regression.
Ballings contributed this comprehensive benchmark
comparison using as the criteria of rating each method its
accuracy in predicting the direction of stock prices of 5,767
publicly listed European companies, for the time period of one
year ahead. He observed that “Stock price direction prediction
is an important issue in the financial world. Even small
improvements in predictive performance can be very
profitable.” [8]
C. Particle Swarm Optimization and Recursive Least-
Squares
Using data from the Taiwan Stock Index (TAIEX), Feng
developed a prediction system that delivered output that
predicted the trend (up or down) of the TAIEX stock price
better than the available stock trend index. Feng’s system
uses four learning algorithms: regression analysis, dynamic
learning, as well as hybrid particle swarm optimization and
recursive least-squares. [9]
D. Software Robots
The field of intelligent virtual agents, or software robots, is
almost ready to join human brokers in the stock market. In
Hungary, there is an e-learning website that is teaching the
people of Hungary how to work the Hungarian stock market
by interacting with a “intelligent chatter robot connected to a
specialized knowledge base and with responses based on
emotional modeling of the user” as well as interacting with
human agents, in a virtual artificial stock market that applies
real trading rules and realistic market information. [10]
Humans and software robot-agents do not trade with the
same behavior in all environments, according to Feldman. [11]
He studied human traders and robot traders in a “financial
market simulation prone to bubbles and crashes.” [11]
Feldman’s results show that human traders earn lower profits
overall, but human traders earn higher profits in crash-
intensive periods. Both human and robots respond in manners
similar to each other in buying and selling choices when the
market is not in a bubble or crash. Feldman divided his
humans into inexperienced traders and experienced traders.
Experienced human traders did not make an impact on the
market, but inexperienced human traders “tend to destabilize
the smaller (10 trader) markets”. [11] Also different between
human and robot: after losses, humans are faster to sell than
robots are.
E. Preprocessing Data to Reduce Storage
When there is a vast amount of data that needs to be
processed at fast speed because of the perishability of the
information to be found in it, algorithms that preprocess the data
to reduce data storage and speed up calculations are useful. The
best example of this is Al-Jaroodi’s paper [12], in which he
describes how the wealth of real-time financial market data
available over the Internet can be retrieved, preprocessed with
pre-defined conditions of price and time, and only the relevant
data be stored, using storage algorithm he describes.
F. Parallel Processing of Artificial Neural Networks
Casas [13] demonstrates by experiment that making the
backpropagation algorithm run in parallel on four processors
simultaneously meant that the training time of the algorithm
was reduced by 61% over running the same algorithm over
one processor when using four inputs: (1) stock market data
of the Standard & Poor’s 500 Index, (2) the US Prime Rate,
(3) the Consumer Price Index and (4) the Oil Price in an
Artificial Neural Network (ANN). The purpose of Casas’s
experiment was to measure the improvement in the time
required to train an artificial neural network using multiple
cores. He conducted validation tests under three parallel
processing scenarios, confirming the increasing the speed and
effectiveness of parallel processing in learning to predict the
output with one, two, and, four cores. Training in all three
scenarios required 1,644 epochs. Training stopped after the
error reached less than 2%. His experiment proves that the
parallelization of training algorithms results in improved
training time. Training of ANN for stock markets involves
processing a volume of historical data which takes time.
Saving time increases the usefulness of the information which
is perishable.
G. Data Mining with Big Data
According to IBM, “Every day, we create 2.5 quintillion
bytes of data – so much that 90 percent of the data in the
world today has been created in the last two years alone.” [14]
“Big Data starts with large-volume, heterogeneous,
autonomous sources with distributed and decentralized
control, and seeks to explore complex and evolving
relationship among data.” [15] The most important topic with
software being used today for big data is the speed that useful
information can be obtained from it. Pre-processing and then
processing the large volumes of data quickly avoids trying to
store it. Near real-time results are more useful with big data.
One of the greatest challenges of big data is aggregation. Data
from different sources may be about the same subject, but be
difficult to use because each source has its own format or
protocols. [15]
One of the most important changes to computer science
that big data allows us to do is add a feature to our
computations that was not possible before: account for
unstructured text data that relates to news or sentiment. For
example, in the retail markets today, big data means that in
addition to keeping track of customers with data fields to find
where they have common values: age, gender, income,
location, education, market basket, there is something new -
with big data, retail businesses can add the many additional
values brought by including the data of relationships – the
social functions of family, friends, co-workers, and interest
groups and user communities – to study the behavior patterns
of customers of a business in exciting new ways. [15]
III. COMBING UNSTRUCTURED TEXT DATA FOR NEW OUTPUTS
“…recent research has shown that using social networks,
such as Twitter, it is possible to predict the stock market
upward/downward trends with good accuracies.” [16] How is
this accomplished using unstructured text data?
Data mining of unstructured text data, combined with
numeric data of the stock market, yields exciting returns.
Jaybhay [17] makes the case for this in his model, which uses
data for the Bombay Stock Exchange. Jaybhay says
“Technical analysts believe that the market is only 10 percent
logical and 90 percent psychological.” To account for the
psychological factors, he adds three unstructured text data of
news to the total of four financial information sources that
input into his model, all available on the Web:
(1) www.yahoo.finance.com, (2) www.stockwatch.in (3)
Financial Times (www.ft.com) and (4) Reuters
(www.investools.com).
The information that Jaybhay collected from yahoo/finance
is available all the time in comma separated value (.CSV)
files, and includes the daily prices of stocks. He also collected
the daily published news (unstructured, unlabeled text data) of
the Bombay Stock Exchange that came from three web sites:
www.finance.yahoo.com, reuters.com and
www.stockwatch.in.
The big data portion of this is revealed in how Jaybhay
handles the text data. First step for Jaybhay was
preprocessing: text data is preprocessed in three steps: (1) by
removing stop words (such as a, an, the, of, etc,) (2) by
stemming, which is, to reduce a word to its stem – such as
trimming the word “selling” to “sell” and (3) harvesting
keyword phrases against a global list of top keyword phrases.
The top 12 keyword phrases were: hike, jump, flat, buy, loses,
down, lower, steady, recession, scam, swoon, slump. The data
was then classified as 0 or 1, to indicate upward trend or
downward trend information as shown by these keywords and
stem words.
Jaybhay then used a neural network that took in the input
values of the stock prices and the input of the presence or
absence of the key phrases extracted from the news text of that
day, and the output value as the closing price of the stock of
that day.
In a test of 543 stocks, using the Jaybhay method he
predicted if the next day’s closing stock price would be an
uptrend or a downtrend correctly 481 times, which is a success
rate of 88%.
Jaybhay’s advice for future work is to develop better
preprocessing methods for news text “more work on refining
key phrases extraction will definitely produce better results.”
In a very similar way that Jaybhay uses for the Bombay
stock market, Gunduz studies the stock market of Turkey.
Gunduz [18] applies the mining of unstructured text data,
combined with the numerical information of the prices, to the
stock market in Istanbul, Turkey. He successfully predicts
68% of the time the direction that the Borsa Istanbul 100
Index (BIST100) prices move on opening. He [19]is using
two inputs to his model: (1) text data from news articles
released and (2) the price the day before. What he did was
new, because Turkish news articles – Turkish language text
mining – had not been used before, and yet, they influence the
Turkish stock market. “Factors like the political situation,
economic conditions, companies future targets, investors’
expectations, global stock exchanges and psychology of
investors influence the stock market behavior and make it hard
to predict the future stock price or directions.” [18]
Gunuz pointed out that until recently, analysis methods to
predict stock prices were one of two types: (1) technical
analysis that uses historic stock prices, and (2) fundamentals
analysis, that uses economic data such as demand for a
company’s products, inflation, interest rates, unemployment
rate, and trading volume. [19] He developed a third way, to
include news articles in the data that is used to influence the
outputs.
The news articles of text came from three sources: (1) the
“official news” source from the Turkish Publish Disclosure
Platform (http://www.kap.gov.tr) which has digitally signed
documents such as financial statements and revenue reports.
The other two are internet news from finance web portals, (2)
Mynet Finans (http://finans.mynet.com) and (3) Bigpara
(http://www.bigpara.com).
The text data from these three sites were preprocessed by
following these steps: (1) stop word removal (taking out of
the text the words with meanings like a, of, as, the), (2)
stemming (shorting words to their stem, such as the Turkish
root word for “go” is “git” so he goes, she went, another goes,
he will go, those words were all shorted from gitti, gittim,
gidecek, giden, to git. This stemming was done using a
program written in 2007 in Java, the Zemberek Turkish
Language Processing Framework. (3) filtering out words that
occurred less than 1000 times in the document collection (4)
this resulted in a Bag Of Words (BOW) of 2888 unique words,
each of which were then rated as positive (value of 1) and
negative (value of 0) impact on the stock price.
Then all of the text information was grouped by news day
and associated with the movement of the stock price from the
close of one day to the opening of the next day.
Gunduz divided all of his data into two parts, one of 18
months of data to use for the training set, and the other was of
6 months of data in their test set.
Gunduz found that the Official News information did not
work well, because Turkish traders buy or sell that day that the
Official News is published, so this did not help his two day
trading day model. However, the Internet news was of
significant value in predicting the direction of the stock price
the following day.
For future, Gunduz thinks that using the prices of other
world stock markets such as US, German, and Japanese, could
be used to improve performance as well as using text mining
that used both English words as well as Turkish words.
FUTURE WORK AND CONCLUSIONS
Two conclusions come from this survey and one clear
future work direction. First, text mining the news and
combining that factor with the numerical data from the stock
market to predict the market will spread to other countries and
other markets beyond the English and Turkish language and
linguistics will become a hot research area as more
sophisticated methods of text mining will be discovered. This
will result in robot assisted management of portfolios in all
world stock markets.
The second conclusion is that the advances in the stock
market algorithms and information processing speeds will lead
to advances in other scientific research areas as some of those
advances will find useful purposes in new applications far
different than the original.
The new direction for future work will be to combine the
unstructured text data in Bloomberg News reporting with the
stock prices of the New York stock market exchange to
develop a combination predictive model, furthering the work
done in the Bombay and Turkish stock markets.
IV. REFERENCES
[1] White House, "Big Data: Seizing Opportunities, Preserving Values,"
1 May 2014. [Online]. Available:
https://www.whitehouse.gov/sites/default/files/docs/big_data_privacy
_report_may_1_2014.pdf. [Accessed 20 October 2015].
[2] J. Bughin, M. Chui, and J. Manyika, "Clouds, Big Data, and Smart
Assets: Ten Tech-Enabled Business Trends to Watch," McKinsey
Quarterly, August 2010. [Online]. Available:
http://www.mckinsey.com/insights/high_tech_telecoms_internet/clou
ds_big_data_and_smart_assets_ten_tech-
enabled_business_trends_to_watch. [Accessed 20 Oct 2015].
[3] Federal Reserve, "Guidance on Model Risk Management,"
Supervision and Regulation (SR) 11-7 of Federal Reserve System, 4
April 2011. [Online]. Available:
http://www.federalreserve.gov/bankinforeg/srletters/sr1107.htm.
[Accessed 1 Nov 2015].
[4] Subhabrata Choudhury, Subhajyoti Ghosh, Arnab Bhattacharya,
Kiran Jude Fernandes, and Manoj Kumar Tiwari "A real time
clustering and SVM based price-volatility prediction for optimal
trading strategy," Journal of Neurocomputing, vol. May, pp. 419-426,
2014.
[5] J. Vesanto and E. Alhoniemi, "Clustering of the self-organizing map,"
IEEE Transactions of Neural Networks, vol. 11, no. 3, pp. 586-600,
2000.
[6] V. N. Vapnik, The Nature of Statistical Learning Theory, 2nd Ed.,
New York: Springer, 1995.
[7] Ash Booth, Enrico Gerding, and Frank McGroaty, "Automated
trading with performance weighted random forests and seasonality,"
Expert Systems With Applications, An International Journal, vol.
June, pp. 3651-3661, 2014.
[8] Michel Ballings, Dirk Van den Poel, Nathalie Hespeels, and Ruben
Gryp "Evaluating multiple classifiers for stock price direction
View publication stats
prediction," Journal of Expert Systems with Applications, vol.
November, pp. 7046-7056, 2015.
[9] Hsuan-Ming Feng and Hsiang-Chai Chou, "Evolutional RBFNs
prediction systems generation in the application of financial time
series data," Journal of Expert Systems with Applications, vol. 38, no.
7, pp. 8285-8292, 2011.
[10] Gavor Tatai, Laszlo Gulyas, Laszlo Laufer, Marton Ivanyi
"vBroker: agents teaching stock," Proceedings of 5th International
Working Conference of Intelligent Virtual Agents (IVA), vol.
September, p. 503, 2005.
[11] Todd Feldman and Daniel Friedman, "Human and Artificial Agents in
a Crash Prone Financial Market," Journal of Computational
Economics, vol. 36, no. 3, pp. 201-229, 2010.
[12] Jameela Al-Jaroodi, Nader Mohamed and K. Al-Nuaimi, "An
efficient algorithm for temporal financial information monitoring,"
Middleware Technologies Lab, Bahrain, UAE, 2013.
[13] C. Augusto Casas, "Parallelization of artificial neural network training
algorithms: a financial forecasting application," IEEE Conference on
Computational Intelligence for Financial Engineering and
Economics, pp. 1-6, 2012.
[14] IBM, "What is Big Data," IBM, 2015. [Online]. Available:
http://www-01.ibm.com/software/data/bigdata/what-is-big-data.html.
[Accessed 5 Nov 2015].
[15] Xindong Wu, Xingquan Zhu, Gong-Qing Wu, and Wei Ding "Data
mining with big data," IEEE Transactions on Knowledge and Data
Engineering, vol. 26, no. 1, pp. 97-107, 2014.
[16] J. Bollen, H. Mao, and X. Zeng, "Twitter Mood Predicts the Stock
Market," Journal of Computational Science, vol. 2, pp. 1-8, 2011.
[17] Kranti M. Jaybhay, Rajesh V. Argiddi, and S.S. Apte, "Stock market
prediction model by combining numeric and news textual mining,"
International Journal of Computer Applications, vol. 57, no. 19,
2012.
[18] Hakan Gunduz and Zehra Cataltepe, "Borsa Istanbul (BIST) daily
prediction using financial news and balanced feature selection,"
International Journal of Expert Systems with Applications, vol. 42,
no. 22, pp. 9001-9011, 2015.
[19] G. Gidofalvi and E. Elkan, "Using news articles to predict stock price
movements," Department of Computer Science and Engineering,
University of California, San Diego, 2001.