STOCK MARKET

PREDICTION USING
LSTM AND BERT

MINI PROJECT REPORT

Submitted by

SRI VAISHNAVI K 727622BAD054

SRINIVASAN N 727622BAD068
PRIENGA S 727622BAD074

in partial fulfilment for the award of the

degree of

BACHELOR OF TECHNOLOGY

in

ARTIFICIAL
INTELLIGENCE AND
DATA SCIENCE

i
Dr. MAHALINGAM COLLEGE OF
ENGINEERING AND
TECHNOLOGY
An Autonomous Institution
Affiliated to ANNA
UNIVERSITY CHENNAI - 600
025
MAY-2023

i
 Dr. MAHALINGAM COLLEGE OF ENGINEERING
AND TECHNOLOGY, POLLACHI -642 003

(An Autonomous Institution Affiliated to Anna University, Chennai - 600 025)

BONAFIDE CERTIFICATE

Certified that this Mini project report titled “STOCK MARKET PREDICTION
USING LSTM AND BERT” is the bonafide work of

SRI VAISHNAVI K (727622BAD054)

SRINIVASAN N (727622BAD068)
PRIENGA S (727622BAD074)

who carried out the mini project under my supervision.

Dr. N. SUBA RANI Dr. N. SUBA

Supervisor RANI
Head of the department(i/c)
Associate Professor &HOD/IC
Department of AI&DS Associate Professor
Dr. Mahalingam college of Department of AI&DS
Engineering and Technology,
Dr. Mahalingam college of
NPT-MCET campus
Engineering and Technology
Pollachi- 642003 India
NPT-MCET campus
Pollachi- 642003 India

ii
Submitted for the Autonomous End Semester Examination Project Viva-voce
held on

INTERNAL EXAMINER EXTERNAL EXAMINER

iv
 ACKNOWLEDGEMENT

First and foremost, we wish to express our deep unfathomable feeling,

gratitude to our institution and our department for providing us a chance to fulfil our

long cherished dreams of Department of Artificial Intelligence and Data Science.

We express our sincere thanks to our honorable Secretary Dr. C.

Ramaswamy for providing us with required amenities.

We wish to express our hearty thanks to Dr. P. Govindasamy,

Principal and Deans of our college, for his constant motivation and continual

encouragement regarding our projectwork.

We are grateful to Dr. N. Suba Rani, Head of the Department

Incharge, Artificial Intelligence and Data Science, for her direction delivered at all times

required. We also thankher for her tireless and meticulous efforts in bringing out this

project to its logical conclusion.

Our hearty thanks to our guide Dr.M.Balakrishnan , Associate

Professor/AD for her constant support and guidance offered to us during the course of

our project by being one among us and all the noble hearts that gave us immense

encouragement towards the completion of ourproject.

We also thank our project coordinators Dr.M.Balakrishnan ,

AP(SS) & Mrs. Thelungu Laxmi AP for their continuous support and guidance.

v
 ABSTRACT

Abstract: The stock market is a venue for people to buy and sell their stocks. This
background involves risk and the goal is to maximize the profit and minimize the
losses to maximum extent. And to make this possible they try to predict stock
prices but, predicting the stock prices is a hectic job and involves large amount of
risk. But this can be uncovered by using some deep learning and machine learning
techniques. The techniques include the machine learning’s BERT and deep
learning’s LSTM. In this paper we suggest a hybrid model of the LSTM (Long
Short Term Memory) and BERT (Bi-directional Encoder Representations from
Transformers). BERT model is a Transformer based model and is able to
understand the contextual meaning of the data and capture complex relationships
and LSTM model is a type of RNN (Recurrent Neral Network) that is commonly
used for time series forecasting. LSTM has the potential to capture the long-term
dependencies in the data making it suitable for analyzing and predict stock market
trends. The hybrid model of BERT and LSTM combines the strengths of both the
model to improve the accuracy in stock market price prediction. Values that
evolve in the data are open, close ,high ,low and volume .

Keywords: Stock market prediction, Bi-directional Encoder Representations from

Transformers, Long Short Term Memory, Recurrent Neral Network, open, close
,high ,low and volume.

vi
 TABLE OF CONTENTS

CHAPTER TITLE PAGE NO

NO
ABSTRACT iv
LIST OF FIGURES vi

LIST OF ABBREVATIONS vii

1 INTRODUCTION 1

2 LITERATURE SURVEY 3

3 PROBLEM STATEMENT 6
4 OBJECTIVE OF THE PROJECT 7

5 EXISTING MODEL 8
6 PROPOSED MODEL 10

7 SOFTWARE DESCRIPTION 14

8 RESULT AND DISCUSSION 15

9 CONCLUSION 16
1 FUTURE SCOPE 18
0

1 REFERENCES 19
1

1 APPENDIX 20
2
A. SOURCE CODE

B. SCREENSHOT 25

C. COURSE COMPLETION 26
CERTIFICATES

vi
D. CONTEST PARTICIPATION 30

E. PLAGIARISM REPORT 34

vi
 LIST OF FIGURES

FIGURE TITLE PAGE NO

NO
5.1 EXISTING MODEL DIAGRAM 9

6.1 PROPOSED MODEL DIAGRAM 10

ix
 LIST OF ABBREVIATIONS

LSTM Long Short-Term Memory

BERT Bidirectional-Encoder
Representations from
Transformers

NLP Natural Language Processing

RNN Recurrent Neural Network

x
 CHAPTER 1
INTRODUCTION

The stock market is a challenging environment with prices depending on different

factors such as economic indicators, company performance, industry trend,
geopolitical events, and investor mood. Therefore, predicting stock prices and trends
is essential for investors, financial analysts, and stakeholders to take the right
position, distribute resources, and minimize risk. Yet, the high level of
unpredictability and volatility of the stock market make this task complicated.
Traditionally, there are two ways to predict the stock market: fundamental analysis
and technical analysis. The former focuses on evaluating the financial health of a
company by its earning, revenue, asset, liabilities, management as well as general
business circumstances. In contrast, technical analysis forecasts future fluctuations
in stock prices by utilizing past price and volume trends. Recent developments in
machine learning and artificial intelligence have completely changed how stock
market analysis is carried out. Large data sets can be processed using machine
learning algorithms, which can also reveal patterns that conventional analysis can
find challenging to identify. Among these algorithms, Bidirectional Encoder
Representations from Transformers (BERT) and Long Short-Term Memory (LSTM)
networks have drawn interest due to their special qualities. Recurrent neural
networks (RNNs) with long-term dependencies can keep their dependencies when
working with sequential input thanks to a specialized kind of RNN called LSTM
networks. Because of this, LSTM is perfect for evaluating time series data, like stock
prices, where historical values can have a big impact on trends in the future.
Financial forecasting, natural language processing, speech recognition, and other
fields have all benefited from the effective use of LSTM. BERT is a transformer-
based approach intended for natural language processing (NLP), in contrast. It is an
effective tool for assessing news stories, sentiment on social media,
1
and other textual data since it has been pre-trained on big datasets and can
comprehend

2
contextual linkages inside text. Considering the impact of sentiment and news on
stock prices, BERT can offer insightful information that enhances more
comprehensive prediction models. A special advantage in stock market prediction is
provided by combining LSTM and BERT. BERT can handle the qualitative
component by extracting sentiment, whereas LSTM concentrates on the quantitative
side by evaluating past stock prices and other numerical data. and background from
written materials. With a wider range of factors influencing stock prices captured,
projections may be more accurate as a result of this combination. With the help of
both LSTM and BERT, this study seeks to create a hybrid model that can forecast
stock market movements. The strategy will evaluate textual data, such as financial
news and social media sentiment, to comprehend external factors that can effect
stock prices. Meanwhile, LSTM will be used to analyze consecutive stock price
data, finding patterns and trends. We intend to combine these two models to produce
a reliable and efficient stock market forecast system. If this project is completed
successfully, investors and financial institutions may benefit practically by having a
more precise tool for estimating stock prices and selecting investments. This project
also intends to demonstrate the potential of hybrid models in stock market
prediction, contributing to the ongoing research in the field of financial machine
learning.

3
 CHAPTER 2
LITERATURE SURVEY

"Optimizing LSTM for stock market prediction in India stock market "
Authors: Anita Yadav, C K Jha, Aditi Sharan . Published: 2019
An LSTM model was designed and a dataset sourced from the Indian stock market
was created for this study. Then, by contrasting stateful and stateless models and
adjusting for the number of hidden layers, it was optimized.

"Stock Price Prediction Using Machine Learning and LSTM-Based Deep

Learning Models" Authors: Sidra Mehtab, Jaydip Sen and Abhishek Dutta.
Published: 2020
This paper proposes hybrid modeling, which combines deep learning and machine
learning to develop several models for stock price prediction. The analysis uses the
NIFTY 50 index values from the National Stock Exchange (NSE) of India for the
period of December 29, 2014, to July 31, 2020. Using training data comprised of
NIFTY 50 index records from December 29, 2014, to December 28, 2018, eight
regression models were constructed. These regression models are used to forecast the
open values of NIFTY 50 for the months of December 2018 through July 2020. The
grid-searching technique is used to optimize the LSTM models' hyperparameters,
ensuring that validation losses stabilize as the number of epochs increases and that the
convergence of the Accurate validation is accomplished. Utilizing four distinct
models with varying architectures and input data structures, the power of LSTM
regression models is harnessed to estimate future NIFTY 50 open values.
Comprehensive outcomes are displayed using diverse indicators for every regression
model. The findings unequivocally show that the most accurate model is the LSTM-
based
4
univariate model, which predicts the open value of the NIFTY 50 time series for the
following week using data from one week ago as input.

"BERT for Stock Market Sentiment Analysis" Authors: Matheus Gomes de

Sousa, Kenzo Sakiyama and Lucas de Souza Rodrigues. Published: 2020
This paper suggests using Bidirectional Encoder Representations from Transformers
(BERT) to analyze news articles' emotion and offer pertinent data for stock market
decision-making. Through a self-learning task, a huge number of general-domain
papers are used to pre-train this model. In order to optimize this robust sentiment
analysis algorithm for the stock market, news stories on stocks are manually
classified as either positive, neutral, or negative. This dataset, which includes 582
documents from several financial news sources, is freely accessible. Using this
dataset, we refine a BERT model and obtain 72.5% of F score. Then, some tests are
carried out to demonstrate how the output of the developed model might offer useful
information to forecast the Dow Jones's future moves.

"FinBERT-LSTM: Deep Learning based stock price prediction using News

Sentiment Analysis" Authors: Shayan Halder. Published: 2022
This paper uses deep learning networks to anticipate stock values by ingesting news
articles about technology, business, and finance that provide market data. Initially, a
basic Multilayer Perceptron (MLP) network is developed, which is subsequently
extended into a more intricate Recurrent Neural Network (RNN) such as Long Short
Term Memory (LSTM). Ultimately, the FinBERT-LSTM model is suggested, which
incorporates sentiments from news articles to more accurately predict stock price by
examining short-term market data. Then, using mean absolute error (MAE), mean
absolute percentage error (MAPE), and accuracy measures, the model is trained on

5
data from the NASDAQ-100 index stocks and news items from the New York
Times to assess the performance of the MLP, LSTM, and FinBERT-LSTM models.

"LSTM-based sentiment analysis for stock price forecast" Authors: Ching-Ru

Ko and Hsien-Tsung Chang Published: 2022
The fundamental analysis in this paper is derived on news articles and PTT forum
discussions, while the technical analysis is based on historical stock transaction data.
The long short term memory neural network (LSTM), which excels at analyzing
time series data, is used to forecast the stock price using stock historical transaction
information and text sentiments. The cutting-edge natural language processing tool
BERT is used to identify the sentiments of text. Based on the outcomes of our
proposed models' experiments, the average root mean square error (RMSE) has an
accuracy improvement of 12.05.

6
 CHAPTER 3
PROBLEM STATEMENT

Predicting the performance of the stock market is difficult because of its inherent
volatility and the intricate relationships between its many affecting elements. A
variety of factors, including business performance, industry developments,
geopolitical events, investor attitude, and economic indicators, influence stock
prices. Conventional prediction techniques, such technical and fundamental analysis,
frequently depend on human interpretation and may have biases or scope
restrictions. Furthermore, these techniques might not be able to capture the complex
patterns and abrupt changes in the market that result from breaking news, opinions
expressed on social media, or worldwide trends. The main issue is the requirement
for a reliable prediction model that can take into consideration the qualitative (like
financial news, market sentiment, and economic developments) as well as the
quantitative (like historical price trends, trading volumes, and technical indicators)
aspects of stock market data. To tackle this issue, a method that blends sentiment
and news context with sequential analysis of stock prices is needed. While useful in
certain situations, current machine learning models frequently concentrate on one
kind of data qualitative or quantitative—which results in imprecise or insufficient
predictions. Accurate stock market predictions need to be made using a hybrid
technique that incorporates both kinds of data. In order to close this gap, this
research is creating a model that mixes Bidirectional Encoder Representations from
Transformers (BERT) is used for text-based sentiment analysis, and Long Short-
Term Memory (LSTM) networks are used for time series analysis.
7
 CHAPTER 4
OBJECTIVE OF THE
PROJECT

The principal aim of this project is to create a hybrid stock market prediction model
that enhances the accuracy of stock price forecasts by integrating LSTM and BERT.
The following are some ways that the model hopes to take advantage of both LSTM
and BERT's strengths:
Sequential Analysis with LSTM: Apply LSTM to examine past stock prices and
other numerical data so that long-term relationships can be captured and patterns or
trends can be recognized. This part addresses the time series character of stock prices
and concentrates on the quantitative side of stock market prediction.
Contextual Understanding with BERT: Employ BERT to evaluate textual
information from press releases, social media posts, and financial news stories in
order to determine market sentiment and significant events that may have an effect on
stock prices. This part concentrates on the qualitative side, including more
background information and comprehension of outside variables affecting stock
values.

8
 CHAPTER 5
EXISTING MODEL

Existing Model: Recurrent Neural Networks (RNN) in Stock Market Prediction

One kind of neural network intended for processing sequential input is the recurrent
neural network (RNN). Because RNNs feature internal loops that preserve
information, they are more appropriate for jobs involving time sequences or data that
has inherent order than typical feedforward neural networks. Because of this
property, RNNs are a well-liked option for stock market prediction, where past price
trends and other sequential data are important factors in predicting future
movements.By saving outputs from earlier time steps and transmitting them back
into the network, RNNs preserve a kind of memory. RNNs are able to adjust to
changes in sequential data and learn patterns over time thanks to this feedback loop.
RNNs are able to capture the time-dependent nature of stock values, which are
frequently impacted by past trading session occurrences.
Limitations of RNN:
Vanishing Gradient Problem: RNNs may have difficulties with the "vanishing
gradient" problem as sequences lengthen. This issue arises when the gradients
utilized for learning tend to decrease, resulting in the loss of information about
previous time steps.
Long-Term Dependency Capture Difficulty: Despite RNNs' ability to store
knowledge, they may struggle to capture long-term dependencies over protracted
periods of time, which could influence the accuracy of predictions for longer-term
trends.
Incapacity to Process Non-Sequential Data: RNNs are less effective in processing
contextual information such as financial news, market sentiment, or other qualitative
elements because they are built to process sequential data.
9
These drawbacks imply that RNNs might not fully account for all the variables
affecting stock prices. Because of this gap, improved RNN variations have been
created, namely Long Short-Term Memory (LSTM) networks, which solve some of
the problems with conventional RNNs.

FIGURE:5.1

1
 CHAPTER 6
PROPOSED SYSTEM

FIGURE 6.1 PROPOSED SYSTEM

1
DATA COLLECTION: Gathering historical stock prices, trade volumes, and
technical indicators is known as technical data collection. While textual data is
gathered from news stories, social media, and other sources that could impact stock
values, data usually consists of daily or hourly prices.
DATA PREPROCESSING: Technical Data Preprocessing: Creating time-series
sequences for LSTM, addressing missing values, and normalizing stock price data to
a consistent scale. Tokenizing and cleaning the textual data are steps in the textual
data preprocessing process. Typical procedures involve eliminating special
characters, stopwords, and tokenizing text to create BERT tokens or subwords.
MODEL DEVELOPMENT: To process the technical data, an LSTM model is
built by the LSTM Model portion of the code. One or more LSTM layers may be
part of the architecture, with dense (completely connected) layers coming next to
produce predictions.Initializing a BERT model to process textual data is covered in
the BERT Model section. Usually, the BERT model is optimized for contextual
understanding or sentiment analysis.
INTEGRATION: The results from the BERT and LSTM models are combined in
this stage. To generate the stock market prediction, the combined output is sent into
a final prediction layer (like a dense layer).
MODEL TRAINING: Using the training data, the combined model is trained
during the training process. It entails defining the optimizer, loss function, and
training parameters, such as epochs and batch size. To keep an eye on performance
and prevent overfitting, the model is verified against a validation dataset throughout
training.
RESULT AND ANALYSIS: The outcome is anticipated, and the model is assessed.
To overcome the shortcomings of previous models, this project's suggested model
mixes Long Short-Term Memory (LSTM) networks with Bidirectional Encoder

1
Representations from Transformers (BERT). To provide a more thorough approach

1
to stock market prediction, the suggested model integrates both qualitative and
quantitative data.

LSTM Component

The LSTM component of the model focuses on analyzing historical stock price data,
capturing long-term dependencies, and identifying patterns in price movements.
LSTM networks are well-suited for time series data, as they can retain memory of
past states and adapt to new patterns. In this project, LSTMs are used to analyze
various stock-related metrics, including closing prices, trading volumes, and
technical indicators, to generate predictions based on historical trends.

BERT Component
The BERT component addresses the qualitative aspect of stock market prediction by
processing textual data from financial news articles, social media, and other relevant
sources. BERT, a transformer-based model, is pre-trained to understand context and
relationships within text, allowing it to extract sentiment and key information from
textual sources. This component is crucial for capturing broader contextual factors
that might impact stock prices, such as economic news, company announcements, or
geopolitical events.
Integration and Prediction
To produce stock market predictions, the suggested model combines the results from
the BERT and LSTM components. The goal of merging quantitative and qualitative
data is to enhance the accuracy and resilience of the model. Combining the
contextual insights from BERT with the sequential patterns found by LSTM is the
integration process. With a wider range of parameters taken into account, the model
may be able to anticipate stock prices more accurately as a result of this
combination. The suggested model aims to address the shortcomings of the current

1
models by employing

1
this hybrid approach, offering a more thorough and dependable technique for stock
market prediction. The goal of the experiment is to show how combining LSTM and
BERT can increase prediction accuracy and provide fresh perspectives on stock
market fluctuations.

1
 CHAPTER7
SOFTWARE DESCRIPTION

 Datasets of real stock market.

 Python 3 or above.
 Sypder 5.0 version or above.
 Google collaboratory. Laptop or pc over intel i5 11th gen processor for fast
processing.
The following tools and technologies are used in the project:
Programming Language: Python is renowned for having a robust machine learning
and data science library environment.
Deep Learning Frameworks: The LSTM and BERT models were constructed and
trained using TensorFlow and PyTorch.
Data Manipulation: NumPy and Pandas for manipulating and preparing data.
Natural Language Processing: Additional NLP tools for text processing, as well as
the Hugging Face Transformers library for BERT implementation.
Web Scraping and APIs: Financial APIs for technical data; Beautiful Soup,
Scrapy, or other scraping tools for textual data collection.
Visualization: To visualize model performance and trends in stock prices, use
Seaborn with Matplotlib.

1
 CHAPTER 8

RESULTS AND DISCUSSION

The stock market prediction project using Long Short-Term Memory (LSTM)
networks and Bidirectional Encoder Representations from Transformers (BERT) is
presented in this part. This project's main goal was to increase the accuracy of stock
market predictions by combining technical and qualitative (textual) data sources.
The hybrid model, which combines BERT and LSTM, was trained using text from
financial news and social media as well as historical stock price data. The hybrid
model fared better than the traditional models that employed various conventional
machine learning techniques or just LSTM. This suggests that stronger prediction
skills can be achieved by combining BERT for text-based context and sentiment
analysis with LSTM for time-series analysis. Lower values for the average error
metrics (MSE, RMSE) suggested a better match to the test data. By using BERT to
integrate textual input, the model was able to take market sentiment and pertinent
news events into account, adding context to the predictions. The model's overall
performance was enhanced by this innovation, which also revealed information
about outside variables affecting stock prices. The LSTM analysis of the technical
data enhanced the model's resilience to time-series data by identifying long-term
relationships and sequential patterns in stock prices. The hybrid model was
contrasted with more conventional approaches, such as LSTM models without
textual data and simple models. The hybrid model continuously demonstrated
improved forecast precision and flexibility in response to shifting market
circumstances. This comparison emphasizes how crucial it is to combine qualitative
and quantitative data in order to make accurate stock market predictions.

15
 CHAPTER 9
CONCLUSION

In summary, the goal of our project was to anticipate stock price changes by
analyzing stock market data using LSTM (Long Short-Term Memory) and BERT
(Bidirectional Encoder Representations from Transformers) models. Several
significant revelations and conclusions have come from our investigation.First, the
efficiency of LSTM in capturing temporal relationships in stock price data was
shown. For short-term forecasting, its capacity to retain long-term patterns and apply
them to forecasts proved useful. We achieved competitive predictive performance by
using historical stock price data to train LSTM models.Second, by adding textual data
like financial news and stories, integrating BERT into our study gave our analysis a
fresh perspective. The models' comprehension of the sentiment and context
surrounding stock movements was improved by BERT's pre-trained contextual
knowledgeAn in-depth investigation is provided here:Improved Model Tuning: The
performance of LSTM and BERT models can be greatly improved by further
adjusting hyperparameters and architectures.It is possible to find ideal configurations
by using methods like grid search, random search, or Bayesian
optimization.Combining predictions from several LSTM and BERT models can be
done through the use of ensemble techniques like stacking and blending, or ensemble
learning.In doing so, you can strengthen the forecasts and lessen the impact of the
individual model flaws. The process of feature engineeringBetter model performance
can be achieved by adding new features that are obtained from textual and numerical
data.Further complex information can be captured by using sentiment ratings, event-
based characteristics, or technical indicators.Look at dynamic time series forecasting
techniques to help models adjust to shifting market

1
conditions. Temporal Attention Mechanisms: In LSTM models, employ temporal
attention mechanisms to direct attention toward more pertinent past data points.
In making forecasts, this might assist the model in highlighting significant time periods
and patterns.

1
 CHAPTER 10
FUTURE SCOPE

There are a lot of opportunities for future research and development in the field of
stock market analysis employing LSTM and BERT projects.
Model Architecture Fine-Tuning: To increase the efficiency of LSTM and BERT
designs in recognizing intricate patterns and comprehending textual input, they need
be continuously improved.
Try out different LSTM implementations, like stacked LSTMs, attention-based
LSTMs, or hybrid architectures that combine LSTMs with other convolutional or
recurrent layers.
Examine BERT variants such as pre-trained models tailored to a certain domain or
models optimized for financial language.
Combining Data from Multiple Modes:
Integrate data from sources other than stock prices and financial news, such social
media emotions on the market, economic indicators, or alternative data like site
traffic or satellite imagery.
Provide strategies for integrating various data modalities, such as textual, numerical,

1
 CHAPTER 11
REFERENCES

1. Sidra Mehtab, Jaydip Sen, Abhishek Dutta: Stock Price Prediction Using Machine Learning
and LSTM-Based Deep Learning Models, Springer, 2021.
2. Greff, Klaus, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, and Jürgen
Schmidhuber. (2017) “LSTM: A Search Space Odyssey.” IEEE Transactions on Neural
Networks and Learning Systems 28 (10): 2222 – 2232
3. Sherstinsky, A. (2020), “Fundamentals of recurrent neural network (RNN) and long short-
term memory (LSTM) network”, Physica D: Nonlinear Phenomena, Vol. 404, 132306.
4. K. A. Althelaya, E. M. El-Alfy and S. Mohammed, "Evaluation of bidirectional LSTM for
short-and long-term stock market prediction," 2018 9th International Conference on
Information and Communication Systems (ICICS), Irbid, 2018, pp. 151-156
5. Devlin J, Chang M-W, Lee K, Toutanova K. 2018. Bert: pre-training of deep bidirectional
transformers for language understanding. ArXiv preprint. arXiv:1810.04805
6. Venkata Sasank Pagolu, Kamal Nayan Reddy Challa, Ganapati Panda, Babita Majhi:
Sentiment Analysis of Twitter Data for Predicting Stock Market Movements
7. M. Alsulmi: Reducing Manual Effort to Label Stock Market Data,Riyadh 11451, Saudi
Arabia,2021
8. M. Nabipour et al.: Predicting Stock Market Trends Using Machine Learning and Deep
Learning Algorithms,2020
9. Willem Sebastian,Balamurugan G,International Journal of Emerging Trends in Engineering
Research, 8(9), September 2020, 5430 – 5433
10. Shayan Halder:FinBERT-LSTM: Deep Learning based stock price prediction using News
Sentiment Analysis,2022
11. Karlo Puh and Marina Bagic Babac:Predicting stock market using natural language
processing,American Journal of Business Vol. 38 No. 2, 2023 pp. 41-61
12. Rouf, N.; Malik, M.B.; Arif, T.; Sharma, S.; Singh, S.; Aich, S.; Kim, H.-C. Stock Market
Prediction Using Machine Learning Techniques: A Decade Survey on Methodologies,
Recent Developments, and Future Directions. Electronics 2021

1
 CHAPTER 12
APPENDIX
A) SOURCE CODE:

pip install pandas numpy matplotlib sklearn tensorflow transformers

keras import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from transformers import BertTokenizer
stock_data = pd.read_csv("stock3.csv")
news_data = pd.read_csv("news_3.csv", encoding='latin-1')
merged_data = pd.merge(stock_data, news_data, on="Date", how="inner")
merged_data.dropna(inplace=True)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
tokenized_headlines = [tokenizer.encode(text, add_special_tokens=True, max_length=128,
truncation=True) for text in merged_data['headline']]
max_length = 128
from keras.preprocessing.sequence import pad_sequences
X_lstm = pad_sequences(tokenized_headlines, maxlen=max_length, dtype='int32', padding='post')
y_lstm = merged_data['Close'].valuesA
X_train, X_test, y_train, y_test = train_test_split(X_lstm, y_lstm, test_size=0.2, random_state=42)
from keras.models import Sequential
from keras.layers import LSTM, Dense, Dropout, Embedding
model_lstm = Sequential()
model_lstm.add(Embedding(input_dim=len(tokenizer.vocab), output_dim=128,
input_length=max_length))
model_lstm.add(LSTM(units=50, return_sequences=True))
model_lstm.add(Dropout(0.2))
model_lstm.add(LSTM(units=50))
model_lstm.add(Dropout(0.2))
model_lstm.add(Dense(1))
model_lstm.compile(loss='mean_squared_error', optimizer='adam')
2
model_lstm.fit(X_train, y_train, epochs=10, batch_size=32, validation_data=(X_test,
y_test)) from transformers import TFBertModel
import tensorflow as tf
bert_model = TFBertModel.from_pretrained('bert-base-uncased')
bert_embeddings = bert_model.predict(X_test)
from tensorflow.keras.layers import Input, Concatenate, Dense
from tensorflow.keras.models import Model
import tensorflow as tf
print(type(bert_embeddings))
print(dir(bert_embeddings))
bert_embeddings_last_hidden_state = bert_embeddings.last_hidden_state
bert_embeddings_reshaped = tf.reshape(bert_embeddings_last_hidden_state, (-1,
bert_embeddings_last_hidden_state.shape[-1]))
input1 = Input(shape=(bert_embeddings_reshaped.shape[1],))
input2 = Input(shape=(X_lstm.shape[1],))
concatenated = Concatenate()([input1, input2])
dense_layer = Dense(64, activation='relu')(concatenated)
output = Dense(1, activation='linear')(dense_layer)
model = Model(inputs=[input1, input2], outputs=output)
model.compile(loss='mean_squared_error', optimizer='adam')
print(f"X_lstm shape: {X_lstm.shape}")
print(f"y_train shape: {y_train.shape}")
print(f"bert_embeddings_reshaped shape:
{bert_embeddings_reshaped.shape}") import tensorflow as tf
from tensorflow.keras import layers
from tensorflow.keras.models import Model
print(X_lstm.shape)
print(y_train.shape)
import numpy as np
y_train_2d = np.reshape(y_train, (-1, 1))
print("Original shape of y_train:", y_train.shape)
print("Reshaped shape of y_train:", y_train_2d.shape)
2
print(X_lstm.shape)
print(y_train_2d.shape)
import pandas as pd
import numpy as np
from transformers import BertTokenizer
from keras.preprocessing.sequence import
pad_sequences stock_data = pd.read_csv("stock3.csv")
news_data = pd.read_csv("news_3.csv", encoding='latin-1')
merged_data = pd.merge(stock_data, news_data, on="Date", how="inner")
merged_data.dropna(inplace=True)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
tokenized_headlines = [tokenizer.encode(text, add_special_tokens=True, max_length=128,
truncation=True) for text in merged_data['headline']]
max_length = 128
X_lstm = pad_sequences(tokenized_headlines, maxlen=max_length, dtype='int32', padding='post')
if X_lstm.shape[0] == 0:
raise ValueError("X_lstm is empty. Check your data or preprocessing steps.")
y_train = merged_data['Close'].values
print("Shape of X_lstm:", X_lstm.shape)
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from keras.layers import Input, Concatenate, Dense
from keras.models import Model
from keras.layers import LSTM, Dense, Dropout, Embedding
from keras.preprocessing.sequence import pad_sequences
from transformers import BertTokenizer, TFBertModel
stock_data = pd.read_csv("stock3.csv")
news_data = pd.read_csv("news_3.csv", encoding='latin-1')
merged_data = pd.merge(stock_data, news_data, on="Date", how="inner")
merged_data.dropna(inplace=True)
2
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
tokenized_headlines = [tokenizer.encode(text, add_special_tokens=True, max_length=128,
truncation=True) for text in merged_data['headline']]
max_length = 128
X_lstm = pad_sequences(tokenized_headlines, maxlen=max_length, dtype='int32', padding='post')
if X_lstm.shape[0] == 0:
raise ValueError("X_lstm is empty. Check your data or preprocessing steps.")
y_train = merged_data['Close'].values
bert_model = TFBertModel.from_pretrained('bert-base-uncased')
headline_texts = [tokenizer.decode(tokens, skip_special_tokens=True) for tokens in
tokenized_headlines]
bert_inputs = tokenizer(headline_texts, return_tensors="tf", padding=True, truncation=True)
bert_outputs = bert_model(bert_inputs)
bert_embeddings = bert_outputs.last_hidden_state.numpy()
headline_texts = [tokenizer.decode(tokens, skip_special_tokens=True) for tokens in
tokenized_headlines]
bert_inputs = tokenizer(headline_texts, return_tensors="tf", padding=True, truncation=True)
bert_outputs = bert_model(bert_inputs)
bert_embeddings = bert_outputs.last_hidden_state.numpy()
headline_texts = [tokenizer.decode(tokens, skip_special_tokens=True) for tokens in X_lstm]
bert_inputs = tokenizer(headline_texts, return_tensors="tf", padding=True,
truncation=True) bert_outputs = bert_model(bert_inputs)
bert_embeddings = bert_outputs.last_hidden_state.numpy()
bert_embeddings_reshaped = np.reshape(bert_embeddings, (bert_embeddings.shape[0], -1))
input1 = Input(shape=(bert_embeddings_reshaped.shape[1],))
input2 = Input(shape=(X_lstm.shape[1],))
concatenated = Concatenate()([input1,
input2])
dense_layer = Dense(64, activation='relu')(concatenated)
output = Dense(1, activation='linear')(dense_layer)
model = Model(inputs=[input1, input2], outputs=output)
model.compile(loss='mean_squared_error', optimizer='adam')
2
history = model.fit(x=[bert_embeddings_reshaped, X_lstm], y=y_train, epochs=10,
batch_size=32, validation_split=0.2)
plt.plot(history.history['loss'], label='Training Loss')
plt.plot(history.history['val_loss'], label='Validation
Loss') plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.title('Training and Validation Loss')
plt.legend()
plt.show()
print("Number of samples in bert_embeddings_reshaped:",
bert_embeddings_reshaped.shape[0]) print("Number of samples in X_test:", X_test.shape[0])
if bert_embeddings_reshaped.shape[0] != X_test.shape[0]:
X_test = X_test[:bert_embeddings_reshaped.shape[0]]
y_test = y_test[:bert_embeddings_reshaped.shape[0]]
print("Number of samples after adjustment:", X_test.shape[0])
bert_embeddings_reshaped = bert_embeddings_reshaped[:X_test.shape[0]]
print("Number of samples after adjustment:", bert_embeddings_reshaped.shape[0])
predicted_prices = model.predict([bert_embeddings_reshaped, X_test])
plt.figure(figsize=(10, 6))
plt.plot(y_test, label='Actual Prices', color='blue')
plt.plot(predicted_prices, label='Predicted Prices', color='red', alpha=0.7)
plt.xlabel('Timestep')
plt.ylabel('Stock Prices')
plt.title('Actual vs Predicted Stock Prices')
plt.legend()
plt.grid(True)
plt.show()

2
B) SCREENSHOTS

2
COURSE COMPLETED

2
2
2
2
 .

CONTEST
PARTICIPATED

3
3
3
3
PLAGIARISM REPORT

3
3
3
3