Stock Price Prediction Based on Deep Learning (2024)

The stock market is known as a place where people can make a fortune if they can crack the mantra to successfully predict stock prices. Though it’s impossible to predict a stock price correctly most the time. So, the question arises, if humans can estimate and consider all factors to predict a movement or a future value of a stock, why can’t machines? Or, rephrasing, how can we make machines predict the value for a stock? Scientists, analysts, and researchers all over the world have been trying to devise a way to answer these questions for a long time now.

In this article, I will try to demonstrate an approach towards so-called Algorithmic Trading. This is a complete research purpose-based approach. Please do not invest based on this algorithm. So, let’s start.

The Whole Idea

A stock price may depend on several factors operating in the current world and stock market. We will try to take into account a combination of mainly two factors:

1. The impact and correlation of stock prices of other companies i.e, how the increase and decrease of stock prices of the other companies affect the stock price of a given target company
2. The past performances and records of the target company

I have seen several blogs that have mostly focussed on one of the factors, mostly 2nd factor. I think if we can manage to bring both these factors into effect, we can make our predictor a bit more robust.

As a result, I have tried to achieve this using a combination of three deep learning models. Firstly, a neural network-based Regressor model which takes into account the impact of other companies on the target company. Secondly, a Recurrent Neural Network Model to study the past behavior of the target company and give results accordingly. For this purpose, I have used an LSTM layer. And, lastly, an Artificial Neural Networks which takes in both their predictions and helps reach a firm and robust conclusion.

Dataset

In this article, we will be using the S&P 500 companies database, though I have used the details of only 200 companies. I have collected the S&P list from the Wikipedia page using web scrapping. I am using this as a source because it is updated live I think. I have seen this method in some blogs, so I also used it. It seems to be a dependable source.

import bs4 as bs
import pickle
import requestsdef save_tickers():
 resp=requests.get('https://en.wikipedia.org/wiki/List_of_S%26P_500_companies')
 soup=bs.BeautifulSoup(resp.text)
 table=soup.find('table',{'class':'wikitable sortable'})
 tickers=[]
 for row in table.findAll('tr')[1:]:
 ticker=row.findAll('td')[0].text[:-1]
 tickers.append(ticker)with open("tickers.pickle",'wb') as f:
 pickle.dump(tickers, f)return tickerssave_tickers()

This code will help you to scrap the tickers of the top 500 companies.

I have then collected their details from Yahoo using the Pandas web-data reader.

import bs4 as bs
import pickle
import requests
import datetime as dt
import os
import pandas as pd
import pandas_datareader.data as webdef fetch_data():
 with open("tickers.pickle",'rb') as f:
 tickers=pickle.load(f)if not os.path.exists('stock_details'):
 os.makedirs('stock_details')
 count=200start= dt.datetime(2010,1,1)
 end=dt.datetime(2020,6,22)
 count=0
 for ticker in tickers:
 if count==200:
 break
 count+=1
 print(ticker)
 try:
 df=web.DataReader(ticker, 'yahoo', start, end)
 df.to_csv('stock_details/{}.csv'.format(ticker))
 except:
 print("Error")
 continuefetch_data()

This snippet of code will help to collect data from Yahoo using the web reader. I have kept a count of 200 because I wanted to use only 200 company details. In these 200 companies, we will have a target company and 199 companies that will help to reach a prediction about our target company.

This code will generate a ‘stock_details’ folder which will have 200 company details from 1st January 2010 to 22nd June 2020.

Each detail file will be saved by its stock’s ticker. I have chosen Amazon as my target stock. So, I will try to predict the stock prices for Amazon. Its ticker is AMZN.

So, let’s see the structure of the detail files.

This is the structure. It has ‘Date’ as the index feature. ‘High’ denotes the highest value of the day. ‘Low’ denotes the lowest. ‘Open’ is the opening Price and ‘Close’ is the closing for that Date. Now, sometimes close values are regulated by the companies. So the final value is the ‘Adj Close’ which is the same as ‘Close’ Value if the stock price is not regulated. ‘Volume’ is the amount of Stock of that company traded on that date.

We will consider this ‘Adj Close’ value of each stock as the contributing feature of each stock on our target stock. So, we will rename the Adj Close of each stock as the corresponding stock ticker and include it in our feature set.

import os
import pandas as pd
import pickledef compile():
 with open("tickers.pickle",'rb') as f:
 tickers=pickle.load(f)main_df=pd.DataFrame()for count,ticker in enumerate(tickers):
 if 'AMZN' in ticker:
 continue
 if not os.path.exists('stock_details/{}.csv'.format(ticker)):
 continue
 df=pd.read_csv('stock_details/{}.csv'.format(ticker))
 df.set_index('Date',inplace=True)df.rename(columns={'Adj Close': ticker}, inplace=True)
 df.drop(['Open','High','Low',"Close",'Volume'],axis=1,inplace=True)if main_df.empty:
 main_df=df
 else:
 main_df=main_df.join(df,how='outer')print(main_df.head())
 main_df.to_csv('Dataset_temp.csv')compile()

This snippet will help us to pick the Adjusted Close column of each stock other than our target stock which is AMZN, rename the column as the ticker and merge it in our feature set.

It will produce a feature set like this. The Date is the index and corresponding to the Date, each ticker’s “Adj Close” value. Now, We will see there are a few empty columns initially. This is because these companies didn’t start to participate in the stock market back in 2010. This will give us a feature set of 200 columns containing 199 company’s values and the Date.

Now, let’s focus on our target stock the AMZN stock. If we start visualizing each of our given column values for the target stock we obtain these.

Now, let’s visualize, our stock using the candlestick notation. I am using Pandas version 0.24.2 for this. There may be an issue as in the current versions this module is depreciated.

import datetime as dt
import matplotlib.pyplot as plt
from matplotlib import style
from mpl_finance import candlestick_ohlc
import matplotlib.dates as mdates
import pandas as pd
df=pd.read_csv('stock_details/AMZN.csv',index_col=0,parse_dates=True)
df_ohlc= df['Adj Close'].resample('10D').ohlc()
df_volume=df['Volume'].resample('10D').sum()
df_ohlc.reset_index(inplace=True)
df_ohlc['Date']=df_ohlc['Date'].map(mdates.date2num)ax1=plt.subplot2grid((6,1), (0,0), rowspan=5, colspan=1)
ax2=plt.subplot2grid((6,1), (5,0), rowspan=1, colspan=1 , sharex=ax1)
ax1.xaxis_date()candlestick_ohlc(ax1,df_ohlc.values, width=2, colorup='g')
ax2.fill_between(df_volume.index.map(mdates.date2num),df_volume.values,0)plt.show()

This code will give us the candlestick notation.

Now, let’s devise some features that will help us to predict our target.

We will calculate the 50 moving average. This characteristic is used by a lot of traders for predictions.

df['Moving_av']= df['Adj Close'].rolling(window=50,min_periods=0).mean()

This snippet will help us produce the moving average. Actually, it is the mean of the i-50 to i values of the “Adj Close” values for the ith index.

Now, we will try to obtain two more features, Rate of increase in volume and rate of increase in Adjusted Close for our stock

i=1
rate_increase_in_vol=[0]
rate_increase_in_adj_close=[0]while i<len(df):
 rate_increase_in_vol.append(df.iloc[i]['Volume']-df.iloc[i-1]['Volume'])
 rate_increase_in_adj_close.append(df.iloc[i]['Adj Close']-df.iloc[i-1]['Adj Close'])
 i+=1
 df['Increase_in_vol']=rate_increase_in_vol
df['Increase_in_adj_close']=rate_increase_in_adj_close

This snippet will help us obtain these features.

Now, our feature file for our target stock is ready.

Now, we merge both these feature files to make the main feature set.

Index(['High', 'Low', 'Open', 'Close', 'Volume', 'Adj Close', 'Moving_av',
 'Increase_in_vol', 'Increase_in_adj_close', 'MMM',
 ...
 'FITB', 'FE', 'FRC', 'FISV', 'FLT', 'FLIR', 'FLS', 'FMC', 'F', 'Date'],
 dtype='object', length=207)

These are the columns of our main feature set.

It has 207 columns 200 from the other companies list and 7 from our target stock feature set anchored on the date column. So, we have 207 columns.

Regression

In this part, we will study the impact of the other stocks on our target stock. We will try to predict the High, Low, Open, and Close of our Amazon stock.

At first, let’s analyze our data.

Here, we can see we have found some correlations.

So, from our plots, it is evident that we will have some correlations in our dataset. So, next, we drop the Open, Close, High, Low values from our training dataset and use them as the target labels or values set. We also drop the Volume and Date because they won’t have a correlation.

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

We split our sets into training and testing.

from tensorflow.keras.callbacks import ModelCheckpoint
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation, Flatten
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error 
from sklearn.metrics import accuracy_scoredef model():
 mod=Sequential()
 mod.add(Dense(32, kernel_initializer='normal',input_dim = 200, activation='relu'))
 mod.add(Dense(64, kernel_initializer='normal',activation='relu'))
 mod.add(Dense(128, kernel_initializer='normal',activation='relu'))
 mod.add(Dense(256, kernel_initializer='normal',activation='relu'))
 mod.add(Dense(4, kernel_initializer='normal',activation='linear'))
 mod.compile(loss='mean_absolute_error', optimizer='adam', metrics=['accuracy','mean_absolute_error'])
 mod.summary()
 return mod

This is our model that will be used for regression.

It can be run using this snippet:

from tensorflow.keras.wrappers.scikit_learn import KerasRegressor
regressor = KerasRegressor(build_fn=model, batch_size=16,epochs=2000)
import tensorflow as tf
callback=tf.keras.callbacks.ModelCheckpoint(filepath='Regressor_model.h5',
 monitor='mean_absolute_error',
 verbose=0,
 save_best_only=True,
 save_weights_only=False,
 mode='auto')
results=regressor.fit(X_train,y_train,callbacks=[callback])

I have used Keras Regressor for this purpose. Saved best weights and used 2000 epochs. Mean Absolute Error is our Loss function. We have an input of 200 after dropping the columns. We are going to obtain four values for each input, High, Low, Open, Close.

y_pred= regressor.predict(X_test)
import numpy as np
y_pred_mod=[]
y_test_mod=[]for i in range(0,4):
 j=0
 y_pred_temp=[]
 y_test_temp=[]
 while(j<len(y_test)):
 y_pred_temp.append(y_pred[j][i])
 y_test_temp.append(y_test[j][i])
 j+=1
 y_pred_mod.append(np.array(y_pred_temp))
 y_test_mod.append(np.array(y_test_temp))

This gives us the predicted values for our test set. This snippet helps to obtain them in a parsed format.

Now, we can see our model has performed quite well considering most of the points lie on the regression line and there are very few outliers.

Here we complete our regression part. Next, we will move to the RNN part.

Recurrent Neural Network

We will be using RNN for the past analysis of our target stock. So we will be working with the Target Stock Feature file only. Now, here we will use LSTM layers that work on RNN principles but work on a gated approach.

The LSTM helps the RNN to keep the memory over a long period of time, solving the vanishing and exploding gradient problems. We are not going to talk about it in detail here. You can find the details here.

This is our dataset for LSTM.

Now, we need to predict High, Low, Open, Close. So, now, we will drop the “Date” as it has no correlation.

High Low Open Close Volume Adj Close \
0 137.279999 134.520004 137.089996 134.520004 4523000 134.520004 
1 136.610001 133.139999 136.250000 133.899994 7599900 133.899994 
2 135.479996 131.809998 133.429993 134.690002 8851900 134.690002 
3 134.729996 131.649994 134.600006 132.250000 7178800 132.250000 
4 132.320007 128.800003 132.009995 130.000000 11030200 130.000000  Moving_av Increase_in_vol Increase_in_adj_close 
0 134.520004 0.0 0.000000 
1 134.209999 3076900.0 -0.620010 
2 134.370000 1252000.0 0.790009 
3 133.840000 -1673100.0 -2.440002

Now, our input will be this Dataframe’s values. So our input will have 9 columns and correspondingly our output will have 4 columns, High, Low, Open, Close of our target stock.

We need to create the train and test set for the LSTM model.

For this purpose, we will split our data of length 2636 records into two parts. We will be using 0–2200 records as the train set and 2200–2636 as our test set. We select the target set comprising of the 4 target columns from our train set.

df_train=df_main[:2200]
df_target=df_train[['High','Low','Open','Close']]

Now, we scale our data to make our model converge easily, as we have large variations of values in our data set.

sc = MinMaxScaler(feature_range = (0, 1))
target_set=df_target.values
train_set=df_train.values
training_set_scaled = sc.fit_transform(train_set)
target_set_scaled = sc.fit_transform(target_set)

Thus, we have obtained the scaled data for our LSTM model.

LSTM model takes in series data and produces output. Our LSTM is many to many RNN model. So, we need to produce a series of data for this purpose. To do this, we have started from the 50th index and move to the length of the training set. We have appended 0–49, which is 50 values to a list. We have created such lists for all our features.

So, our Input has (n x 50 x 9) dimensional data for our training set. We have 9 features and each feature is a list of the feature values for an extended period of 50 days. n is the number of such series obtained from the given dataset. Now, our target set is the value of the target columns on the 51st day.

We obtain this using,

X_train = []
y_train = []
for i in range(50,len(train_set)):
 X_train.append(training_set_scaled[i-50:i,:])
 y_train.append(target_set_scaled[i,:])
 X_train, y_train = np.array(X_train), np.array(y_train)

Now, let’s observe the shapes of the train and target sets.

print(X_train.shape)(2150, 50, 9)print(y_train.shape)(2150, 4)

Let’s design our models.

from sklearn.metrics import accuracy_score
from tensorflow.keras.layers import BatchNormalization
import datetime as dt
from sklearn import model_selection
from sklearn.metrics import confusion_matrix
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
import numpy as np
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import LSTM
from tensorflow.keras.layers import Dropoutdef model():
 mod=Sequential()
 mod.add(LSTM(units = 64, return_sequences = True, input_shape = (X_train.shape[1], 9)))
 mod.add(Dropout(0.2))
 mod.add(BatchNormalization())
 mod.add(LSTM(units = 64, return_sequences = True))
 mod.add(Dropout(0.1))
 mod.add(BatchNormalization())
 mod.add((LSTM(units = 64)))
 mod.add(Dropout(0.1))
 mod.add(BatchNormalization())
 mod.add((Dense(units = 16, activation='tanh')))
 mod.add(BatchNormalization())
 mod.add((Dense(units = 4, activation='tanh')))
 mod.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy','mean_squared_error'])
 mod.summary()
 return mod

This is our LSTM model. I have used Keras layers here. The loss function is the Mean Squared Error. We have used Adam Optimizer.

To train our model we will be using this snippet.

import tensorflow as tf
callback=tf.keras.callbacks.ModelCheckpoint(filepath='./RNN_model.h5',
 monitor='mean_squared_error',
 verbose=0,
 save_best_only=True,
 save_weights_only=False,
 mode='auto',
 save_freq='epoch')
RNN_model.fit(X_train, y_train, epochs = 2000, batch_size = 32,callbacks=[callback])

Our model is now trained.

Let us focus on the testing part.

We have 2200 to 2636 records for our test values. So we obtain our target values by picking the four columns Open, Close, High, Low of our stock.

df_test=df_main[2200:]
df_target_test=df_test[['High','Low','Open','Close']]
target_set_test=df_target_test.values
test_set=df_test.values

To test also, we need to transform our test feature dataset and form a series of 50 feature values for this set as we did in case of the train set.

predicted_stock_price = RNN_model.predict(X_test)
predicted_stock_price = sc.inverse_transform(predicted_stock_price)
print(predicted_stock_price)

This code snippet helps to obtain the predicted results as we desired.

array([[1690.364 , 1610.5382, 1643.4277, 1643.8912],
 [1687.384 , 1607.5323, 1640.3225, 1640.7366],
 [1688.7346, 1608.965 , 1641.6777, 1642.0984],
 ...,
 [2567.6138, 2510.703 , 2522.8406, 2538.787 ],
 [2576.5056, 2519.5195, 2531.803 , 2547.9304],
 [2578.5886, 2521.65 , 2533.9177, 2550.0896]], dtype=float32)

The results are so obtained.

Now, if we plot them accordingly with the actual values of our target columns, we will obtain the following plot.

There are 4 lines if we observe, this is due to the fact that we have 4 columns as the target. We can see the model has quite obtained the curve patterns, in most of the places.

Now, if we plot them individually, we will obtain 4 plots as follows.

Thus, we conclude the study of the individual target stock based on its past values of the stock using LSTM. Now, we move to the Covering ANN or concluding portion.

The Covering Artificial Neural Network

Now, this is the concluding portion. We have obtained two different results studying two different areas that may affect the stock of a company using two exclusive models. Now, we will try to merge both the results. We will do this by merging the obtained results and training our Artificial Neural Network over the whole thing.

We had 4 target columns for the above two discussed models. For the final conclusion, we will try to predict only two columns, the High and the Low Column of our target stock.

So, for this, we predict our full dataset using both the models. The Regressor model produces 2636 predicted values.

The snippet and results are as follows:

saved_model_regressor=tf.keras.models.load_model('Regressor_model.h5')
Regressor_prediction=saved_model_regressor(X)
import numpy as np
y_pred_mod=[]for i in range(0,4):
 j=0
 y_pred_temp=[]
 while(j<len(Regressor_prediction)):
 y_pred_temp.append(Regressor_prediction[j][i])
 j+=1
 y_pred_mod.append(np.array(y_pred_temp))
Y_pred=pd.DataFrame(list(zip(y_pred_mod[0],y_pred_mod[1],y_pred_mod[2],y_pred_mod[3])),columns=['High_regress','Low_regress','Open_regress','Close_regress'])

This is the snippet. It creates a Dataframe of length 2636 with four predicted value columns.

Y_pred.head()

We save it as a CSV for further use.

Now, similarly, we do this for the LSTM network. We obtain one value for the first 50 values as the first 50 values are our first test set. So, the division is 0–49 obtains 1st value, 1–50 obtains the 2nd value, and so on. We obtain a total of 2586 values from the RNN.

Snippet:

saved_model_RNN=tf.keras.models.load_model('RNN_model.h5')
RNN_prediction=saved_model_RNN.predict(X_test)
import numpy as np
y_pred_mod=[]for i in range(0,4):
 j=0
 y_pred_temp=[]
 while(j<len(RNN_prediction)):
 y_pred_temp.append(RNN_prediction[j][i])
 j+=1
 y_pred_mod.append(np.array(y_pred_temp))
Y_pred=pd.DataFrame(list(zip(y_pred_mod[0],y_pred_mod[1],y_pred_mod[2],y_pred_mod[3])),columns=['High_RNN','Low_RNN','Open_RNN','Close_RNN'])

This snippet generates a Dataframe with the RNN prediction of the target values.

Our predictions of both models are ready. We need to drop the 0–50 index from the Regression predictions because there are no RNN predictions for those values. We are now ready to merge the results.

So, after merging the results we obtain a Dataframe of 2586 values with 8 columns.

df=pd.concat([df1,df2],axis=1)df.head()

This is the concatenated result.

The whole thing will become the feature set for our ANN model. So, what about our target set. Our target set will comprise of the original High and Low values of our Target stock of Amazon from the original dataset.

df1=pd.read_csv("dataset_target_2.csv")
target_high=[]
target_low=[]
i=50
while i<len(df1):
 target_high.append(df1.iloc[i]['High'])
 target_low.append(df1.iloc[i]['Low'])
 i+=1
df['Target_high']=target_high
df['Target_low']=target_lowdf.to_csv('feature.csv',index=False)

This snippet helps to merge all the columns in our feature set. There are 10 columns. 8 feature columns from the two models and the High and Low values of the target stock as the target values.

Our new feature set looks like this.

This is our feature histogram plots. Here also, we get a pretty good correlation from the plots. Now let’s design and train our Artificial Neural Network model.

X_Df=df_main[['High_regress','Low_regress','Open_regress','Close_regress','High_RNN','Low_RNN','Open_RNN','Close_RNN']].values
y_Df=df_main[['Target_high','Target_low']].values
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X_Df, y_Df, test_size=0.3)

Now, let’s see our model design.

from tensorflow.keras.callbacks import ModelCheckpoint
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation, Flatten
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error 
from sklearn.metrics import accuracy_scoredef model():
 mod=Sequential()
 mod.add(Dense(32, kernel_initializer='normal',input_dim = 8, activation='relu'))
 mod.add(Dense(64, kernel_initializer='normal',activation='relu'))
 mod.add(Dense(128, kernel_initializer='normal',activation='relu'))
 mod.add(Dense(2, kernel_initializer='normal',activation='linear'))
 mod.compile(loss='mean_absolute_error', optimizer='adam', metrics=['accuracy','mean_absolute_error'])
 mod.summary()
 return mod

This our ANN model. As we see, the input dimension of our model is 8. This is because we have input 8 feature columns 4 columns obtained from each model’s output. The output layer has 2 nodes for High and Low columns for the target stock.

I have used the loss function as Mean Absolute Error, Optimizer as Adam’s optimizer.

We can run the model using the following snippet:

import tensorflow as tf
model_ANN=model()
callback=tf.keras.callbacks.ModelCheckpoint(filepath='ANN_model.h5',
 monitor='mean_absolute_error',
 verbose=0,
 save_best_only=True,
 save_weights_only=False,
 mode='auto')
results=model_ANN.fit(X_train,y_train, epochs = 2000, batch_size = 32,callbacks=[callback])

After we train the model, we will be using the predicted model to predict our test set and check for the performance.

y_pred=model_ANN.predict(X_test)
import numpy as np
y_pred_mod=[]
y_test_mod=[]for i in range(0,2):
 j=0
 y_pred_temp=[]
 y_test_temp=[]
 while(j<len(y_test)):
 y_pred_temp.append(y_pred[j][i])
 y_test_temp.append(y_test[j][i])
 j+=1
 y_pred_mod.append(np.array(y_pred_temp))
 y_test_mod.append(np.array(y_test_temp))
df_res=pd.DataFrame(list(zip(y_pred_mod[0],y_pred_mod[1],y_test_mod[0],y_test_mod[1])),columns=['Pred_high','Pred_low','Actual_high','Actual_low'])

Our predictor predicts two values for each featured record. So, this code snippet helps to obtain the predicted values of the two columns and the test values and represent them as a Dataframe.

Now, let’s plot and check.

import matplotlib.pyplot as pltax1=plt.subplot2grid((4,1), (0,0), rowspan=5, colspan=1)ax1.plot(df_res_2.index, df_res_2['Pred_high'], label="Pred_high")
ax1.plot(df_res_2.index, df_res_2['Actual_high'], label="Actual_high")plt.legend(loc="upper left")
plt.xticks(rotation=90)plt.show()

This snippet helps to see the predicted and actual variations.

If we want to check our model performances. We can do this by using this snippet:

fig, ax = plt.subplots()
ax.scatter(y_test_mod[0], y_pred_mod[0])
ax.plot([y_test_mod[0].min(),y_test_mod[0].max()], [y_test_mod[0].min(), y_test_mod[0].max()], 'k--', lw=4)
ax.set_xlabel('Measured')
ax.set_ylabel('Predicted')
plt.show()

This will obtain:

These linear plots show that there is a linear relationship between our models predicted values and the actual values.

So, our composite model did quite well on the predictions of High and Low column values of our target Amazon Stock. So, here we conclude our entire purpose.

Conclusion

In this article, we have seen how we can build an algorithmic trading model based on composite Neural Nets. I hope this article will help.

The whole Github code is: here.

Please find the further task on this subject: here.