利用matlab对时间序列数据(牛仔裤销售数据集)进行LSTM预测-matlab实现（包含完整源码+数据集+项目说明）.zip

# Deep learning project - Time Series Data Prediction (Matlab, LSTM) Writer : Harim Kang # Organized Blog 'Read Me' posting in Korean is located at the address below. [https://davinci-ai.tistory.com/11](https://davinci-ai.tistory.com/11) # Project subject The first topic was to analyze online product price data to predict current product prices. However, due to the insufficient time and limited computing power compared to the vast amount of data, we modified the project to analyze the online price data of jeans and predict the jeans price in the near future. The reason why I chose jeans out of many items is that I have more data than other items and I can wear them all season. (Cell phones and TV items were selected as jeans because of lack of historical data.) For more items, I'll try to find a way later. ## Usage data [https://www.data.go.kr/dataset/15004449/fileData.do](https://www.data.go.kr/dataset/15004449/fileData.do) Online collection price information data consists of 8 items including price information collected online, collection date, item name and sales price.  The data collection period is available from January 2014 to October 2019. The data I used in the analysis used data from January 2015 to October 2019. ## Data analysis process 1. Data Purification 2. Explore the data 3. Purify additional data for analysis 4. Model Selection for Prediction 5. Data prediction 6. Predictive Assessment (RMSE) 7. Semantic Analysis ## Data refining Due to the lack of computing power (my laptop) compared to the huge amount of data (about 100 million data), I decided to use the average daily selling price per day.   To explain the code, we retrieved the data by date and extracted only the collection date, item name, and selling price from the eight items. The code that finds and averages only the data whose item name is jeans and writes it to the new data. It's been a while since I'm working with a lot more data than I thought. It took more than 30 minutes per year of data, and a good cpu computer seems to be faster. Using the code above, we obtained a refined data set by calculating the average of the daily jeans sale price for each day.  ## Explore data  This is a graph of the average price of jeans made for own use. The data summary is shown below.  - Item: Jeans - Period: January 01, 2015 ~ October 31, 2019 (total 1765 days) - Minimum value: 32732 won - Average value: 51998 won - Maximum value: 166220 won  The graph above is a graph overlaid by year. It doesn't look very good. ## Purify additional data for analysis Take the sales price average data for the newly created jeans and fill in the blanks for better learning results. % Jeans mean data refinement jean_data = readtable('mean\tt.xlsx'); % Fill the NaN value with the Nearest value. jean_data.sales_price = fillmissing(jean_data.sales_price, 'nearest'); lenofdata = length(jean_data.sales_price); for i=1 : length(jean_data.collect_day) jean_data.collect_day(i) = strip(jean_data.collect_day(i),"'"); end Y = jean_data.sales_price; data = Y'; matlab handles white space in data with a function called fillmissing. At this time, we put the nearest value 'nearest' as a parameter because we thought the data had a trend. The code below uses the strip function to delete special characters in string characters. % 2015.01.01 ~ 2019.05.06 (90%) : Training Data Set % 2019.05.07 ~ 2019.10.31 (10%) : Test Data Set numTimeStepsTrain = floor(0.9*numel(data)); dataTrain = data(1:numTimeStepsTrain+1); dataTest = data(numTimeStepsTrain+1:end); In order to train the model well and evaluate it, we distinguished between training and test datasets. Since the amount of data is not very large, I divided the ratio by 9: 1 to get better results. % Normalize sales_price to a value between 0 and 1 (Training Data Set) mu = mean(dataTrain); sig = std(dataTrain); dataTrainStandardized = (dataTrain - mu) / sig; XTrain = dataTrainStandardized(1:end-1); YTrain = dataTrainStandardized(2:end); Also, the selling price is a very large number. (10,000 units) So I did a normalization to learn more. We are going to train after changing the number to a relative value between 0 and 1. ## Model Selection for Prediction I chose a LSTM (Long Short-Term Memory models) model and applied it to the data. My data is organized by date in daily order. This is called 'Time Series' data, which is one of the Sequence data types. Sequence type data uses a deep learning model called Recurrent Neural Networks (RNN).  RNN However, the data that you want to use has to play an important role. In the case of the existing RNNs, the farther you are from the earlier data, the more oblivious the forgetting becomes.  The LSTM adds input gates and output gates to memory cells in the hidden layer to clear out unnecessary memory and determine what to remember. That's why LSTM is more suitable for Time Series than RNN.  Detailed algorithm descriptions will be further summarized as you study Deep Learning. In Matlab, set the LSTM option with the following code: This is the code that increased MaxEpochs to 500 in the existing Matlab LSTM tutorial. %LSTM Net Architecture Def numFeatures = 1; numResponses = 1; numHiddenUnits = 200; layers = [ ... sequenceInputLayer(numFeatures) lstmLayer(numHiddenUnits) fullyConnectedLayer(numResponses) regressionLayer]; options = trainingOptions('adam', ... 'MaxEpochs',500, ... 'GradientThreshold',1, ... 'InitialLearnRate',0.005, ... 'LearnRateSchedule','piecewise', ... 'LearnRateDropPeriod',125, ... 'LearnRateDropFactor',0.2, ... 'Verbose',0, ... 'Plots','training-progress'); You can train the network using the above options as shown below. The Matlab code is like the code below. % Train LSTM Net net = trainNetwork(XTrain,YTrain,layers,options); Running the above code will train the model as shown below. Iteratively evaluates itself within the training dataset.  LSTM Net Architecture Model Training Progress (Epoch: 250) ## Data prediction Training data was normalized before training. Normalize test data. Then start predicting against the test data set. % Normalize sales_price to a value between 0 and 1 (Testing Data Set) dataTestStandardized = (dataTest - mu) / sig; XTest = dataTestStandardized(1:end-1); net = predictAndUpdateState(net,XTrain); [net,YPred] = predictAndUpdateState(net,YTrain(end)); % Predict as long as the test period (2019.05.07 ~ 2019.10.31) numTimeStepsTest = numel(XTest); for i = 2:numTimeStepsTest [net,YPred(:,i)] = predictAndUpdateState(net,YPred(:,i-1),'ExecutionEnvironment','cpu'); end The code that executes the for statement as much as the test data, puts the predicted value in YPred, and updates and initializes the net. ## Predictive evaluation (RMSE) YPred contains the predicted value and YTest contains the actual correct answer. To assess the performance of the model, we use the MSE or RMSE to check the error of the data. I will use RMSE here. (I'll add a description later.) % RMSE calculation of test data set YTest = dataTest(2:end); YTest = (YTest - mu) / sig;