Professional Documents
Culture Documents
How To Use and Remove Trend Information From Time Series Data in Python
How To Use and Remove Trend Information From Time Series Data in Python
How To Use and Remove Trend Information From Time Series Data in Python
com/time-series-trends-in-python/
Navigation
Search...
A trend is a continued increase or decrease in the series over time. There can be benefit in identifying,
modeling, and even removing trend information from your time series dataset.
In this tutorial, you will discover how to model and remove trend information from time series data in
Python.
The importance and types of trends that may exist in time series and how to identify them.
How to use a simple differencing method to remove a trend.
How to model a linear trend and remove it from a sales time series dataset.
Discover how to prepare and visualize time series data and develop autoregressive forecasting models
in my new book, with 28 step-by-step tutorials, and full python code.
1 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
How to Use and Remove Trend Information from Time Series Data in Python
Photo by john78727, some rights reserved.
In general, a systematic change in a time series that does not appear to be periodic is known
as a trend.
Identifying and understanding trend information can aid in improving model performance; below are a
few reasons:
Faster Modeling: Perhaps the knowledge of a trend or lack of a trend can suggest methods and
make model selection and evaluation more efficient.
Simpler Problem: Perhaps we can correct or remove the trend to simplify modeling and improve
model performance.
More Data: Perhaps we can use trend information, directly or as a summary, to provide additional
information to the model and improve model performance.
Types of Trends
There are all kinds of trends.
2 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
In general, deterministic trends are easier to identify and remove, but the methods discussed in this
tutorial can still be useful for stochastic trends.
Global Trends: These are trends that apply to the whole time series.
Local Trends: These are trends that apply to parts or subsequences of a time series.
Identifying a Trend
You can plot time series data to see if a trend is obvious or not.
The difficulty is that in practice, identifying a trend in a time series can be a subjective process. As such,
extracting or removing it from the time series can be just as subjective.
Create line plots of your data and inspect the plots for obvious trends.
Add linear and nonlinear trend lines to your plots and see if a trend is obvious.
Removing a Trend
A time series with a trend is called non-stationary.
An identified trend can be modeled. Once modeled, it can be removed from the time series dataset. This
is called detrending the time series.
If a dataset does not have a trend or we successfully remove the trend, the dataset is said to be trend
stationary.
Specifically, a trend can be removed from your time series data (and data
preparation and cleaning exercise. This is common when using statistical
forecasting, but does not always improve results when using machine lear
3 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
Alternately, a trend can be added, either directly or as a summary, as a new input variable to the
supervised learning problem to predict the output variable.
One or both approaches may be relevant for your time series forecasting problem and may be worth
investigating.
Click to sign-up and also get a free PDF Ebook version of the course.
The units are a sales count and there are 36 observations. The original dataset is credited to
Makridakis, Wheelwright, and Hyndman (1998).
Below is a sample of the first 5 rows of data, including the header row.
Below is a plot of the entire dataset, where you can learn more and downl
4 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
The dataset can be loaded with a custom date parsing routine as follows:
5 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
Detrend by Differencing
Perhaps the simplest method to detrend a time series is by differencing.
Specifically, a new series is constructed where the value at the current time step is calculated as the
difference between the original observation and the observation at the previous time step.
This has the effect of removing a trend from a time series dataset.
6 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
Running the example creates the new detrended dataset and then plots the time series.
Because no difference value can be created for the first observation (there is nothing for it to be
subtracted from), the new dataset contains one less record. We can see that indeed the trend does
appear to have been removed.
This approach works well for data with a linear trend. If the trend is quadratic (the change in the trend
also increases or decreases), then a difference of the already-differenced
second level of differencing. This process can be further repeated if neede
7 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
Because differencing only requires the observation at the previous time step, it can easily be applied to
unseen out-of-sample data to either preprocess or provide an additional input for supervised learning.
Linear trends can be summarized by a linear model, and nonlinear trends may be best summarized
using a polynomial or other curve-fitting method.
Because of the subjective and domain-specific nature of identifying trends, this approach can help to
identify whether a trend is present. Even fitting a linear model to a trend that is clearly super-linear or
exponential can be helpful.
In addition to being used as a trend identification tool, these fit models can also be used to detrend a
time series.
For example, a linear model can be fit on the time index to predict the observation. This dataset would
look as follows:
The predictions from this model will form a straight line that can be taken as the trend line for the
dataset. These predictions can also be subtracted from the original time series to provide a detrended
version of the dataset.
The residuals from the fit of the model are a detrended form of the dataset. Polynomial curve fitting and
other nonlinear models can also be used.
8 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
Running the example first fits the linear model to the integer-indexed observations and plots the trend
line (green) over the original dataset (blue).
Next, the trend is subtracted from the original dataset and the resulting de
9 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
Again, we can see that this approach has effectively detrended the dataset. There may be a parabola in
the residuals, suggesting that perhaps a polynomial fit may have done a better job.
Because the trend model takes only the integer index of the observation as input, it can be used on new
data to either detrend or provide a new input variable for the model.
Further Reading
Below are some additional resources on trend estimation and detrending i
Summary
In this tutorial, you discovered trends in time series data and how to remov
10 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
About the importance of trend information in time series and how you may be able to use it in
machine learning.
How to use differencing to remove a trend from time series data.
How to model a linear trend and remove it from time series data.
11 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
REPLY
Carmen Mardiros January 27, 2017 at 11:40 pm #
Really useful Jason. One question. Would the de-trended values be used as additional features
or instead of the original features. Also, is it common to de-trend the target variable as well? If yes, do the
de-trended values become the new target variable? If yes, then what is the process for obtaining full
predictions from ML models (for sklearn at least, I’m guessing a custom transformer needs to store the
removed trend and add it back in using inverse_transform).
Thank you!
REPLY
Jason Brownlee January 28, 2017 at 7:49 am #
Hi Carmen,
The discussion in this post is for de-trending a univariate time series. If you wish to detrend a
univariate time series, you would treat it as a suite of univariate time series and de-trend one at a
time.
The whole series may be de-trended and is used as input for a learning algorithm like ARIMA. See
this post:
http://machinelearningmastery.com/arima-for-time-series-forecasting-with-python/
Alternatively, the detrended series may be used as an additional input for a machine learning
algorithm.
REPLY
Giovanni May 24, 2017 at 7:28 pm #
12 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
I guess you can just start with the initial value of the original timeseries, and then recreate
the original timeseries by adding each delta obtained with the differentiation.
REPLY
Marianico July 21, 2017 at 10:52 pm #
REPLY
Jason Brownlee July 22, 2017 at 8:35 am #
It can. Try it and see. ARIMA has it built in with the q var.
REPLY
Ahmed August 11, 2017 at 12:55 pm #
Great tutorial! really thanks for such useful information. My question is, Is it possible to have
linear and non-linear trend added to the same data set? If yes, Shall I remove the linear part first, then
the non-linear part (which is a straight line fitted to the data and then subtract it) and then remove the
non-linear part (fit a low-order polynomial to the signal and then subtract it)? Or just removing the non-
linear part would be sufficient ??
REPLY
Jason Brownlee August 12, 2017 at 6:46 am #
Interesting question. Generally, I would recommend trying both and see which model results
in better forecasts.
REPLY
Weber August 12, 2017 at 9:50 pm #
Suppose the time series shows downward trend most of the times
previous time stamp may result in a negative value. Perhaps simple moving averages can be used in this
case..?
13 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
REPLY
Weber August 13, 2017 at 7:00 pm #
Thanks for your reply. This technique would fail for Poisson Auto regression model. Do
you have any suggestions to detrending in event count models?
REPLY
Jason Brownlee August 14, 2017 at 6:24 am #
REPLY
Mamta August 21, 2017 at 4:35 am #
Dear Jason,
I followed your approach to detrend the time series data before fitting a neural network model on the
training set.
When I have to use my trained model to predict on unseen data, how do I deal with the trend ?
Do I have to predict the trend along with the the usual observations, as my model is trained on detrended
data? Should this predicted trend be added to the predicted output of the trained model ?
Thanks
Mamta
REPLY
Jason Brownlee August 21, 2017 at 6:10 am #
You can remove a trend by differencing. You can then invert the differencing after you have
made a prediction.
See here:
http://machinelearningmastery.com/remove-trends-seasonality-difference
and here:
http://machinelearningmastery.com/difference-time-series-dataset-pyth
Dear Jeson
Sir, I have a time series data obtained from integration of another data. The
exponentially decaying in amplitude with time. Would you please suggest m
14 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
(identify) linear or non-linear trend (Equation of poly nomial) present in the derived time series data (i.e.
obtained from integration of another data)?
Thanks
REPLY
Jason Brownlee February 9, 2018 at 9:09 am #
I don’t have an example. Perhaps you can use a power transform on the obs to stabilize the
changing variance, then try removing the cycle?
Or perhaps you can use an scipy function call to model the exponential trend and remove it
manually?
REPLY
Faiza March 16, 2018 at 12:58 am #
Dear Jason
Sir plzz give me any example i have a data of time series in 4 quarters.how i solve this “assuming the
multiplicative model and using the movig average method,obtain the detrended time seies”??????
REPLY
Jason Brownlee March 16, 2018 at 6:20 am #
REPLY
Rahul May 21, 2018 at 7:28 pm #
Hi Jason,
Really useful tutorial. Just wanted to inform you that Detrending Notes, GEOS 585A, Applied Time Series
Analysis [PDF] in Further Reading section is not working. If possible, could
Regards,
Rahul Sathyajit
15 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
REPLY
hamid August 16, 2018 at 8:47 pm #
Jason,
Thank you for your incredible post.
So, in one of your LSTM posts, you used multivariate weather input features (x1,x2,x3,x4) to predict
another weather parameter (Y). Should I also use detrending techniques for the input features which also
have trends ( for example, suppose x1 and x4 have trend) ? If yes, I don’t feel confident about my results,
as I’ve changed my input data to a new one. If no, maybe my model cannot predict well. What am I
missing?
REPLY
Jason Brownlee August 17, 2018 at 6:27 am #
REPLY
HAMID August 21, 2018 at 6:31 am #
Hello Jason
What is the best way to select the optimal subset(rows) to be trained by LSTM?
regards
Salah
REPLY
Jason Brownlee August 21, 2018 at 6:38 am #
REPLY
Jakub Kajan October 18, 2018 at 3:59 am #
Hello Jason.
Thank you very much for great tutorial. I tried to apply diff to data via pandas diff, but with diff data model
was not able to converge, I tried different topologies, neurons in layers etc.
But when I don’t modify data, fit is really good. Have you ever meet such issue, or could you know some
good reading related to this topic?
Regards Jakub.
16 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
Yes, MLPs and CNNs seem to perform well even better when the data is non-stationary, at
least in a few tests I performed.
REPLY
Theekshana November 7, 2018 at 7:38 pm #
Hi Janson,
Is there any well-known method for identifying the time series trend. As an example given a time series,
the method should be able to give a value that says the series increasing, decreasing or stabilized (the
value should represent the trend perfectly. some kind of a scale maybe 0-100, 0 for decreasing 100 for
increasing at a higher rate, 50 for stable etc).
From your example, I can see that the fitted line can be used to come up with some value about the trend
( maybe using the gradient ). Is there any sophisticated way to do this?
Thank you.
Great tutorial. Big fan of this site…
REPLY
Jason Brownlee November 8, 2018 at 6:06 am #
REPLY
S February 2, 2019 at 9:52 pm #
Question – once the trend has been removed (as you have in your last image), one could use that
detrended data to make some predictions with ARIMA. How would one interpret the predicted value? Let
us assume it goes up by 2 ‘units’ in the next period = could expect an increase of 2 x the scale that was
used in the original data set?
The trend can be added back to any predictions made and the interpretation would then be
specific to the domain.
Hi Jason,
17 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
Does any of your other tutorial talk about it? Searched for it but couldn’t find it.
REPLY
Jason Brownlee February 15, 2019 at 8:05 am #
Perhaps add the new variables as exog variables for a time step.
Otherwise, perhaps use an ensemble model that combined predictions from a model on the time
series and a model no the static data?
REPLY
Leen March 16, 2019 at 10:59 pm #
Hello Jason,
I wanted to ask if we are to detrend and deseasonalize, van’t we just use the seasonal_decompose()
function, and extract the trend and seasonality components from it and leave the residual ?? Is the
residual considered detrended and deseasonalized ?
Example
result = seasonal_decompose(self.series, model=’additive’, freq=frequency)
residual = result.resid # this is what I am reffering to.
Thanks in advance
18 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
from seasonal_decompose()
REPLY
Jason Brownlee March 18, 2019 at 6:01 am #
REPLY
deepika March 20, 2019 at 10:15 pm #
Thanks a lot!
REPLY
Jason Brownlee March 21, 2019 at 8:12 am #
Try modeling the data directly, then with differencing on the series with trends to see if there
is an impact.
For categorical vars, explore integer encode, one hot encode and perhaps embeddings.
REPLY
Teyang July 8, 2019 at 6:22 pm #
Hi Jason,
Great post!
When you mentioned subtracting the trend from the original data, is it the s
of the linear/quadratic/polynomial model?
Thanks!
19 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
REPLY
Jason Brownlee July 9, 2019 at 8:06 am #
Similar.
REPLY
Laura August 23, 2019 at 12:38 am #
Hi Jason,
If the goal is to forecast the future N timesteps of a timeseries showing a stochastic trend, then we should
fit a model to approximate the trend, remove this approximated trend from the original data and use
models such as ARIMA to predict there residuals. However, the final forecast will be Arima forecast +
trend forecast.
The timeseries data I work with is not well approximated by a linear regression, it consists of random
patterns with the trend going up and down at different amplitudes (like a random walk). I didn’t find yet a
model which can approximated it well enough. What model do you recommend using for this kind of
stochastic trend approximation?
Thanks!
REPLY
Jason Brownlee August 23, 2019 at 6:31 am #
Typically an ARIMA can remove the trend for you via the d parameter.
I recommend testing a suite of linear and non-linear models in order to discover what works best for
your specific dataset:
https://machinelearningmastery.com/how-to-develop-a-skilful-time-series-forecasting-model/
REPLY
Ha My Nguyen February 29, 2020 at 5:42 pm #
Hi, I run the code on Python and I dont know why this function kept showing error
datetime.strptime. Is that possible to write in another way?
20 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
Leave a Reply
Name (required)
Website
SUBMIT COMMENT
Welcome!
My name is Jason Brownlee PhD, and I help developers get results with machine learning.
Read more
21 of 22 4/5/2020, 2:19 PM
How to Use and Remove Trend Information from Time Series Data in P... https://machinelearningmastery.com/time-series-trends-in-python/
22 of 22 4/5/2020, 2:19 PM