Linear Models and Linear Relationship

Update 3/20/2015: sharing a blog post from Freakonometrics On Some Alternatives to Regression Models.
In many papers, we can find the statements similar to the one below:

Because linear models can hardly capture the nonlinear relationship between load and temperature, we use Artificial Neural Networks (or other black-box models) in this paper. 

The major conceptual error of the above statement is due to a common misunderstanding that linear models cannot capture nonlinear relationship.

I'm showing a nonlinear curve in the figure below, which is from a 3rd order polynomial function.

Is this a linear model? Yes! Polynomial regression models in general are linear models.

Why?

The "linear" in linear models refers to the equations we use to solve the parameters. By definition, a regression model is linear if it can be written as Y = XB + E, where Y is a vector of values of the response variable; B is a vector of parameters to be solved; X is a matrix of values of explanatory variables; E is a vector of independent normally distributed errors.

Maybe the above description is too abstract. Let's check out an example of parameter estimation for a 3rd order polynomial regression model:

y = b0 + b1x1 + b2x2 + b3x3 + e, 

where x2 is the square of x1, and x3 is the cube of x1. There are 4 parameters to be estimated. Now let's say we have 6 observations as shown in the table below.

Then we can come up with 6 linear equations:

3 = b0 + b1 + b2 + b3;

1 = b0 + 2b1 + 4b2 + 8b3;

4 = b0 + 3b1 + 9b2 + 27b3;

9 = b0 + 4b1 + 16b2 + 64b3;

5 = b0 + b1 + b2 + b3;

2 = b0 + 3b1 + 9b2 + 27b3;

These equations can be written in the following form:

Therefore, a 3rd order polynomial regression mode is a linear model. By solving the above equations, we can obtain the values of b0, b1, b2 and b3:

To further clarify the concept of linear vs. nonlinear, here are a few examples of nonlinear regression models:

y = b0 + b1x1/(b2x2 + b3) + e, ...... (1)

y = b0(exp(b1x))U, ...... (2)

Not to make it too complicated, some nonlinear regression problems can be solved in a linear domain. For instance, eq(2) can be transformed to a linear model by taking the logarithm on both side:

Ln(y) = Ln(b0) + b1x + u, .... (3)

Having said all above, there is no problem using linear models for load forecasting. Some good and successful examples of linear regression models can be found from my PhD dissertation. The models and methodologies I developed over there are already commercialized and deployed to utilities worldwide.

Back to Load Forecasting Terminology.