What is Nonlinear Regression:Linear Least Square Error Regression of Multi Variable Function (Part II)
A simple form of multi variable function is a 2-variable function expressed as
The Sum of Square Error function for a set of measurements where is the measurement of and is the number of measurements, is
The expanded form of the right hand side (RHS) of Eq.(2) is
To further simplify the example, assume that the constant term in the linear function of Eq.(1) is zero (i.e., ). Then Eq.(3) is rewritten as
In the previous post, the graph was like a vertical slice of a bowl. Figure 1 shows inside of the seed holder of an avocado as bowl shape.
Figure 2 shows how a vertical slice of bowl would look like in the length wise quarter slice of avocado.
graph of is displayed as an example of a polynomial of two input variables. To compare this as a function with the Sum of Square Error function of a function with two input random variables, replace by and by . The minimum is a point at the bottom of the bowl of Figure 3. The base of the bowl is not a perfect circle in Figure 3. The heat map of the surface in the top part of Figure 3 shows the lowest value of the surface is possibly a negative value. In the lower part of Figure 3, the red color contour line at the center is for -0.1. A contour line is a line joining all the points at same height at z-axis assuming that the bottom of the bowl is touching the horizontal plane of x-axis and y-axis. Any smaller value is so small that drawing an ellipse is not possible. Also, it is not possible to draw a point size ellipse, which would represent the minimum point of the surface of Figure 3. The smallest ellipse representing the value of -0.1 in Figure 3 indicates that the lowest point can be found in the area encircled by the contour line for -0.1. From the lower part of Figure 3, the center of the contour line at level -0.1 seems to be very close to the origin of the horizontal surface formed by x-axis and y-axis. The values of the other contour lines are listed in the right side above the lower part of Figure 3.
In the previous post, the conditional expression at the minimum for one random variable was
An extension of the above condition to take into account 2 random variables will be
To test this on the function of Figure 3, write as
When the condition of Eq.(5) is applied to the example graph of Figure 3, the following equations are obtained
As a result, the following system of linear equations are found
The solution of the system of linear equations of Eq.(8) is and or the point on the horizontal plane formed by x-axis and y-axis in Figure 3. So, the precise point of the minimum is at the point in the x-y plane of Figure 3 and is encircled by the smallest red contour at level -0.1 near the origin of the contour plots in the lower part. The precise value of the minimum is found by substituting by 0.8 and by 0.8 in the expression of Eq.(6)
The above derived value of -2.4 as minimum explains why the contour line is so small for the value -0.1.
A generalized multi variable linear function is formally stated as
The coefficients need to be estimated using Linear Least Square Error Regression from a set of measurements of the random variables and the corresponding measurements of . The theory explained in the previous post is extended here to deal with the multi variable function of Eq.(9). As was done in the previous post, let the measurement of be denoted by .
The sum of square of error function is expressed as
An example of is explained in the beginning and in Figure 3 to have an understanding of the graph of of Eq.(10), as this graph will need more than 3-dimension and is not possible to display.
The extension of the conditional expression of Eq.(5) is
Substituting the RHS of Eq.(10) into Eq.(11)and expressing as we obtain
Dividing both sides of the above expression by -2
The above expression can be expressed as a matrix equation as shown below.
The solution of the above matrix equation will give the desired estimates of . The solution involves finding the inverse of of the matrix
Once the inverse is found the solution is found by multiplying both sides of Eq.(12) by the inverse matrix of Eq.(13) and obtain the solution
In the next post, we will find out how to apply the technique of linear least square error regression for linear functions of multiple variable to nonlinear functions of single variable.