Linear analysis > Jacobian matrix

The Jacobian matrix is a matrix of partial derivatives. Each column of this matrix pertains to a single adjustable parameter. The column contains the partial derivatives of all model-generated observations with respect to this parameter. Where a parameter is log-transformed, the derivative is with respect to the log (to base 10) of the parameter.

jacobian

A Jacobian matrix.

The Jacobian matrix file is binary and compressed. This saves storage space. Saving storage is necessary because a Jacobian matrix can become very large if the number of parameters and/or observations is large. Utility programs that are discussed on this page allow you to extract some or all of the elements of a Jacobian matrix from the file in which it is stored so that you can view them.

If PEST does not work, clues to its malperformance can often be found in the Jacobian matrix. So inspecting it may prove really useful. If some elements of the Jacobian matrix are excessively large, this sometimes indicates that the model solver failed to converge when the model was provided with a parameter set in which only one parameter was incrementally varied. This is not good. Or perhaps it indicates that a particular parameter is super-sensitive. In the latter case, consider log-transforming the offending parameter, or set its SCALE to a low value while increasing the initial value of the parameter in the "parameter data" section of the PEST control file.

Also, don't forget about the composite sensitivity file written by PEST. This is named case.sen where case is the filename base of the PEST control file. The composite sensitivity of a parameter sums the weighted sensitivities of all model outputs and prior information equations for that parameter. The contents of this file can also be useful in fault-diagnosis. (However the inclusion of prior information in composite sensitivities may sometimes conceal aberrant parameter insensitivity.)

Programs in the following table are supplied with PEST.

Program	Task
JACWRIT	Re-writes the entirety of a Jacobian matrix file in text format. The resulting file may be very large.
JCO2MAT	Similar to JACWRIT. However the text file which holds the Jacobian matrix respects PEST matrix file protocols. This allows manipulation and processing by PEST matrix utilities.
MAT2JCO	Performs the reverse function to JCO2MAT. Sometimes JCO2MAT followed by MAT2JCO facilitates transportation of a PEST-written Jacobian matrix file from one computing platform to another.
JROW2MAT	Extracts a row of a Jacobian matrix, recording its contents in PEST matrix file format. (This is a text format that is easily readable by humans.)
JROW2VEC	Similar to JROW2MAT, but turns a row matrix into a vector (i.e. a single column matrix). This utility is often a useful precursor to linear predictive uncertainty analysis. This is the case where the extracted row of the matrix pertains to a prediction rather than an observation.
JCOL2VEC	Extracts a column of the Jacobian matrix and records it as a vector in PEST matrix file format.

We give JCO2JCO a special mention because of how useful it is.

Suppose that you have a PEST control file and a corresponding JCO file. Now create a new PEST control file by modifying the original PEST control file in any of the following ways:

•remove observations;

•remove prior information;

•alter observation weights;

•remove parameters;

•fix or tie parameters;

•alter the transformation status of parameters to or from log-transformed.

JCO2JCO will build a new JCO file for the new PEST control file. You do not need to run PEST with NOPTMAX set to -1 or -2 to obtain a new Jacobian matrix.

jco2jco

Programs that are listed in the following table (all supplied with PEST) can sometimes be useful. They can save the numerical expense of building a large Jacobian matrix from scratch if only part of it needs to be modified.

Program	Task
JCOPCAT	Adds sensitivities of all model outputs with respect to new parameters to an existing Jacobian matrix. Hence a Jacobian matrix can be updated by running the model only a few times.
JCOMIX	Combine the elements of two Jacobian matrices according to a user-prescribed order of priority where elements overlap.
JCOORDER	Removes and reorders rows and columns of a Jacobian matrix.
JCOADDZ	Adds zero-valued rows and columns to a Jacobian matrix.
JCODIFF	Subtracts the contents of one Jacobian matrix file from that of another. This can be useful for quick calculation of sensitivities with respect to predictive differences.
JCOSUM	Constructs a new Jacobian matrix through weighted summation of existing Jacobian matrices.
JCOCOMB	Calculates sensitivities for observations that are linear combinations of existing observations.
JCOSUB	Substitutes part of an existing Jacobian matrix by elements of another Jacobian matrix.
JCOZERO	Sets individual, user-specified elements of a Jacobian matrix to zero.
JCOBLANK	Zeroes elements of a Jacobian matrix. Also prepares a "simultaneous increment file". PEST_HP can use this file to dramatically increase the efficiency of finite-difference derivatives calculation by incrementing more than one parameter at the same time.
JCOWT	Calculates a weighted Jacobian matrix from the contents of a Jacobian matrix file together with weights contained in a PEST control file. The Jacobian matrix which it calculates can be used for computing the singular value spectrum of an inverse problem.
WTSENOUT	Similar to JCOWT but also calculates weighted observations.
JCOCHEK	Checks that a Jacobian matrix file is compatible with a PEST control file.

modify_jco

When searching for reasons why PEST is not working as well as it should, it is sometimes useful to compute the singular value spectrum of the inverse problem that PEST is trying to solve.

Let J be a weighted Jacobian matrix. This can be produced from a PEST-calculated Jacobian matrix using the WTSENOUT or JCOWT utilities that are supplied with PEST. Through singular value decomposition, J can be decomposed as:

J=USVT

To obtain singular values, ordered from largest to smallest, do the following:

1.Use JCOWT to construct a weighted Jacobian matrix from a PEST-calculated Jacobian matrix.

2.Re-write the weighted Jacobian matrix in PEST matrix file format using JCO2MAT.

3.Subject the weighted Jacobian matrix to singular value decomposition using the MATSVD utility supplied with PEST.

4.Inspect the contents of the S matrix that MATSVD writes.

If the first few singular values are very much larger than the remaining singular values, this suggests a problem. Perhaps some parameters, or some combinations of parameters, are super-sensitive. Perhaps the model is experiencing numerical difficulties. Alternatively, perhaps the super-sensitive parameters need to be multiplied by a large factor and then their SCALE set to the reciprocal of this factor in the PEST control file. (This only applies if these parameters are not log-transformed.) This reduces their sensitivities so that they are commensurate with the sensitivities of other parameters.

The singular value spectrum can be useful for other purposes. If there is no noise associated with the calibration dataset, the number of singular values that are larger in magnitude than about 1E-7 of the maximum singular value sets the upper limit on the dimensionality of the calibration solution space. This marks the maximum possible number of units of extractable information from the calibration dataset.

singular_value_spectrum