The Fréchet derivative is the standard way in the setting of functional analysis to take derivatives with respect to vectors. The matrix derivative is a convenient notation for keeping track of partial derivatives for doing calculations. See the layout conventions section for a more detailed table. In the following three sections we will define each one of these derivatives and relate them to other branches of mathematics. However, these derivatives are most naturally organized in a tensor of rank higher than 2, so that they do not fit neatly into a matrix. Notice that we could also talk about the derivative of a vector with respect to a matrix, or any of the other unfilled cells in our table. Moreover, we have used bold letters to indicate vectors and bold capital letters for matrices. Here, we have used the term 'matrix' in its most general sense, recognizing that vectors and scalars are simply matrices with one column and one row respectively. For a scalar function of three independent variables, f ( x 1, x 2, x 3 ) f(x_ Matrix notation serves as a convenient way to collect the many derivatives in an organized way.Īs a first example, consider the gradient from vector calculus. Each different situation will lead to a different set of rules, or a separate calculus, using the broader sense of the term. In general, the independent variable can be a scalar, a vector, or a matrix while the dependent variable can be any of these as well. Matrix calculus refers to a number of different notations that use matrices and vectors to collect the derivative of each component of the dependent variable with respect to each component of the independent variable. Definitions of these two conventions and comparisons between them are collected in the layout conventions section. Serious mistakes can result when combining results from different authors without carefully verifying that compatible notations have been used. Authors of both groups often write as though their specific conventions were standard. However, even within a given field different authors can be found using competing conventions. econometrics, statistics, estimation theory and machine learning). A single convention can be somewhat standard throughout a single field that commonly uses matrix calculus (e.g.

Both of these conventions are possible even when the common assumption is made that vectors should be treated as column vectors when combined with matrices (rather than row vectors). The two groups can be distinguished by whether they write the derivative of a scalar with respect to a vector as a column vector or a row vector. Two competing notational conventions split the field of matrix calculus into two separate groups. The notation used here is commonly used in statistics and engineering, while the tensor index notation is preferred in physics. This greatly simplifies operations such as finding the maximum or minimum of a multivariate function and solving systems of differential equations.

It collects the various partial derivatives of a single function with respect to many variables, and/or of a multivariate function with respect to a single variable, into vectors and matrices that can be treated as single entities. In mathematics, matrix calculus is a specialized notation for doing multivariable calculus, especially over spaces of matrices.

Differential calculus product rule