9  Quadratic From

A quadratic form is a type of polynomial expression that is a sum of terms where each term is either a variable squared or the product of two variables, often associated with a symmetric matrix. Quadratic forms are useful in various areas of mathematics, physics, statistics, and optimization.

9.1 Definition of Quadratic Form

A quadratic form is a type of mathematical expression that can be written as a sum of terms, each of which involves the product of a variable with itself (squared terms) or with another variable (cross terms). In general, a quadratic form is a homogeneous polynomial of degree 2 in a set of variables.

9.1.1 2D Quadratic Form

Let the vector and symmetric matrix in 2D be defined as follows:

\[ \mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}, \quad \mathbf{Q} = \begin{bmatrix} q_{11} & q_{12} \\ q_{12} & q_{22} \end{bmatrix}. \]

The quadratic form is expressed as:

\[ Q(\mathbf{x}) = \mathbf{x}^T \mathbf{Q} \mathbf{x}. \]

Expanding this expression gives:

\[ Q(\mathbf{x}) = \begin{bmatrix} x_1 & x_2 \end{bmatrix} \begin{bmatrix} q_{11} & q_{12} \\ q_{12} & q_{22} \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}. \]

The resulting quadratic form is:

\[ Q(x_1, x_2) = q_{11}x_1^2 + q_{22}x_2^2 + 2q_{12}x_1x_2. \]

9.1.2 3D Quadratic Form

For 3D, the vector and symmetric matrix are defined as follows:

\[ \mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix}, \quad \mathbf{Q} = \begin{bmatrix} q_{11} & q_{12} & q_{13} \\ q_{12} & q_{22} & q_{23} \\ q_{13} & q_{23} & q_{33} \end{bmatrix}. \]

The quadratic form in this case is expressed as:

\[ Q(\mathbf{x}) = \mathbf{x}^T \mathbf{Q} \mathbf{x}. \]

Expanding it gives:

\[ Q(\mathbf{x}) = \begin{bmatrix} x_1 & x_2 & x_3 \end{bmatrix} \begin{bmatrix} q_{11} & q_{12} & q_{13} \\ q_{12} & q_{22} & q_{23} \\ q_{13} & q_{23} & q_{33} \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix}. \]

The expanded quadratic form is:

\[ Q(x_1, x_2, x_3) = q_{11}x_1^2 + q_{22}x_2^2 + q_{33}x_3^2 + 2q_{12}x_1x_2 + 2q_{13}x_1x_3 + 2q_{23}x_2x_3. \]

9.1.3 nD Quadratic Form

In \(n\)-dimensions, the vector and symmetric matrix are defined as:

\[ \mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}, \quad \mathbf{Q} = \begin{bmatrix} q_{11} & q_{12} & \dots & q_{1n} \\ q_{12} & q_{22} & \dots & q_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ q_{1n} & q_{2n} & \dots & q_{nn} \end{bmatrix}. \]

The quadratic form is expressed as:

\[ Q(\mathbf{x}) = \mathbf{x}^T \mathbf{Q} \mathbf{x}. \]

Steps to Calculate a Quadratic Form:

1. Compute the transpose of \(\mathbf{x}\):
   First, express \(\mathbf{x}\) as a column vector of variables: \[ \mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}. \]
   The transpose of \(\mathbf{x}\), denoted as \(\mathbf{x}^T\), is: \[ \mathbf{x}^T = [x_1 \, x_2 \, \dots \, x_n]. \]

2. Multiply \(\mathbf{x}^T\) with the matrix \(\mathbf{Q}\):
   Multiply the row vector \(\mathbf{x}^T\) by the symmetric matrix \(\mathbf{Q}\): \[ \mathbf{x}^T \mathbf{Q} = [x_1 \, x_2 \, \dots \, x_n] \begin{bmatrix} q_{11} & q_{12} & \dots & q_{1n} \\ q_{12} & q_{22} & \dots & q_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ q_{1n} & q_{2n} & \dots & q_{nn} \end{bmatrix}. \]
   This produces a row vector: \[ \bigg[\sum_{j=1}^{n} q_{1j} x_j, \sum_{j=1}^{n} q_{2j} x_j, \dots, \sum_{j=1}^{n} q_{nj} x_j \bigg]. \]

3. Multiply the result by \(\mathbf{x}\):
   Multiply the resulting row vector by the column vector \(\mathbf{x}\):

   \[ Q(\mathbf{x}) = \bigg[\sum_{j=1}^{n} q_{1j} x_j, \sum_{j=1}^{n} q_{2j} x_j, \dots, \sum_{j=1}^{n} q_{nj} x_j\bigg] \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}. \]
   This produces the final quadratic form expression.

4. Final Expression:

   In summation form: \[ Q(\mathbf{x}) = \sum_{i=1}^{n} \sum_{j=1}^{n} q_{ij} x_i x_j. \]

   Separating diagonal and off-diagonal terms: \[ Q(\mathbf{x}) = \sum_{i=1}^n q_{ii} x_i^2 + 2 \sum_{i=1}^{n} \sum_{j=i+1}^n q_{ij} x_i x_j. \]

   Or, in fully expanded form: \[ Q(\mathbf{x}) = q_{11}x_1^2 + q_{22}x_2^2 + \dots + q_{nn}x_n^2 + 2 \Big(q_{12}x_1x_2 + q_{13}x_1x_3 + \dots + q_{n-1,n}x_{n-1}x_n \Big). \]

Key Insights:

• The diagonal terms \(q_{ii}\) correspond to the squared variables \(x_i^2\).
• The off-diagonal terms \(q_{ij}\) (where \(i \neq j\)) represent the cross terms \(x_i x_j\).

9.2 Key Concepts

1. Symmetry of the Matrix \(\mathbf{Q}\):
   The matrix \(\mathbf{Q}\) must be symmetric, meaning that \(q_{ij} = q_{ji}\). This symmetry ensures that the quadratic form only contains real coefficients and simplifies the expression of the form.

2. Diagonal and Off-Diagonal Terms:
   The quadratic form contains two types of terms:

   • Diagonal terms (\(q_{ii}\)), which represent the squared terms of the variables \(x_i^2\).
   • Off-diagonal terms (\(q_{ij}\), for \(i \neq j\)), which correspond to the interactions between different variables \(x_i\) and \(x_j\) (cross terms like \(x_i x_j\)).

3. Scalar Output:
   The result of applying the quadratic form to the vector \(\mathbf{x}\) is a scalar, i.e., a single numerical value.

9.3 Geometric Interpretation of Quadratic Forms

In the context of quadratic forms, the matrix \(\mathbf{Q}\) defines the geometric properties of the surface associated with the quadratic form \(f(\mathbf{x}) = \mathbf{x}^T \mathbf{Q} \mathbf{x}\). The geometry of the surface described depends on the properties of \(\mathbf{Q}\).

9.3.1 Ellipsoid

If \(\mathbf{Q}\) is a positive definite matrix, all the eigenvalues of \(\mathbf{Q}\) are positive. This means that the quadratic form represents a surface where all directions curve outward, such as an ellipsoid in 3D or an ellipse in 2D. In this case, the quadratic form is always positive for any non-zero vector \(\mathbf{x}\), indicating a closed surface. An ellipsoid can be described with the equation:

\[ \frac{x^2}{a^2} + \frac{y^2}{b^2} + \frac{z^2}{c^2} = 1 \]

9.3.2 Hyperboloid

If \(\mathbf{Q}\) is indefinite, meaning that \(\mathbf{Q}\) has both positive and negative eigenvalues, the quadratic form describes a hyperboloid. In this case, the surface could take on one of two general forms:

• A one-sheeted hyperboloid, if there’s a pair of positive and negative eigenvalues.A one-sheeted hyperboloid can be described with the equation:

\[ \frac{x^2}{a^2} - \frac{y^2}{b^2} - \frac{z^2}{c^2} = 1 \] - A two-sheeted hyperboloid, if there’s more complex mixing of positive and negative eigenvalues. A two-sheeted hyperboloid can be described with the equation:

\[ \frac{x^2}{a^2} - \frac{y^2}{b^2} - \frac{z^2}{c^2} = -1 \]

A hyperboloid is an open surface, unlike the ellipsoid. Let consider the following one-sheeted hyperboloid visualization:

9.3.3 Paraboloid

A paraboloid arises when the quadratic form has a special structure, often reflecting a situation where the matrix \(\mathbf{Q}\) has both positive and zero eigenvalues. This typically occurs when there is a critical point along one of the axes. A paraboloid can open upwards, downwards, or sideways, depending on the sign of the nonzero eigenvalue.

• An elliptic paraboloid can be described with the equation:

\[ \frac{x^2}{a^2} + \frac{y^2}{b^2} = z \]

This involves using a positive definite matrix for the elliptic paraboloid.

• A hyperbolic paraboloid can be described with the equation:

\[ \frac{x^2}{a^2} - \frac{y^2}{b^2} = z \]

The type of surface described by a quadratic form is determined by the positive, negative, or zero nature of the eigenvalues of the matrix \(\mathbf{Q}\), which governs the curvature and openness of the graph.

9.4 Applications of Quadratic Forms

Quadratic forms are essential in various disciplines:

• Optimization: In finding the minimum or maximum of a function, especially quadratic functions in constrained optimization problems.
• Statistics: In the context of variance-covariance matrices and regression analysis.
• Physics and Engineering: For representing the energy of a system or describing various physical systems.
• Machine Learning: In algorithms like Support Vector Machines (SVM) and in kernel methods where the quadratic form is used to map data into higher-dimensional spaces.

9.5 Simple Implementation in Python

Below is a Python implementation of a quadratic form:

import numpy as np

# Define values for x1 and x2
x1, x2 = 1, 2  # Example 

# Matrix Q and vector x
Q = np.array([[2, 2],
              [2, 3]])
x = np.array([[x1], [x2]])

# Compute the quadratic form
Q_x = np.dot(x.T, np.dot(Q, x))
print("Quadratic Form Q(x):", Q_x)

Quadratic Form Q(x): [[22]]