Multivariate Gaussian Distributions
Defining a Gaussian process can be thought of as providing a way to create arbitrary Multivariate Guassian Distributions (aka Multivariate Normal Distributions). The steps to fit and predict with a Gaussian process can then be thought of mostly as manipulations of Multivariate Gaussians. Here we provide a brief overview of some manipulations.
Starting with a univariate Gaussian,
we can add a scalar,
multiply by a scalar,
or add two distributions,
Similar operations exist for multivariate Gaussian distributions, we can add a vector, \(\mu \in \mathcal{R}^n\)
multiply by a matrix, \(A \in \mathbb{R}^{m, n}\)
add two distributions,
Adding and multiplying distributions can be useful, but the most important operation for Gaussian processes is the conditional distribution. Start by splitting a multivariate Gaussian distribution into two variables,
notice that the two random variables, \(\mathbf{a}\) and \(\mathbf{b}\), are correlated with each other. The conditional distribution, \(\mathbf{a} |b\), gives us the distribution of \(\mathbf{a}\) if we knew the value of \(\mathbf{b} = b\),
Here we’ve started with a joint prior distribution of \(\mathbf{a}\) and \(\mathbf{b}\) and found the posterior distribution, \(\mathbf{a}|b\), of \(\mathbf{a}\) given \(\mathbf{b} = b\). These identities and more can be found in the The Matrix Cookbook which is an extremely valuable resource for linear algebra in general.