Matlab implementations of Gaussian processes and other

GPmat ==== The GPmat toolbox is the 'one stop shop' on github for a number of dependent toolboxes, each of which used to be released independently. Since May 2015 each toolbox is a sub-directory within GPmat. They are included as subtrees from the relevant repositories. The summary of demos from each toolbox is given below. It is advisable that you also include netlab (https://github.com/sods/netlab) as a dependency. The first release of the full GPmat toolbox is version 1.0.0, released on 28th May 2015 to coincide with the reformatting of the toolbox as sub-trees. GP == ### Version 0.136 Changes to gpReadFromFID for compatibility with C++ code. #### Version 0.135 Modifications by Carl Henrik Ek for compatability with the SGPLVM toolbox. #### Version 0.134 Updates to allow deconstruction of model files when writing to disk (gpWriteResult, gpLoadResult, gpDeconstruct, gpReconstruct). #### Version 0.133 Updates for running a GPLVM/GP using the data's inner product matrix for Interspeech synthesis demos. #### Version 0.132 Examples transfered from oxford toolbox, variational approximation from Titsias added as an option with 'dtcvar'. #### Version 0.131 Changes to allow compatibility with SGPLVM and NCCA toolboxes. #### Version 0.13 Changes to allow more flexibility in optimisation of beta. #### Version 0.12 Various minor changes for enabling back constraints in hierarchical GP-LVM models. #### Version 0.11 Changes include the use of the optimiDefaultConstraint('positive') to obtain the function to constrain beta to be positive (which now returns 'exp' rather than 'negLogLogit' which was previously the default). Similarly default optimiser is now given by a command in optimiDefaultOptimiser. #### Version 0.1 The first version which is spun out of the FGPLVM toolbox. The corresponding FGPLVM toolbox is 0.15. Release 0.1 splits away the Gaussian process section of the FGPLVM toolbox into this separate toolbox. ## Other GP related software The GP-LVM C++ software is available from <a href="/~neill/gplvmcpp/">here</a>. The IVM C++ software is available from <a href="/~neill/ivmcpp/">here</a>. The MATLAB IVM toolbox is available here <a href="/~neill/ivm/">here</a>. The original MATLAB GP-LVM toolbox is available here <a href="/~neill/gplvm/">here</a>. ## Examples ### Functions from Gaussians This example shows how points which look like they come from a function to be sampled from a Gaussian distribution. The sample is 25 dimensional and is from a Gaussian with a particular covariance. ```matlab >> demGpSample ``` <center><img src="./gp/diagrams/gpSample.png" width ="50%"><img src="./gp/diagrams/gpCovariance.png" width ="50%"><br> <i>Left</i> A single, 25 dimensional, sample from a Gaussian distribution. <i>Right</i> the covariance matrix of the Gaussian distribution.. </center> ### Joint Distribution over two Variables Gaussian processes are about conditioning a Gaussian distribution on the training data to make the test predictions. To illustrate this process, we can look at the joint distribution over two variables. ```matlab >> demGpCov2D([1 2]) ``` Gives the joint distribution for <i>f</i>₁ and <i>f</i>₂. The plots show the joint distributions as well as the conditional for <i>f</i>₂ given <i>f</i>₁. <center><img src="./gp/diagrams/demGpCov2D1_2_3.png" Width ="50%"><img src="./gp/diagrams/demGpCov2D1_5_3.png" width ="50%"><br> <i>Left</i> Blue line is contour of joint distribution over the variables <i>f</i>₁ and <i>f</i>₂. Green line indicates an observation of <i>f</i>₁. Red line is conditional distribution of <i>f</i>₂ given <i>f</i>₁. <i>Right</i> Similar for <i>f</i>₁ and <i>f</i>₅. </center> ### Different Samples from Gaussian Processes A script is provided which samples from a Gaussian process with the provided covariance function. ```matlab >> gpSample('rbf', 10, [1 1], [-3 3], 1e5) ``` will give 10 samples from an RBF covariance function with a parameter vector given by [1 1] (inverse width 1, variance 1) across the range -3 to 3 on the <i>x</i>-axis. The random seed will be set to 1e5. ```matlab >> gpSample('rbf', 10, [16 1], [-3 3], 1e5) ``` is similar, but the inverse width is now set to 16 (length scale 0.25). <center><img src="./gp/diagrams/gpSampleRbfSamples10Seed100000InverseWidth1Variance1.png" width ="50%"><img src="./gp/diagrams/gpSampleRbfSamples10Seed100000InverseWidth16Variance1.png" width ="50%"><br> <i>Left</i> samples from an RBF style covariance function with length scale 1. <i>Right</i> samples from an RBF style covariance function with length scale 0.25. </center> Other covariance functions can be sampled, an interesting one is the MLP covariance which is non stationary and can produce point symmetric functions, ```matlab >> gpSample('mlp', 10, [100 100 1], [-1 1], 1e5) ``` gives 10 samples from the MLP covariance function where the &quot;bias variance&quot; is 100 (basis functions are centered around the origin with standard deviation of 10) and the &quot;weight variance&quot; is 100. ```matlab >> gpSample('mlp', 10, [100 1e-16 1], [-1 1], 1e5) ``` gives 10 samples from the MLP covariance function where the &quot;bias variance&quot; is approximately zero (basis functions are placed on the origin) and the &quot;weight variance&quot; is 100. <center><img src="./gp/diagrams/gpSampleMlpSamples10Seed100000WeightVariance100BiasVariance100Variance1.png" width ="50%"><img src="./gp/diagrams/gpSampleMlpSamples10Seed100000WeightVariance100BiasVariance1e-16Variance1.png" width ="50%"><br> <i>Left</i> samples from an MLP style covariance function with bias and weight variances set to 100. <i>Right</i> samples from an MLP style covariance function with weight variance 100 and bias variance approximately zero. </center> ### Posterior Samples Gaussian processes are non-parametric models. They are specified by their covariance function and a mean function. When combined with data observations a posterior Gaussian process is induced. The demos below show samples from that posterior. ```matlab >> gpPosteriorSample('rbf', 5, [1 1], [-3 3], 1e5) ``` and ```matlab >> gpPosteriorSample('rbf', 5, [16 1], [-3 3], 1e5) ``` <center><img src="./gp/diagrams/gpPosteriorSampleRbfSamples5Seed100000InverseWidth1Variance1bw.png" width ="50%"><img src="./gp/diagrams/gpPosteriorSampleRbfSamples5Seed100000InverseWidth16Variance1bw.png" width ="50%"><br> <i>Left</i> samples from the posterior induced by an RBF style covariance function with length scale 1 and 5 &quot;training&quot; data points taken from a sine wave. <i>Right</i> Similar but for a length scale of 0.25. </center> ### Simple Interpolation Demo This simple demonstration plots, consecutively, an increasing number of data points, followed by an interpolated fit through the data points using a Gaussian process. This is a noiseless system, and the data is sampled from a GP with a known covariance function. The curve is then recovered with minimal uncertainty after only nine data points are included. The code is run with ```matlab >> demInterpolation ``` <center><img src="./gp/diagrams/demInterpolation3.png" width ="50%"><img src="./gp/diagrams/demInterpolation4.png" width ="50%"><br> Gaussian process prediction <i>left</i> after two points with a new data point sampled <i>right</i> after the new data point is included in the prediction.<br> <img src="./gp/diagrams/demInterpolation7.png" width ="50%"><img src="./gp/diagrams/demInterpolation8.png" width ="50%"><br> Gaussian process prediction <i>left</i> after five points with a four new data point sampled <i>right</i> after all nine data points are included.<br> </center> ### Simple Regression Demo The regression demo very much follows the format of the interpolation demo. Here the difference is that the data is sampled with noise. Fitting a