Add Gaussian Processes

[FriesischScott]

This PR replaces #245 with one originating from within the repository.

There is still some work needed to clean the code:

• Implement noise directly in GP
  • Add or un-add sigma in optimization here
• Refine user interface
• Schlepp data with
• Add refinement
• Check and aufräum data transformation

[lukasfritsch]

I started some work on adaptive GPs based on the current state of the GP implementation. Do we want to add this in this PR or later?

[FriesischScott]

I started some work on adaptive GPs based on the current state of the GP implementation. Do we want to add this in this PR or later?

Let's do the base here and the refinement in a second step.

[Copilot]

Pull request overview

Copilot reviewed 25 out of 26 changed files in this pull request and generated 6 comments.

Suppressed comments (3)

src/models/gp/gaussianprocess.jl:126

• Docstring says σ² defaults to 0.0, but the constructor default is σ²::Float64 = 1.0e-10. Please keep the documented default consistent with the implementation.

- `σ²`: The noise variance. Defaults to 0.0.

src/models/gp/gaussianprocess.jl:208

• Docstring says σ² defaults to 0.0, but the constructor default is σ²::Float64 = 1.0e-10. This should be aligned for accurate docs.

- `σ²`: The noise variance. Defaults to 0.0.

src/models/gp/gaussianprocess.jl:280

• Docstring says σ² defaults to 0.0, but the constructor default is σ²::Float64 = 1.0e-10. Please update the docs or the default.

- `σ²`: The noise variance. Defaults to 0.0.

[lukasfritsch]

The tests in models/gp/hyperparametertuning.jl seem to be unstable and I'm unsure why that is. I thought it was due to the variance initially being set to 0.

Also, I'm unsure how the ARDTransform works. When I understand the docs right, adding ARDTransform([1.0, 1.0]) should just multiply each element by 1.0.
However, when I removed it from the Himmelblau example, the results are much better (in terms of MSE). But then, when I tried out the adaptive GPs for reliability, the learning only converges when using the kernel along with the ARDTransform([1.0])