When Gaussians Get Uncertain: Probabilistic 3D Reconstruction

W29 Trend Tutorial | Difficulty: Advanced | Research area: Probabilistic 3D reconstruction

The Point-Estimate Problem

Last week we argued that the machinery of scale — bigger transformers, more data — quietly abandoned a tool it badly needs: calibrated uncertainty. This week we watch the same story play out in 3D.

3D Gaussian Splatting (Kerbl et al., SIGGRAPH 2023) represents a scene as a few million anisotropic 3D Gaussians, each carrying a position $\mu \in \mathbb{R}^3$ , a covariance $\Sigma$ (scale + rotation), an opacity $\alpha$ , and view-dependent color coefficients. Optimization fits these parameters to a set of posed images by differentiable rasterization, and the result renders novel views in real time at 1080p — the explicit, real-time alternative to NeRF's implicit MLP radiance field (Mildenhall et al., 2020).

Here is the catch. Every one of those parameters is a point estimate. The optimizer returns a scene — the single configuration that best explains the training views — and says nothing about how many other scenes would have explained them just as well. Where you photographed a surface from twenty angles, that best-fit is trustworthy. Where you saw it once, or never, the optimizer still commits to a confident answer. The failure mode is familiar to anyone who has trained a splat: floaters — confident wisps of geometry hallucinated into regions the cameras never constrained.

Why Geometry Needs to Know What It Doesn't Know

A point-estimate scene is fine for a fly-through demo. It is dangerous the moment a decision depends on the geometry being real.

A robot planning a grasp needs to distinguish "there is definitely a table edge here" from "something might be here — I only saw it once, at a grazing angle."
An AR system occluding virtual objects behind real ones must not trust a reconstructed surface that is actually a reconstruction artifact.
An active-capture pipeline deciding where to photograph next needs to know which regions are still ambiguous.

In each case the question is not "what is the geometry?" but "how sure are we about the geometry, and where?" That is a request for a posterior, not a point estimate — the same kind of object last week's Gaussian Process gave us for free (there, a posterior over function values; here we want one over geometry — an analogy we make precise, and qualify, below).

From a Best-Fit Scene to a Distribution Over Scenes

The Bayesian reframing is direct. Let $\theta$ collect all the scene parameters (every Gaussian's mean, covariance, opacity, color) and let $\mathcal{I}$ be the observed images. Standard splatting maximizes the likelihood (equivalently, minimizes photometric loss) to get a single $\hat{\theta}$ :

$\hat{\theta} = \arg\max_\theta \; p(\mathcal{I} \mid \theta)$

The probabilistic view instead asks for the posterior $p(\theta \mid \mathcal{I}) \propto p(\mathcal{I} \mid \theta)\, p(\theta)$ and, through it, the posterior over any rendered quantity. This is the 3D analogue of the GP posterior from W28:

$\mu_* = K_{*X}(K_{XX} + \sigma_n^2 I)^{-1} y, \qquad \Sigma_* = K_{**} - K_{*X}(K_{XX} + \sigma_n^2 I)^{-1} K_{X*}$

There the variance $\Sigma_*$ was large where data was sparse and small where it was dense. We want the same behavior in a splat: a per-Gaussian — or per-ray — variance that grows in under-observed regions and shrinks where many cameras agree. The obstacle is that $\theta$ has millions of dimensions and the renderer is nonlinear, so the closed-form GP posterior is out of reach. The last two years have produced three tractable — if still compute-hungry — ways around that.

Three Routes to an Uncertain Splat

1. Post-hoc perturbation (a Laplace posterior). Bayes' Rays (Goli et al., CVPR 2024) quantifies uncertainty for any pretrained radiance field (demonstrated on NeRFs) without retraining it. It introduces a small spatial deformation field around the converged geometry and asks how much the rendered images resist being perturbed — formally, a Laplace approximation: near a loss minimum $\hat{\theta}$ , the posterior is approximately Gaussian with covariance equal to the inverse Hessian,

$p(\theta \mid \mathcal{I}) \approx \mathcal{N}\!\left(\hat{\theta},\; H^{-1}\right), \qquad H = \nabla^2_\theta \big[-\log p(\mathcal{I}\mid\theta)\big]\Big|_{\hat\theta}.$

Flat directions of the loss (large inverse curvature) are directions the data failed to constrain — high uncertainty. The output is a volumetric uncertainty field you can render alongside color.

2. Fisher information (which view would tell us most?). FisherRF (Jiang et al., ECCV 2024) works with the Fisher information of the radiance-field parameters — the expected curvature of the log-likelihood, which is exactly the term the Laplace Hessian approximates. Because Fisher information measures how much each observation pins down $\theta$ , FisherRF can both report uncertainty and choose the next camera by maximizing expected information gain (EIG) — the active-learning move that a point estimate cannot even phrase.

3. Uncertainty native to the Gaussians. More recent work bakes uncertainty into 3DGS itself. PUP 3D-GS uses a principled, Hessian-based sensitivity score to decide which Gaussians are well-supported enough to keep and which are expendable — pruning as uncertainty in disguise. View-Dependent Uncertainty Estimation of 3D Gaussian Splatting (2025) carries Bayesian inference through the splatting pipeline to produce uncertainty that varies with viewing direction, adapting the NeRF-era ideas above to the explicit-Gaussian setting.

The through-line: all three are finite-dimensional stand-ins for the same posterior variance the GP handed us analytically. Curvature — Hessian, Fisher information, or a perturbation field — is how you recover "where is the data quiet?" when a closed form is impossible. None of it is free: forming or inverting even a diagonal or low-rank curvature matrix over millions of Gaussians is the real bottleneck, which is exactly why these methods lean on approximations — diagonal Hessians, Kronecker-factored Fisher, or perturbation-field surrogates — rather than the full posterior. Cheap-enough uncertainty, not exact uncertainty, is what makes them usable.

Where the Prior Comes From: Diffusion and Score

Bayes' theorem needs a prior $p(\theta)$ . With dense views the likelihood dominates and the prior barely matters; with few views the prior does the heavy lifting, and a weak prior is why sparse-view splats hallucinate. Where does a strong prior over "plausible 3D scenes" come from?

This is the bridge to the generative side of the probabilistic renaissance. Score-based generative models (Song et al., ICLR 2021) learn the score — the gradient of the log-density,

$s_\theta(x) \approx \nabla_x \log p(x)$

— the gradient of the log-density — and turn it into a sampler by reversing a noising SDE. That learned score encodes a prior: it points toward the configurations that are probable before any observation arrives. Plugging a diffusion prior over geometry into a Bayesian rendering likelihood is how modern sparse-view reconstruction fills in the parts the cameras never saw with plausible, rather than arbitrary, structure.

It also closes a loop back to W28 — carefully. The score is a direction in data space (where does density increase?); the GP posterior mean is a prediction (what is the value here?). They are not the same object. But they read out the same underlying fact — where the data actually lives — from two sides: the score as a gradient over configurations, and the GP as low posterior variance wherever observations are dense. Confident geometry, in both languages, is geometry the data has pinned down. We unpack score matching and this density connection in this week's second basics tutorial.

The Arc Ahead

W28 established the theory: attention as kernel regression, GPs as principled probabilistic extensions, uncertainty as the missing ingredient. W29 (this week) grounds it in geometry — distributions over Gaussians, curvature as uncertainty, diffusion priors for the parts we cannot see.

W30 turns uncertainty into reasoning: how calibrated confidence enables reliable neuro-symbolic inference.
W31 closes the loop with probabilistic world models that plan under uncertainty — acting when they know a state is safe, gathering information when they do not.

A splat that reports its own uncertainty is a small thing. But it is the difference between a model that renders a confident guess and one that can say, honestly, here be dragons — and that honesty is what every downstream decision has been waiting for.

Companion Notebooks

One pure-Python notebook accompanies this tutorial:

NB 00: Bayesian Gaussian Splatting — extends the deterministic 2D-Gaussian fit from the W16 notebook with a mean and variance per parameter, then renders the resulting confidence map: uncertainty is high where few "cameras" observed a region and collapses where many agree. Demonstrates the Laplace/inverse-curvature intuition on a toy scene small enough to run on CPU.

This tutorial is part of the Artifocial research-multimodal series. Previous: W28 Trend Tutorial — Attention Is a Kernel. Next: W30 Neuro-Symbolic Reasoning (coming soon).

Build with AI — early access

Want to go from understanding uncertainty to shipping systems that act on it? Build with AI is our early-access program for engineers moving from tutorials like this to production AI — hands-on guidance instead of guesswork. Early access is opening now.

Join the waitlist →