Benchmark11 October 2026

SplatBench

A benchmark and RL environment for raw spatial reasoning.

We give a language model eight photos of an object and ask it to rebuild the object in 3D. It places every Gaussian itself through tool calls, with no code and no optimizer, and we grade the build from eight camera angles it never saw, so only a correct 3D structure scores well.

This explorer replays all 1,230 episodes from the SplatBench report call by call, with the scene after every edit and the held-out score that the model never sees. The environment, the 30 tasks and every trajectory are on Hugging Face.

Leaderboard

Each model ran every task twice at medium reasoning effort. SplatScore measures how closely a build matches the real photos from the eight held-out angles: 0 is an empty scene, and 100 is a gradient-descent fit with the same 300 Gaussians and the same reference photos.

SplatScore on held-out views at medium reasoning effort. Each dot is one episode, each bar a model's mean and each whisker a 95% bootstrap interval over tasks. The dashed lines mark two baselines: a single Gaussian in the object's average color, and gradient descent with 64 Gaussians. Shaded rows have no significant gap to their neighbors, and scores below −40 are drawn at the left edge. Select a dot to replay its episode.

Model	SplatScore [95% CI]	Cost per episode	Time per episode
GPT-6 Astra	49.6 [41.4, 58.0]	$1.69	6.6 min
GPT-6.1 Sol	38.6 [30.6, 46.3]	$0.26	5.1 min
Claude Opus 5.5	25.0 [17.1, 32.6]	$0.71	5.0 min
GPT-6 Luna	8.6 [1.8, 15.2]	$0.02	4.5 min
Claude Sonnet 5.5	5.7 [0.2, 10.6]	$0.04	0.7 min
Claude Haiku 5.5	3.6 [−5.8, 12.4]	$0.01	1.9 min
Grok 4.7	2.1 [−8.8, 11.8]	$1.38	24.0 min
GPT-5.6 Terra	1.2 [−7.4, 9.4]	$0.19	2.1 min

GPT-6 Astra beats GPT-6.1 Sol on 29 of the 30 tasks, and Sol beats Claude Opus 5.5 on 28. The remaining models score near a single Gaussian in the object's average color, and most of the differences among them are within noise.

Replay any episode

Every episode can be replayed call by call, at each effort level, with the early submit refused and in one shot. The line above the timeline is the held-out score. The Groups view colors each Gaussian by the label the model gave it, and the Held-out view shows the scene from a grading camera next to its photo.

The episode player. Choose a setting, a model, a task and a run, or select a score in the leaderboard or the task table.

Every task

The procedural objects are built from spheres, boxes, cylinders and cones, so their 3D ground truth is exact. The synthetic objects come from NeRF-Synthetic, and the photographed objects come from Tanks and Temples and BlendedMVS. Every object appears both as a build task and as a reassembly task, in which the model moves the scrambled parts of a fitted splat back into place.

Mean SplatScore per task and model at medium effort, next to gradient descent with 64 Gaussians on the same task. The bold score is the best on each task. Select a score to replay that model's episode.

Reasoning effort

All models except Grok 4.7 ran again at high and maximum effort on half of the tasks. Higher effort raises almost every model's score, and the cost per episode rises much faster than the score. GPT-6.1 Sol at maximum effort beats GPT-6 Astra at medium effort for less than half the cost.

Mean SplatScore against the cost of one episode at medium, high and maximum reasoning effort, on half of the tasks. Along each line, effort rises from left to right. Point at a line or a name to label its effort levels.

Render feedback

In one-shot reconstruction the model sees the photos and gets a single tool call, with no chance to render the scene or correct it. GPT-6 Astra, GPT-6.1 Sol and Claude Opus 5.5 are statistically level in one shot, so Astra's lead in the full episode comes from how it uses render feedback.

Mean SplatScore on the build tasks in one shot and in the full episode at medium effort, with the gain from the full episode.

Mean held-out score after each tool call at medium effort, over the build tasks. An episode that ended early keeps its final score.

The environment

An episode is a loop between the model and the environment. Each tool call is one step that returns text and sometimes an image, and the reward arrives once, at the end of the episode. The held-out photos, the gradient-descent fit and the score stay with the grader, and the trajectory log keeps the hidden score after every edit, which can serve as a process reward in training.

The SplatBench environment. The model sees the reference photos, the text of each tool result and the renders of its own scene, while the held-out photos, the gradient-descent fit and the score stay with the grader.

Fixed points on the SplatScore scale, each a mean over the tasks it applies to.

The environment installs from the Hugging Face dataset, with a TRL adapter for GRPO and the per-step trajectories of every episode.

hf download perpetual-research/splatbench --repo-type dataset --local-dir splatbench
cd splatbench && uv sync

from splatbench import SplatEnv, load_task
env = SplatEnv(device="cuda")
obs = env.reset(load_task("data/bench/tasks/table__sculpt.json"))
out = env.add_gaussians([{"pos": [0, 0, 0.1], "scale": [0.45, 0.35, 0.03], "color": "#805e31"}])
out = env.render_views(views=[0, 1, 6, 7])
out = env.submit()
reward = env.get_reward()   # clip(SplatScore / 100, 0, 1)

Citation

@misc{perpetual2026splatbench,
  title  = {SplatBench: Raw Spatial Reasoning by Placing 3D Gaussians},
  author = {{Perpetual Research}},
  year   = {2026},
  url    = {https://perpetualresearch.si/splatbench}
}