Skip to content

Headless state dumps, offscreen screenshots and a screenshot API - #248

Merged
BenjaFriend merged 4 commits into
mainfrom
feature/headless-capture
Nov 17, 2026
Merged

BenjaFriend merged 4 commits into
mainfrom
feature/headless-capture

Conversation

@BenjaFriend

Copy link
Copy Markdown
Member

Feature/Issue

Agents (and scripts) had to open a real window to see the game. This PR lets them look at a run with no window:

  • State as text: --dump-state <file.json> (works with --null-render, no GPU).
  • Rendered frames: --offscreen --screenshot <file.png> renders with Vulkan into images the engine owns. No window, no surface, no GLFW, so it runs on a machine with no display. A software driver (llvmpipe) is enough.
  • C++ API: Fling::Screenshot::Request(path, callback) takes a screenshot from anywhere in the engine, plus a Take screenshot button in the editor's GPU Options window.

Implementation/Solution

--dump-state F [--dump-state-ticks L] (World::SaveStateToJson)

  • Written on the game thread right after each listed tick (default: the last tick of the run).
  • Contains every entity and singleton whose type the state hash covers (rollback state) or a level saves, with every field (Transient ones too), plus tick, simHz, entityCount and stateHash.
  • Entities without an EntityGuid get one made from their handle, so the same run writes the same bytes in every exec mode.

--offscreen, --resolution WxH, --screenshot F [--screenshot-ticks L]

  • Swapchain has an offscreen mode that owns its images. Acquire and present become semaphore-only submits, so VulkanApp::Update runs the same code with or without a window.
  • Instance skips the GLFW/surface extensions offscreen. LogicalDevice uses the graphics queue when there is no surface.
  • The surface and swap chain commands are optional in VulkanFunctions.inl, and checked only when there is a window.
  • Offscreen runs use the headless timing of --null-render (virtual clock, no frame cap), so they give the same ticks.
  • --screenshot saves the first frame that shows each tick. It works offscreen and windowed.
  • A capture that was asked for and not made fails the run with code 1.

Fling::Screenshot (Core/inc/Screenshot.h)

  • Request is safe on any thread and needs no Graphics header.
  • The renderer serves waiting requests at its next frame, before the present, and runs each callback on the main thread.
  • With no path, the file goes to Saved/Screenshots/Screenshot_<local time>_<n>.png.
  • Under --null-render, and when a run ends, waiting requests fail with the reason.
  • --screenshot and the GPU Options button both use it.

Two crashes fixed on software Vulkan (llvmpipe)

Both were already on main, and windowed play crashed the same way:

  • The skybox pipeline declared mesh vertex input but drew with no vertex buffer. Fixed with GraphicsPipeline::m_HasVertexInput = false for the skybox.
  • llvmpipe's uniform inlining miscompiles the two UBO-bounded light loops in deferred.frag (a null pointer read in JIT code). The loop bounds now depend on a value it can't inline; the output is the same.

Docs: Skills/testing.md ("Seeing the game without a window"), AGENTS.md, Docs/agent-guide.md, Plans/improve-ai-iteration.md.

Dependencies (if any?)

  • None new. stb_image_write is already vendored in external/stb.

Screenshots/GIF (if applicable)

Run on a cloud container with no display:

Sandbox --offscreen --unattended --frames 60 --seed 1 --resolution 1280x720 --screenshot out.png --dump-state state.json

It rendered the lit Sandbox (spheres, point lights, skybox) at 1280x720 in about 10 s on llvmpipe. The editor button was also tested on a virtual X display: clicking it saved the frame with the editor UI and showed the path.

Tests

  • Have you reviewed your own code for quality?
  • Did you the Sandbox game project and get the expected results? Offscreen, windowed (Xvfb) and --null-render, in direct, inline and threaded exec modes; screenshots and state dumps are byte-identical across modes.
  • Did you run the FlingTest suite and ensure that there are no regressions? 674/674 locally on Linux GCC Debug, with FLING_TEST_OFFSCREEN=ON.

New tests:

  • ScreenshotTests, WorldStateDumpTests, and new LaunchOptions/RunSettings cases.
  • CTest Sandbox: headless state dump (direct and threaded dumps must match byte for byte) and Sandbox: state dump of a tick that does not run.
  • CTest Sandbox: offscreen screenshots, only with -DFLING_TEST_OFFSCREEN=ON because it needs a Vulkan driver.

Known: TickLimiter ticks are never early failed about 1 in 15 runs locally. It is a wall-clock test of code this PR doesn't touch.

🤖 Generated with Claude Code

BenjaFriend and others added 3 commits November 15, 2026 16:00
Agents can now see a run without opening a window.

--dump-state F [--dump-state-ticks L] writes the simulation state as JSON
on the game thread right after each tick: every entity and singleton of
the types the state hash covers (rollback state) plus the saved ones,
with every field, the tick and its state hash. Works with --null-render.
The same run writes the same bytes in every exec mode.

--offscreen renders with Vulkan into images the renderer owns: no
window, no surface, no GLFW, no window system extension, with the
virtual clock and no frame cap of --null-render. --screenshot F
[--screenshot-ticks L] saves the frame that first shows each tick as a
PNG, offscreen or windowed. --resolution WxH sets the frame or window
size. A capture that was asked for and not made fails the run (code 1).

Also fixes two crashes on software Vulkan (llvmpipe), which agent and
CI machines use. Both were also in windowed play:
- The skybox pipeline declared mesh vertex input but drew with no
  vertex buffer (GraphicsPipeline::m_HasVertexInput).
- llvmpipe's uniform inlining miscompiles the two light loops of
  deferred.frag; the loop bounds now depend on a value it cannot inline.

The surface and swap chain Vulkan commands are optional in the loader
table and checked only when there is a window.

Tests: LaunchOptions and RunSettings cases, World state dump tests, the
CTests "Sandbox: headless state dump" (direct and threaded dumps are
byte-identical) and, with FLING_TEST_OFFSCREEN=ON, "Sandbox: offscreen
screenshots". Documented in Skills/testing.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fling::Screenshot::Request(path, callback) (Core/inc/Screenshot.h) asks
for the next frame the renderer draws to be saved as a PNG. It is safe
on any thread and needs no Graphics header, so the editor, frame
systems, tools and tests can all call it. The renderer serves every
waiting request at its next frame, before the present, and runs each
callback on the main thread. With no path the file goes to
Saved/Screenshots/Screenshot_<local time>_<n>.png. Under --null-render,
and at the end of a run, waiting requests fail with the reason, so a
caller always hears back.

--screenshot now goes through the same API, which replaces
VulkanApp::RequestCapture. FrameCapture reads the frame once and
writes one PNG per request.

The editor's GPU Options window has a Take screenshot button that saves
the next frame, editor UI included, and shows the path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Before, every frame paid for screenshots even with none requested:
- the renderer locked the request queue each frame (and so did the
  --null-render loop);
- every swap chain image, and every offscreen image, was made with
  VK_IMAGE_USAGE_TRANSFER_SRC_BIT, which can turn off framebuffer
  compression on some GPUs.

Now:
- Screenshot::HasPending is one atomic load; TakePending and
  FailPending take no lock and allocate nothing when it is false.
- Swapchain::SetCopyable decides whether its images get the copy usage.
  It is off by default. A run with --screenshot is copyable from the
  start (RenderSurfaceDesc::CopyableFrames) and never rebuilds the swap
  chain. Otherwise a request makes the swap chain again with copyable
  images for its frame (VulkanApp::PrepareScreenshots), saves it, and
  makes it again without them after that frame's present.
- Offscreen images that are not copyable end the frame in
  COLOR_ATTACHMENT_OPTIMAL, as TRANSFER_SRC_OPTIMAL needs the usage.
  RecreateSwapchain works offscreen too.
- Engine::RequestScreenshots returns at once with no --screenshot tick
  pending.

Checked with the validation layer: --screenshot at launch, a runtime
request offscreen, no request, and the GPU Options button in a window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A capture is now a copy in the frame's own command buffer plus a writer
thread, instead of a queue-idle copy and a PNG encode on the main thread.

- FrameCapturer records vkCmdCopyImageToBuffer into the frame after the
  render pass, into a reused, mapped staging buffer. After the frame's
  fence the buffer goes to the "Screenshot Writer" thread, which
  converts, encodes the PNG once and writes it to every requested path.
  Callbacks run on the main thread on a later frame. The thread and
  buffers are made on the first capture only.
- The swap chain images get TRANSFER_SRC once (when the surface allows
  it; always offscreen), so a capture never rebuilds the swap chain.
- Offscreen frames skip the present semaphores, removing two empty
  vkQueueSubmit calls per frame.
- deferred.frag's llvmpipe workaround runs only on CPU drivers, through
  a specialization constant, so GPUs keep the plain light loops.
- FlingPaths::SaveFile replaces three copies of "make the folders, then
  write the file".
- VulkanApp::FinishScreenshots waits for the writes at the end of a run,
  so --screenshot still checks every file before it exits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BenjaFriend
BenjaFriend merged commit fc9581b into main Nov 17, 2026
14 checks passed
@BenjaFriend
BenjaFriend deleted the feature/headless-capture branch November 17, 2026 12:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant