synvael/docs/diagnostics.md

Runtime diagnostics

This document explains how the engine monitors itself, what each crate measures, how we aggregate those measurements into the client's statistics panel, and how the server sends its own figures to the client. The panel implementation lives in crates/client/src/stats.rs, but the data sources are spread across renderer, net, server, and shared.

Why this exists

Every metric here is measured, not declared. The nominal tick rate we advertise during the handshake is just a constant; it states what the server wants to run at, so it can never reveal if the server is actually falling behind. The same logic applies everywhere else. A configured frame cap says nothing about achieved frame times, and a configured load radius says nothing about how many chunks are actually resident in memory. Diagnostics exist to bridge that gap. If you want to know if the game is running slowly (and exactly where), the answer must come from live observation rather than configuration.

The immediate motivation for this is that the client is doing a lot of work per frame (materializing deliveries, dispatching mesh jobs, ingesting meshes, culling, and submitting). If a frame-time regression happens, there are several plausible culprits, and it's impossible to distinguish them just by looking at the code.

The measurement layers

Each crate measures only what it can see and exposes a plain snapshot type. Crate boundaries are strictly respected; no crate formats the data itself, and no crate reaches into another's internals.

Source	Type	What it observes
renderer	RenderStats	Uploaded/visible/culled meshes, draw calls, triangles, vertices, geometry bytes, active render mode, projection, swapchain, frames presented, and frames skipped
renderer	GpuInfo	Device name and class, vendor and device IDs, driver and API versions, and total device-local memory. We only query this once since these fields are immutable for the renderer's lifetime
renderer	MemoryUsage	Two independent views of GPU memory: the driver's heap accounting and the renderer's own internal allocator
net	NetStats	Application counters (chunks received, drops received, subscribes sent) plus QUIC path state (RTT, lost packets, congestion window, path MTU, bytes, and datagrams)
client	ChunkStats	Resident chunks, uploaded meshes, in-flight mesh jobs, pending re-meshes, and desired-set size
client	FrameStats	Frame count, mean/min/max frame time, and achieved FPS over the window
client	HostInfo / HostUsage	CPU brand and core count, OS and kernel, followed by process CPU and memory usage against system totals
server	ServerStats	Measured TPS, mean and max tick body, tick-budget utilization, resident and in-flight chunks, connected clients, entities, players, and total uptime

Separating the immutable from the live

We query GpuInfo and HostInfo exactly once, while MemoryUsage and HostUsage are read per window. This split is deliberate. The device name and driver version cannot change while the renderer is alive, so re-querying them every window would waste string allocations just to learn nothing new. On the flip side, live figures are read on demand precisely because they cannot be cached.

MemoryUsage reports the driver's heap figures as an Option because they rely on the VK_EXT_memory_budget extension. If that extension isn't available, the allocator's own figures still report accurately (since our process's suballocations are always known even if the driver's total isn't). Because of this, the panel must render a missing driver figure as genuinely missing, rather than substituting a zero which would imply "no memory in use".

The decode_driver_version function exists because VkPhysicalDeviceProperties::driverVersion is documented as vendor-specific, and two major vendors deviate from the standard Vulkan packing. NVIDIA uses a 10/8/8/6-bit layout, and Intel's Windows driver uses a 14/18-bit split (though its Mesa driver follows the standard Vulkan convention). We unit-test this decoding per vendor, because a mis-decoded driver version is exactly the kind of wrong-but-plausible bug that nobody notices until it causes a problem.

Windowed measurement

Everything is reported over a window, not instantaneously. Both client::stats::STATS_INTERVAL and server::tick_stats::REPORT_INTERVAL are set to one second. This is short enough to surface stalls promptly, but long enough that producing the report costs effectively nothing compared to the actual work it summarizes.

A window carries both a mean and a maximum for one very specific reason: they answer different questions. A mean that sits comfortably inside budget alongside a spiking maximum indicates an intermittent stall (a "hitch"), whereas a mean sitting at budget indicates sustained overload. If we only reported the mean, we would completely hide the first case, which is what users actually feel while playing.

The server additionally reports tick_budget_percent, which is the share of the nominal tick period consumed by the mean tick body. We derive this rather than measuring it directly, but it is the crucial figure that tells us whether we have any headroom left. A value at or above 100 means the loop is completely tapped out. We time the tick body excluding the sleep that pads a tick out to its period, ensuring the number reflects actual engine work rather than forced pacing.

TickMeter computes this safely without any division-by-zero hazards. A zero period simply means no budget exists to consume, so utilization is undefined and reported as zero rather than causing a crash or returning infinity.

Collection is unconditional, emission is gated

You can toggle the panel using the F1 + I chord (defined in crates/client/src/debug.rs), but that toggle only gates emission. Data accumulation runs continuously whether the panel is visible or not, and the window closes on its normal schedule either way.

This is much more important than it sounds. If we gated collection on the toggle, the first window after enabling the panel would be partial. It would report a fraction of a second of frames as though it were a full window. Since the first thing anyone does when the game feels wrong is turn the panel on, the figures must already be perfectly accurate at that exact moment.

When enabled, the panel emits through tracing at the info level as a multi-line block. This aligns with our project-wide ban on using println! for diagnostics. The server formats its own figures the exact same way, ensuring that a dedicated server's log and a client's panel present identical numbers in identical formats.

Getting the server's figures to the client

ServerStats is a shared protocol type that the server pushes down the authority stream (stream 2) once per window. The design for this stream is documented in ADR-0011. The client drains this stream non-blockingly every frame and simply holds onto the most recent snapshot. This ensures the panel always has a value to display, even though the server and client windows are naturally misaligned.

This retention strategy is fully intentional. Trying to align the two cadences perfectly would require complex synchronization just for a display figure. Trading that complexity for a snapshot that might be up to a second old is the correct choice. If the staleness ever matters, the server's own uptime_secs field makes it completely visible.

The server also formats and logs the same ServerStats locally, which means you can fully diagnose a dedicated host without ever needing a client attached.

What is not on the wire

You might notice that ServerKind (integrated, dedicated local, or dedicated remote) is deliberately not a protocol field. The client already knows the answer without asking; it either spawned a server in-process, or it dialed a socket. If it dialed a socket, checking for a loopback address perfectly distinguishes a locally hosted process from a remote one. Adding a server-declared field for this would be redundant at best and spoofable at worst, so we construct the value client-side using facts the client already holds.

This demonstrates a general rule: a diagnostic should be sourced from whichever side observes it. The server reports its own tick health because only it can measure that, but the client classifies the session because only the client knows how the session was established.

Concurrency

We cross two concurrency boundaries here, and we use a different primitive for each.

Client net counters (NetCounters) are incremented on the async chunk task and read on the winit thread. They are held behind an Arc and mutated using relaxed atomics. Using relaxed ordering here isn't just about performance; it is strictly correct. Each counter is independent, nothing else is ordered against them, and if a reader observes a slightly stale value, it's just reporting a diagnostic figure rather than making a critical gameplay decision. Paying for stronger memory ordering would just buy precision that nobody consumes.

Renderer frame stats are populated at the very end of every successful draw_frame and retained whole until the next frame replaces them. Because of this, a reader polling on the panel's one-second cadence will always observe a complete, self-consistent frame snapshot rather than a half-updated struct or a field-by-field mix. This property is exactly what makes it safe for the panel to run on a completely independent cadence from the render loop.

Testing

Formatting and derivation logic are pure, so we test them extensively. Live capture, however, is not unit-tested.

• crates/server/src/tests/tick_stats.rs: Tests window closing, mean and max derivation, and budget utilization (including the zero-period edge case).
• crates/renderer/src/tests/stats.rs: Tests decode_driver_version for each vendor, and cull_ratio_percent (including the case where nothing is uploaded).
• crates/client/src/tests/stats.rs: Tests frame accumulation and panel formatting, making sure it handles absent optional sources correctly.
• crates/shared/src/tests/session.rs: Tests ServerKind classification from both loopback and non-loopback addresses.

Vulkan device queries, sysinfo host readings, and live QUIC path statistics all fundamentally depend on real hardware and a live network connection, so they are verified manually by running the client.