96 lines
13 KiB
Markdown
96 lines
13 KiB
Markdown
# Chunk streaming
|
|
|
|
This covers how the server keeps resident chunks in sync with player positions and how we keep chunk generation completely off the main simulation tick. The generation code lives in [`crates/shared/src/generator.rs`](../crates/shared/src/generator.rs), while the streaming and storage logic is in [`crates/server/src/world_server.rs`](../crates/server/src/world_server.rs), driven from [`crates/server/src/main.rs`](../crates/server/src/main.rs).
|
|
|
|
## Overview
|
|
|
|
Every tick, the server reconciles chunk residency against a **desired set**, which is the union of cylinders around every player's anchor. If a chunk is in the desired set, we make it resident; if it falls outside, we evict it. Because generating a missing chunk is expensive, we hand that work off to a dedicated thread pool rather than blocking the tick. This makes the reconciliation process entirely non-blocking: each pass drains whatever the pool has finished, evicts chunks that are no longer needed, and dispatches requests for anything still missing, all without ever waiting for a chunk to finish generating.
|
|
|
|
## Desired set
|
|
|
|
The `cylinder_chunks(center, radius, out)` function takes every chunk position within the streaming cylinder around the `center` and inserts it into `out`. This shape is a horizontal disc (`dx² + dz² ≤ radius²`) extruded vertically to `±radius/2`. We use this shape because horizontal view distance matters much more than vertical.
|
|
|
|
We build these desired sets in two places. During the **startup loading gate**, the `stream_chunks` ECS system queries every entity that has `Player`, `Position`, and `ViewDistance` components, unioning each anchor's cylinder to pre-warm the origin region before the network is even up. During **steady-state play**, the desired set becomes the union of every connected client's active subscription (calculated as `cylinder_chunks(center, radius)` around its camera), which we assemble from the `ClientStream` map in the main loop. In both cases, overlapping cylinders deduplicate automatically because we union the sets, meaning a chunk only gets evicted when absolutely *no* subscriber wants it anymore. See the [Multiplayer](#multiplayer) section for more on this.
|
|
|
|
## The worker pool
|
|
|
|
The `ServerWorld::new` function builds the worker pool once. Here is how it is structured:
|
|
|
|
- **Generator sharing:** We wrap the `VoxelGenerator` in an `Arc` and clone a handle into every worker. Because `generate_chunk(&self)` is read-only, we don't need a `Mutex`; all workers share a single immutable generator safely.
|
|
- **Job channel (main to workers):** This is a `crossbeam-channel` that carries `ChunkPos`. We specifically use `crossbeam` instead of `std::sync::mpsc` because the pool requires multiple consumers. Every worker clones the `Receiver` to pull from a single shared queue, and each job goes to exactly one worker. The standard library's `mpsc` only allows a single consumer.
|
|
- **Result channel (workers to main):** Another `crossbeam-channel` carrying `(ChunkPos, Chunk)`. Every worker clones the `Sender`, and the main thread holds the single `Receiver`.
|
|
- **Worker loop:** Each worker blocks on `job_rx.recv()`, generates the requested chunk, and sends the result back. Blocking on `recv` here is perfectly fine since this runs on a worker thread, not the main simulation thread.
|
|
|
|
### Channel disconnection and shutdown
|
|
|
|
A `crossbeam-channel` will only report disconnection (returning an `Err` on `recv`) when all senders or receivers have been completely dropped. Right after spawning the pool, we immediately drop the original `job_rx` and `result_tx` templates. If we held onto them, the channels would stay open forever, meaning the workers would never realize the job channel shut down and the main thread would never see the result channel close. We do keep the worker `JoinHandle`s on `ServerWorld` for a future graceful shutdown path, but right now the process just relies on the OS tearing down threads on exit.
|
|
|
|
## Reconcile: drain, evict, dispatch
|
|
|
|
The `ServerWorld::reconcile(&mut self, desired)` function runs three non-blocking phases during each pass:
|
|
|
|
1. **Drain:** We pull from `result_rx.try_recv()` in a loop until it is empty (which never blocks). We remove each returned position from the `in_flight` set. A returned chunk only gets inserted into the resident map if it is still present in the `desired` set (see [the eviction race](#the-eviction-race)).
|
|
2. **Evict:** Any resident chunks that are no longer in the `desired` set are removed. If an in-flight chunk is no longer wanted, we don't need to handle it here; the drain phase will naturally discard it when it finally arrives.
|
|
3. **Dispatch:** For every position in the `desired` set that isn't resident and isn't already `in_flight`, we insert it into `in_flight` and send it down `job_tx`. The `in_flight` set ensures we don't spam the same position into the queue on every single pass while a worker is busy generating it.
|
|
|
|
This makes `in_flight` our single source of truth for tracking work that has been dispatched but hasn't returned yet.
|
|
|
|
### The eviction race
|
|
|
|
While a chunk is off being generated by a worker, the player might move away, meaning the chunk is no longer wanted. If we weren't careful, the drain phase would insert this unwanted chunk into the map, effectively resurrecting a chunk that the evict phase had already tossed out. By guarding the drain phase (only inserting if `desired.contains(&pos)`), a late arrival is totally harmless. The unwanted chunk just gets dropped on arrival instead of becoming resident.
|
|
|
|
## Startup loading gate
|
|
|
|
Startup uses the exact same worker pool and schedule; we intentionally avoid building a separate synchronous loading path. Before handing control over to the player, `main` runs the streaming schedule in a tight loop and polls `ServerWorld::streaming_idle()` (which returns true when `in_flight` is empty). Once the starting region has at least one resident chunk and zero work in flight, it is ready to go. It is perfectly fine to wait here because actual gameplay hasn't started yet. Once the game is running, this exact same reconcile logic runs every tick but is *never* waited on. We calculate loading progress as a simple fraction: `resident / (resident + in_flight)`.
|
|
|
|
## Network delivery (client and server)
|
|
|
|
While residency keeps chunks loaded in the server's memory, **delivery** is responsible for streaming those chunks to clients. These concepts are strictly decoupled. The reconcile pool knows nothing about clients, and the delivery system never generates chunks. Delivery logic lives in [`crates/net/src/chunk.rs`](../crates/net/src/chunk.rs) for transport and [`crates/server/src/client_stream.rs`](../crates/server/src/client_stream.rs) for server-side bookkeeping. The client counterpart is in [`crates/client/src/chunks.rs`](../crates/client/src/chunks.rs).
|
|
|
|
### The chunk stream
|
|
|
|
Once the handshake finishes, the client opens a single bidirectional QUIC stream (specifically stream 3, `StreamLayout::chunk_lod0`) and the server accepts it. Both directions ride on this one stream. The client sends `ChunkSubscribe { center, radius }` to the server, and the server replies with `ChunkMessage::{Chunk { pos, data }, Drop { pos }}`. These frames use our length-prefixed `postcard` codec, capped at a dedicated 1 MiB limit (`MAX_CHUNK_FRAME_LEN`), which is much larger than the 64 KiB control limit.
|
|
|
|
### The async/sync bridge
|
|
|
|
The QUIC network pump is completely async, but the server simulation and client `winit` loops are totally synchronous. To bridge this gap, we use two channels per connection (one for each direction) and pick different primitives based on the direction:
|
|
|
|
- **Inbound:** A `ChunkSubscribe` arrives via async and is consumed by the sync loop. We route this through the `crossbeam` `ServerEvent` channel, surfacing it as `ServerEvent::ChunkSubscribe { id, request }`. The async `send` is non-blocking, and the sync loop simply drains it with `try_iter`.
|
|
- **Outbound:** A `ChunkMessage` is produced by the sync loop and consumed via async. We use an unbounded MPSC from `tokio` for this. The synchronous send works without needing an async runtime, while the pump can `await` the `recv()` cleanly inside a `tokio::select!`. If we used a blocking `crossbeam` receiver here, it would freeze the current-thread runtime and wouldn't work inside a `select!`. We wrap the tokio sender in `ChunkSink` (server to client) and `ChunkSubscriber` (client to server) so neither the client nor server code ever explicitly names a `tokio` type (refer to [ADR-0010](adr/0010-net-crate-async-runtime.md)).
|
|
|
|
The server runs this in `chunk_stream_task` while the client runs it in `client_chunk_task`. Each one is just a `select!` loop evaluating whether a frame arrived to read or a message is queued to write. The client bundles this into a `ClientLink` containing the handshake outcome, the `ChunkSubscriber`, and the `crossbeam` receiver for incoming deliveries.
|
|
|
|
### Per-client state and the diff
|
|
|
|
Every connected client gets a `ClientStream` tracker holding its `ChunkSink`, its current desired set (clamped to `SERVER_MAX_RADIUS`), and a record of what it has already been sent. When a subscription updates, we calculate the diff (`new - previous` for loads, `previous - new` for drops). For every chunk in the drop list that was previously sent, we emit a `ChunkMessage::Drop`.
|
|
|
|
However, we do *not* send newly desired chunks immediately. Chunk generation is async, so `ClientStream::flush` checks each tick and delivers any newly resident chunks that the client wants but hasn't received yet. If the chunk isn't ready, it simply waits and tries again on a future tick.
|
|
|
|
We bound this delivery to `MAX_DELIVERIES_PER_TICK` (currently 32 chunks per client per tick) to prevent lag spikes. Encoding chunks is expensive. If a client teleports and suddenly needs hundreds of chunks, trying to encode them all at once would immediately blow out the tick budget. Importantly, this limit only applies to chunks we *actually encode*. If most of the desired set is still generating in the worker pool, we don't penalize the tick budget for work that hasn't happened yet. This fixed chunk count is a stopgap until we implement a proper time-based budget, which will be necessary once chunk costs start varying by LOD.
|
|
|
|
### Self-contained payloads (all-air diff)
|
|
|
|
The `ChunkMessage::Chunk` payload carries a `ChunkData` in a sparse, baseline-relative format (detailed in [ADR-0009](adr/0009-baseline-relative-sparse-chunk-persistence.md)). Because the client does absolutely zero worldgen (the server is strictly authoritative), it has no way to reconstruct the worldgen baseline to diff against. Instead, we diff all delivered chunks against a completely empty all-air baseline (`Chunk::default()`). The edits effectively become the chunk's entire non-air content, and the client reconstructs the chunk by applying those edits to its own empty baseline. This keeps every delivery completely self-contained. While we sacrifice the compression benefits of diffing against the true worldgen baseline, we're deferring heavy compression work to the future LOD pass.
|
|
|
|
### Client application
|
|
|
|
Whenever the client's center chunk changes, it subscribes using its own `LOAD_RADIUS`. This ensures the server's per-client resident set perfectly matches what the client intends to keep. The client drains incoming deliveries under its own per-frame budgets. When a `ChunkMessage::Chunk` arrives, it materializes the data and queues the position (along with its six neighbors) for meshing. When a `ChunkMessage::Drop` arrives, it discards the mesh. Since meshing runs on a separate worker pool, the client safely holds onto the chunk voxels even after the mesh is uploaded (this pipeline is covered in [`meshing.md`](meshing.md)). Crucially, the client proactively evicts chunks outside its `LOAD_RADIUS` on its own. It doesn't strictly wait for the server's `Drop` message, ensuring memory usage stays strictly bounded even if the server lags behind.
|
|
|
|
## Multiplayer
|
|
|
|
Residency operates as a single, shared pipeline. We union every client's subscription cylinder into one massive desired set, which is then reconciled against a single chunk store and a single worker pool. This guarantees a chunk is only generated once, regardless of how many clients requested it.
|
|
|
|
**Delivery**, however, is strictly per-client. Each `ClientStream` independently tracks what that specific client has received and diffs against its personal subscription. When a client connects or disconnects, its `ClientStream` is simply added to or removed from the map. Features like backpressure, fairness across clients, and per-chunk flow control are deferred for now.
|
|
|
|
## Level of detail
|
|
|
|
Currently, every job processes a full-detail LOD0 chunk. When we eventually introduce LODs, the job payload will just grow from `ChunkPos` to `(ChunkPos, Lod)`. The entire worker-pool plumbing is already LOD-agnostic and won't need to change.
|
|
|
|
## Testing
|
|
|
|
We unit-test the pure cylinder math and async reconcile behavior inside `world_server.rs`:
|
|
|
|
- `cylinder_chunks` is tested for symmetry, boundary inclusion, and translation invariance.
|
|
- `reconcile_converges_over_multiple_passes` verifies that an initial pass dispatches work but leaves nothing resident, while subsequent passes drain the pool until everything is resident.
|
|
- `evicted_chunk_is_not_repopulated_on_arrival` confirms our eviction race guard works. If a chunk stops being wanted while it's in flight, it gets discarded upon arrival and never enters the resident set. This test works without any timing hacks because subsequent passes simply reconcile against an empty desired set.
|