Chunk streaming

How the server keeps the set of resident chunks in sync with where players are, and how chunk generation is kept off the simulation tick. The generating side lives in crates/shared/src/generator.rs; the streaming and storage side in crates/server/src/world_server.rs, driven from crates/server/src/main.rs.

Overview

Chunk residency is reconciled every tick against a desired set: the union of a cylinder of chunks around every player anchor. Chunks inside the desired set are made resident; chunks outside it are evicted. Generation of a missing chunk is expensive, so it is performed on a dedicated worker-thread pool rather than inline on the tick. Reconciliation is therefore non-blocking: each pass drains whatever chunks the pool has finished, evicts what is no longer wanted, and dispatches what is still missing, without ever waiting for a chunk to be generated.

Desired set

cylinder_chunks(center, radius, out) inserts every chunk position within the streaming cylinder around center into out. The shape is a disc in XZ (dx² + dz² ≤ radius²) extruded vertically to ±radius/2, reflecting the fact that horizontal view distance exceeds vertical.

The ECS streaming system stream_chunks (in main.rs) queries every entity carrying Player, Position, and ViewDistance, and unions each anchor's cylinder into one HashSet<ChunkPos>. Because the sets are unioned, overlapping cylinders deduplicate automatically and a chunk is evicted only when no player wants it. See Multiplayer below.

The worker pool

ServerWorld::new builds the pool once. Its structure:

Generator sharing: the VoxelGenerator is wrapped in an Arc and a handle is cloned into every worker. generate_chunk(&self) is read-only, so no Mutex is required; the workers share one immutable generator.
Job channel (main → workers): a crossbeam-channel carrying ChunkPos. crossbeam-channel is used rather than std::sync::mpsc because the pool needs multiple consumers: every worker clones the Receiver and pulls from the one shared queue, and each job is delivered to exactly one worker. std::sync::mpsc permits only a single consumer.
Result channel (workers → main): a crossbeam-channel carrying (ChunkPos, Chunk). Each worker clones the Sender; the main thread holds the single Receiver.
Worker loop: each worker blocks on job_rx.recv(), generates the chunk, and sends (pos, chunk) back. A blocking recv on a worker thread is acceptable because it is not the simulation thread.

Channel disconnection and shutdown

A crossbeam-channel reports disconnection (its recv returns Err) only once all senders — or, symmetrically, all receivers — have been dropped. After the spawn loop, the template job_rx and result_tx that were cloned from are dropped immediately. Retaining either would keep its channel open forever: workers would never observe job-channel shutdown, and the main thread would never observe the result channel closing. Worker JoinHandles are retained on ServerWorld for a future graceful-stop path that drops job_tx and joins the threads; the process currently relies on OS teardown at exit.

Reconcile: drain → evict → dispatch

ServerWorld::reconcile(&mut self, desired) runs three non-blocking phases per pass:

Drain. result_rx.try_recv() is pulled in a loop until empty (try_recv never blocks). Each returned position is removed from in_flight. A returned chunk is inserted into the resident map only if it is still in desired — see the eviction race.
Evict. Resident chunks absent from desired are removed. In-flight chunks that are no longer wanted need no handling here; the drain guard discards them when they arrive.
Dispatch. For every position in desired that is neither resident nor already in in_flight, the position is inserted into in_flight and sent on job_tx. The in_flight set is what prevents the same position being re-queued on every pass while a worker is still generating it.

in_flight therefore tracks positions dispatched but not yet returned, and is the single source of truth for "work outstanding."

The eviction race

Between a chunk being dispatched and the worker returning it, the anchor may move so that the chunk is no longer wanted. Without a guard, the drain phase would insert the now-unwanted chunk, resurrecting a chunk that the evict phase had already discarded (or would never be asked to discard, since it was never resident). The guard in phase 1 — insert only if desired.contains(&pos) — makes a late arrival harmless: an unwanted chunk is dropped on arrival rather than made resident.

Startup loading gate

Startup reuses the same worker pool and schedule; there is no separate synchronous loading path. Before granting player control, main runs the streaming schedule in a loop and polls ServerWorld::streaming_idle() (true when in_flight is empty). Once the initial region has at least one resident chunk and no work in flight, the region is ready. Waiting here is acceptable because no gameplay is running yet. During play the same reconcile runs every tick but is never waited on. A loading progress fraction is available as resident / (resident + in_flight).

Multiplayer

No per-player streaming pipeline exists. Every player anchor's cylinder is unioned into one desired set, reconciled against one chunk store served by one worker pool. A player joining or leaving is simply an entity entering or leaving the ECS query; it requires no streaming-specific code. The only per-player concern is the loading gate, which for a joining player checks that player's cylinder against the resident set rather than the global set. Backpressure and fairness across players (a bounded job channel, nearest-first priority) are shared-pipeline concerns deferred for later.

Level of detail

Each job is currently a full-detail (LOD0) chunk. When LOD is introduced, the job payload grows from ChunkPos to (ChunkPos, Lod); the worker-pool plumbing is LOD-agnostic and does not change.

Testing

The pure cylinder math and the async reconcile behaviour are unit-tested in world_server.rs:

cylinder_chunks symmetry, boundary inclusion, and translation invariance.
reconcile_converges_over_multiple_passes: an initial pass dispatches work and leaves nothing resident; repeated passes drain the pool until every desired position is resident.
evicted_chunk_is_not_repopulated_on_arrival: a dispatched chunk that stops being wanted is discarded on arrival and never becomes resident. This test is timing-independent because every pass after dispatch reconciles against an empty desired set.

6.8 KiB Raw Blame History