# torrent-peer — Forward Work Plan Forward plan for evolving this from a single-peer, leech-only, TCP block fetcher into a **unified multi-peer download engine**. Derived from REVIEW.md and the architecture discussion that followed it. > **Pivot from the previous plan.** Earlier this library treated one > `peer_handle` as one connection and pushed all concurrency to the parent > ("more peers = more handles"). We are reversing that: the engine now owns > *all* peer connections itself, via an event-loop pool. This brings cross-peer > piece picking, endgame, and request-timeout handling **in scope**, and > removes the thread-per-peer model. If the real target were only a few fast > peers (seedbox-to-seedbox), the old single-peer model would have been fine; > this plan assumes a general client over real swarms (many peers, many > torrents, heavy churn) — the workload a reactor is built for. --- ## Architectural decision: unified multi-peer engine **Why restructure.** BitTorrent's connection profile is "many peers, most slow or idle, with churn" — the classic c10k reactor workload. Thread-per-peer (the current `peer_start` → one net thread per fd, `peer.c:115`) costs MBs of stack per thread × thousands, context-switch overhead, and the `RUNNING_TICK_MS` 2 ms-tick-×-N-threads waste. Every production client (libtorrent, rtorrent, transmission) multiplexes many sockets over a few threads instead. **Shape: a fixed pool of event loops, with torrents pinned to a loop (affinity).** Not one-loop-per-torrent (thread explosion + bad balance), and not a fully global pool with a torrent's peers scattered across loops (which would make the per-torrent piece picker contended). Instead: a small pool of loops (≈ the cores spent on networking), each hosting *many* torrents, but **all of a given torrent's peers live on its one assigned loop**. This yields a lock-free, single-threaded piece picker per torrent, natural load balancing across loops, and thread count decoupled from torrent count. **Escape hatch:** a single torrent hot enough to need more than one core (saturating 10GbE from one swarm) is the only case wanting its peers sharded across loops — and the only case that pays for a sharded/locked picker. Default to affinity; treat cross-loop sharding as an opt-in for the rare mega-torrent. **What's reusable (most of the existing code).** The wire parser (`proto.c`), the per-piece scheduling logic (`scheduler.c`), the arena handoff, and the SPSC rings (`ring.h`) survive as a per-*connection* state machine. The restructure is concentrated in `net.c` (the I/O driver and threading) and the public ABI. **What comes into scope because of the pivot:** - Cross-peer / swarm-wide piece picking (rarest-first across all peers of a torrent), replacing the single-peer priority scan. - Endgame mode + `CANCEL` (was REVIEW P2d "parent's job" — now ours). - Per-request in-flight tracking + timeout/re-request (REVIEW C1) and unsolicited-block validation (REVIEW C2) — now first-class engine concerns. --- ## Target object model ``` Engine process-global; owns the loop pool + worker pools ├─ Loop[0..L) one OS thread each; owns an epoll/io_uring instance, │ ├─ arena slab its set of fds, and ONE per-loop arena slab │ ├─ command queue (eventfd) cross-thread control in (add torrent, set priority…) │ └─ Connection[*] every conn on this loop is single-thread-owned ├─ Torrent[*] pinned to one Loop; owns the piece picker, │ ├─ piece picker availability counts, requested[], in-flight map — │ ├─ priority vector touched ONLY by its loop ⇒ no lock │ └─ peers ──────────────► (its Connections, all on the same loop) └─ HashPool (optional) shared worker threads for in-library verification ``` - **Loop** is the sole owner of its connections, their `reader`/`outbuf`, and its arena. Hot path (recv → parse → handoff → scheduler tick) is lock-free because one thread touches all of it. - **Torrent** state (picker, `have` counts, `requested`, in-flight) is touched only by the torrent's loop ⇒ lock-free picking. A torrent is assigned to a loop at add time (least-loaded loop), and all its peers are opened on that loop. - **Control plane** (add/remove torrent, set priorities, add peer, stop) is cross-thread: enqueue a command on the target loop's command queue and poke its eventfd — the same pattern as today's `ctrl_efd` (`peer.c:124 poke`). - **Data plane / consumer handoff:** two modes (see §7): - *Harness/zero-copy mode (default, today's behavior):* per-connection SPSC `ready_ring` + `free_ring` over the per-loop arena; the consumer drains many rings. Slot lifetime spans the consumer. - *In-library hashing mode:* the loop copies each block out of its slot into a per-piece assembly buffer and **releases the slot immediately**, so slots never leave the loop and no cross-thread free ring is needed. **ABI impact (decision needed).** The public surface moves from `peer_handle` to an engine-centric API: `engine_create/destroy`, `engine_add_torrent`, `torrent_set_priorities`, `engine_add_peer(torrent, ip, port)`, plus a completion-draining call that spans connections. The Python ctypes layer (`peer_ffi.py`) and harness change accordingly. Keep a thin single-peer convenience wrapper so existing tests/usage migrate incrementally. --- ## Guiding constraints (do not violate) - **One thread owns a connection end-to-end.** No connection state is shared across loop threads; all cross-thread interaction is via command queues. This is what keeps the hot path and the piece picker lock-free. - **Keep the zero-copy harness path working.** In-library hashing, io_uring, encryption, and µTP are additive/opt-in. The default (epoll + zero-copy arena handoff) must not regress; the existing end-to-end test must keep passing. - **Affinity by default.** A torrent's peers stay on its loop unless cross-loop sharding is explicitly enabled for that torrent. --- ## Scope table | Item | REVIEW id | Decision | |---|---|---| | Reactor / loop-pool restructure (de-thread-per-peer) | P2a (reframed) | **Keystone — implement** | | Cross-peer rarest-first picker | P2a follow-on | Implement | | Request timeout + in-flight table | C1 | Implement (engine core) | | Unsolicited-block validation | C2 | Implement (engine core) | | Endgame mode + CANCEL | P2d | **In scope now** (engine owns cross-peer policy) | | Transport abstraction (vtable) | enabler | Implement (keystone-b) | | IPv6 | P2e | Implement | | Fast Extension (BEP-6) + Extension Protocol (BEP-10) | P2e | Implement | | io_uring receive path | P2b | Implement (opt-in + epoll fallback) | | In-library SHA hashing | P2c | Implement (opt-in) | | MSE/PE encryption | P2e | Implement | | µTP | P2e | Implement | P1 throughput items from REVIEW (event-driven slot release, don't pin `SO_RCVBUF`, adaptive pipeline depth, true zero-copy recv, `outstanding` data race) are folded into the relevant sections below rather than tracked separately. --- ## §0a. Keystone: reactor restructure (the big one) Convert from thread-per-peer to a loop pool. Do this first; it reshapes everything else. **Today:** `net_thread_main` (`net.c:125`) is one thread per peer running its own `epoll_wait` + `scheduler_tick` + `flush_out`, pulling bytes via `recv()` inside `proto.c`. **Change:** 1. **Loop object:** one thread, one epoll (later io_uring) instance, a set of connections, a per-loop arena, and a command-queue eventfd. The loop body is today's `net_thread_main` body generalized to iterate over *all* ready connections per wakeup instead of one fd. 2. **Push-model parser:** change `proto.c` from *pull* (`rd_refill`/`rd_take_body` calling `recv`) to *push*: a `proto_feed(conn, buf, len)` entry point the loop calls with received bytes. This is also a prerequisite for io_uring (§6) and removes the per-body direct-`recv` syscall pattern. The parser state machine (`RS_*`) is unchanged; only its byte source changes. 3. **Per-loop arena:** one slab shared by the loop's connections; slot allocation is single-threaded ⇒ lock-free. Memory now scales with loops, not peers (kills the ×N arena blowup of per-peer arenas). 4. **Event-driven slot release (REVIEW P1a):** returned credit wakes the owning loop via its command queue (coalesced), removing the 2 ms-tick latency floor. 5. **Don't pin `SO_RCVBUF` (REVIEW P1c):** drop the fixed 8 MiB `SO_RCVBUF` (`net.c:96`) so the kernel autotunes on high-BDP links; keep it an opt-in tunable. 6. **Fix the `outstanding` data race (REVIEW P1f):** make per-connection counters that status reads touch atomic. - **Effort:** ~1–1.5 weeks. **Risk:** medium-high (core rewrite of the I/O layer). - **Tests:** existing end-to-end test passes with a 1-loop / 1-peer config; add a test with multiple peers of one torrent on one loop, and multiple torrents across loops. ## §0b. Keystone: transport abstraction Decouple the parser from the raw socket so io_uring/MSE/µTP can slot in. Small; do alongside §0a (the push-model parser is the shared piece). ```c typedef struct { ssize_t (*recv)(void *ctx, void *buf, size_t n); /* -1/EAGAIN, 0 closed */ ssize_t (*send)(void *ctx, const void *buf, size_t n); int (*want_fds)(void *ctx, int *fds, int max); /* fds the loop polls */ void (*close)(void *ctx); void *ctx; } transport; ``` Implementations: `transport_tcp` (current epoll path), later `transport_uring`, `transport_mse` (wraps an inner transport), `transport_utp`. The loop drives `t->recv`/`t->send`; encryption and µTP compose by wrapping an inner transport. - **Effort:** ~1 day on top of §0a. **Risk:** low. --- ## §1. Cross-peer piece picker + in-flight tracking + timeout (correctness core) Replaces the single-peer `select_next_piece` scan (`scheduler.c:25`) with a per-torrent picker shared across that torrent's peers (lock-free, since one loop owns the torrent). **Design:** - **Availability counts:** maintain per-piece rarity from every peer's `bitfield`/`have` (and `HAVE_ALL`/`HAVE_NONE` from §5). Picker chooses highest-priority, then rarest, then lowest index, among pieces some connected peer has and that aren't fully in-flight. - **In-flight map (REVIEW C1):** track outstanding `(piece, begin)` per request with the issuing connection and a deadline. On timeout: drop the entry, decrement that connection's `outstanding`, and re-arm the block for any peer that has it. This closes the "silent drop → permanent stall" hole. - **Unsolicited-block validation (REVIEW C2):** on an inbound piece message, look it up in the in-flight map; if absent, drop the payload **without** consuming a slot or decrementing `outstanding`. - **Block-level requests:** the picker now hands out blocks (not whole pieces), enabling the same block to be re-issued to another peer (timeout/endgame). - **Adaptive pipeline depth (REVIEW P1d):** size per-connection in-flight to the measured BDP (RTT from request→first-byte × rate from `update_rate`), clamped by arena credit. - **Effort:** ~1 week. **Risk:** medium. **Tests:** multi-peer seed where one peer stalls a block — assert it's re-requested from another and the download completes; negative test for unsolicited blocks. ## §2. Endgame mode + CANCEL Now in scope because the engine owns cross-peer policy. **Design:** when remaining blocks < threshold, allow a block to be requested from multiple peers simultaneously; when one arrives, send `CANCEL` to the others and drop their in-flight entries. Reuses the §1 in-flight map and block-level requests. `CANCEL` send path lives in `proto.c` (currently CANCEL is only recognized inbound). - **Effort:** ~2–3 days. **Risk:** low-medium (depends on §1). **Tests:** seed with one deliberately slow peer near completion; assert endgame races finish the tail and losers receive `CANCEL`. ## §3. IPv6 (small; do early) `AF_INET`-only (`net.c:90`) roughly halves reachable peers. Replace the hard-coded `sockaddr_in`/`inet_pton` with `sockaddr_storage` + `getaddrinfo`-style handling; detect a literal `:` to choose `AF_INET6`. No conceptual ABI change (peer addresses are already strings). - **Effort:** ~0.5 day. **Risk:** low. **Tests:** local seed bound to `[::1]`. ## §4. Fast Extension (BEP-6) + Extension Protocol (BEP-10) **Why:** **Reject Request** turns §1's timeout-based recovery into *instant* recovery — a peer that won't serve a block says so explicitly. `HAVE_ALL`/ `HAVE_NONE` populate availability in one message. BEP-10 is the negotiation base real peers expect (and the prerequisite for PEX/metadata later). **Design:** - Handshake reserved bits: Fast-Ext (`reserved[7] |= 0x04`), Ext-Protocol (`reserved[5] |= 0x10`); use a feature only if the peer also set its bit. - Inbound in `handle_control`: `HAVE_ALL (0x0E)`/`HAVE_NONE (0x0F)` → availability; `REJECT_REQUEST (0x10)` → look up in the §1 in-flight map, decrement, re-arm immediately; `SUGGEST_PIECE`/`ALLOWED_FAST` → optional selection bias (Allowed-Fast lets us request while choked). - Outbound: send `HAVE_NONE` on connect when both support Fast; add a minimal bencode reader for the BEP-10 extended handshake. - **Effort:** ~1.5 days (incl. minimal bencode). **Risk:** low-medium. **Tests:** mock peer advertises Fast, sends `HAVE_ALL` then `REJECT_REQUEST`; assert instant re-request (faster than the timeout) and correct completion. ## §5. io_uring receive path (REVIEW P2b) **Why:** near-syscall-free, genuine zero-copy receive at multi-GB/s. **Reasons it's not first:** kernel-version gating (multishot recv + provided buffers ≈ 5.19+), seccomp/sandbox policies that disable io_uring, and that it's a throughput-only win whose payoff appears only after the other bottlenecks are gone. So: opt-in, with the epoll path as permanent fallback. **Design:** a `transport_uring` selected per loop at runtime (probe at startup; fall back to `transport_tcp`). Multishot `IORING_OP_RECV` with a provided-buffer ring feeds the §0a push-model parser. Stretch: register the per-loop arena with `io_uring_register_buffers` and use fixed-buffer reads into the destination slot, removing the residual `memcpy` (REVIEW P1b). Batch request writes via `IORING_OP_SEND`. - **Dependency:** `liburing`, optional + feature-gated. **Effort:** ~3–4 days. **Risk:** medium (kernel variance). **Tests:** full suite against both backends via a config knob; CI selects by kernel capability. ## §6. In-library SHA hashing (REVIEW P2c) **Why:** with credit backpressure, **wire speed is hostage to hash speed**, and hashing currently runs in Python. Make it an opt-in in-library mode (the harness path stays default). **Design:** - **Engine-level `HashPool`** of N worker threads (shared across loops). - **Config/API:** `verify_in_library` flag, `worker_threads`, `torrent_set_piece_hashes(...)`, and a results channel (`{piece, ok}` ring the consumer drains; verified bytes via in-lib piece buffer / write-fd / callback). - **Flow:** the loop copies each block from its slot into a pooled per-piece assembly buffer and **releases the slot immediately** (wire stays full; slots never leave the loop). When a piece is complete, enqueue a hash job; a worker runs SHA-1 and reports the result. - **SHA-1 with SHA-NI:** runtime-dispatched (`__builtin_cpu_supports("sha")`) with a portable fallback. Reused by MSE key derivation (§7). - **Effort:** ~3 days. **Risk:** medium (new threading/memory subsystem). **Tests:** enable in-lib verification against a local seed; assert all results `ok` and bytes match; negative test with a wrong expected hash (retry path). ## §7. MSE / PE encryption **Why:** many real-swarm peers refuse plaintext. A `transport_mse` wrapping an inner transport (§0b) performs the MSE handshake as initiator before BT bytes flow: DH key exchange (768-bit MSE prime) → obfuscated handshake (`HASH('req1',S)`, `HASH('req2',SKEY) xor HASH('req3',S)`, `SKEY=info_hash`, random pad) → cipher negotiation (`crypto_provide`/`select`, RC4 or plaintext, keys from `HASH('keyA'/'keyB',S,SKEY)`, RC4-drop-1024) → transparent encrypt/decrypt so `proto.c` is unchanged. Primitives: RC4 (~15 lines), SHA-1 (reuse §6), bignum modexp (vendored small bigint or `libcrypto` BN — decision below). Config: `enc_policy` (plaintext-only / prefer / require). - **Effort:** ~4–5 days. **Risk:** high (security-sensitive). **Tests:** libtorrent seed with encryption forced; assert completion; cross-check RC4 and plaintext selection; run under ASan. ## §8. µTP (largest item) **Why:** a large fraction of real peers are µTP-only (BitTorrent over UDP, LEDBAT). A `transport_utp` presents a reliable in-order byte stream to the parser over a UDP socket: packet types `ST_SYN/DATA/STATE/FIN/RESET` with `seq_nr`/`ack_nr`/timestamps + selective-ack; LEDBAT congestion control (one-way delay target ~100 ms, AIMD-style window — the core complexity); retransmission timers, send/recv windows, in-order reassembly; UDP fd registered via the transport's `want_fds`, timers driven off the loop tick; connect as initiator. - **Effort:** ~1–2 weeks. **Risk:** high (timing/congestion correctness). **Tests:** libtorrent seed forced to µTP-only over `127.0.0.1`; verify bytes and LEDBAT back-off under induced delay. --- ## Suggested ordering & milestones 1. **§0a reactor restructure + §0b transport abstraction** (keystones; share the push-model parser). Land with multi-peer/multi-torrent tests green. 2. **§1 picker + in-flight + timeout** and **§2 endgame/CANCEL** — the core that makes multi-peer actually correct and fast. 3. **§3 IPv6** + **§4 Fast Extension/BEP-10** — quick wins; Fast-Ext upgrades §1 recovery from timeout to instant. 4. **§6 in-library hashing** — removes the Python hash ceiling (independent). 5. **§5 io_uring** — throughput, on the transport abstraction. 6. **§7 MSE** — reach; reuses SHA-1 from §6. 7. **§8 µTP** — biggest; reach; last. §5/§6 are independent and can be parallelized; §7 and §8 both sit on §0b. ## Decisions needed before starting - **Loop count / sizing:** fixed (e.g. `min(ncpu, N)`) vs. configurable; pinning loop threads to cores? Default torrent→loop assignment policy (least-loaded)? - **ABI shape:** engine-centric API + thin single-peer compat wrapper — confirm the new surface and how the Python harness/consumer drains completions across many connections. - **Cross-loop sharding for mega-torrents:** ship affinity-only first and add sharded picker later, or design the picker for optional sharding up front? - **Crypto dependency (§7):** vendor minimal bigint + RC4 + SHA-1, or link `libcrypto` (build dep + RC4 legacy-provider wrinkle)? - **liburing (§5):** optional + runtime probe + epoll fallback (recommended). - **In-library hashing output (§6):** results-ring + in-lib piece buffers only, or also a write-to-fd path so the C side can persist without the harness? - **Minimum kernel/libc baseline** for io_uring and µTP timers.