Naut-Peer/PLAN.md

torrent-peer — Forward Work Plan

Forward plan for evolving this from a single-peer, leech-only, TCP block fetcher into a unified multi-peer download engine. Derived from REVIEW.md and the architecture discussion that followed it.

Pivot from the previous plan. Earlier this library treated one peer_handle as one connection and pushed all concurrency to the parent ("more peers = more handles"). We are reversing that: the engine now owns all peer connections itself, via an event-loop pool. This brings cross-peer piece picking, endgame, and request-timeout handling in scope, and removes the thread-per-peer model. If the real target were only a few fast peers (seedbox-to-seedbox), the old single-peer model would have been fine; this plan assumes a general client over real swarms (many peers, many torrents, heavy churn) — the workload a reactor is built for.

---

Architectural decision: unified multi-peer engine

Why restructure. BitTorrent's connection profile is "many peers, most slow or idle, with churn" — the classic c10k reactor workload. Thread-per-peer (the current peer_start → one net thread per fd, peer.c:115) costs MBs of stack per thread × thousands, context-switch overhead, and the RUNNING_TICK_MS 2 ms-tick-×-N-threads waste. Every production client (libtorrent, rtorrent, transmission) multiplexes many sockets over a few threads instead.

Shape: a fixed pool of event loops, with torrents pinned to a loop (affinity). Not one-loop-per-torrent (thread explosion + bad balance), and not a fully global pool with a torrent's peers scattered across loops (which would make the per-torrent piece picker contended). Instead: a small pool of loops (≈ the cores spent on networking), each hosting many torrents, but all of a given torrent's peers live on its one assigned loop. This yields a lock-free, single-threaded piece picker per torrent, natural load balancing across loops, and thread count decoupled from torrent count.

Escape hatch: a single torrent hot enough to need more than one core (saturating 10GbE from one swarm) is the only case wanting its peers sharded across loops — and the only case that pays for a sharded/locked picker. Default to affinity; treat cross-loop sharding as an opt-in for the rare mega-torrent.

What's reusable (most of the existing code). The wire parser (proto.c), the per-piece scheduling logic (scheduler.c), the arena handoff, and the SPSC rings (ring.h) survive as a per-connection state machine. The restructure is concentrated in net.c (the I/O driver and threading) and the public ABI.

What comes into scope because of the pivot:

• Cross-peer / swarm-wide piece picking (rarest-first across all peers of a torrent), replacing the single-peer priority scan.
• Endgame mode + CANCEL (was REVIEW P2d "parent's job" — now ours).
• Per-request in-flight tracking + timeout/re-request (REVIEW C1) and unsolicited-block validation (REVIEW C2) — now first-class engine concerns.

---

Target object model

Engine                          process-global; owns the loop pool + worker pools
 ├─ Loop[0..L)                   one OS thread each; owns an epoll/io_uring instance,
 │    ├─ arena slab              its set of fds, and ONE per-loop arena slab
 │    ├─ command queue (eventfd) cross-thread control in (add torrent, set priority…)
 │    └─ Connection[*]           every conn on this loop is single-thread-owned
 ├─ Torrent[*]                   pinned to one Loop; owns the piece picker,
 │    ├─ piece picker            availability counts, requested[], in-flight map —
 │    ├─ priority vector         touched ONLY by its loop ⇒ no lock
 │    └─ peers ──────────────►   (its Connections, all on the same loop)
 └─ HashPool (optional)          shared worker threads for in-library verification

• Loop is the sole owner of its connections, their reader/outbuf, and its arena. Hot path (recv → parse → handoff → scheduler tick) is lock-free because one thread touches all of it.
• Torrent state (picker, have counts, requested, in-flight) is touched only by the torrent's loop ⇒ lock-free picking. A torrent is assigned to a loop at add time (least-loaded loop), and all its peers are opened on that loop.
• Control plane (add/remove torrent, set priorities, add peer, stop) is cross-thread: enqueue a command on the target loop's command queue and poke its eventfd — the same pattern as today's ctrl_efd (peer.c:124 poke).
• Data plane / consumer handoff: two modes (see §7):
  • Harness/zero-copy mode (default, today's behavior): per-connection SPSC ready_ring + free_ring over the per-loop arena; the consumer drains many rings. Slot lifetime spans the consumer.
  • In-library hashing mode: the loop copies each block out of its slot into a per-piece assembly buffer and releases the slot immediately, so slots never leave the loop and no cross-thread free ring is needed.

ABI impact (decision needed). The public surface moves from peer_handle to an engine-centric API: engine_create/destroy, engine_add_torrent, torrent_set_priorities, engine_add_peer(torrent, ip, port), plus a completion-draining call that spans connections. The Python ctypes layer (peer_ffi.py) and harness change accordingly. Keep a thin single-peer convenience wrapper so existing tests/usage migrate incrementally.

---

Guiding constraints (do not violate)

• One thread owns a connection end-to-end. No connection state is shared across loop threads; all cross-thread interaction is via command queues. This is what keeps the hot path and the piece picker lock-free.
• Keep the zero-copy harness path working. In-library hashing, io_uring, encryption, and µTP are additive/opt-in. The default (epoll + zero-copy arena handoff) must not regress; the existing end-to-end test must keep passing.
• Affinity by default. A torrent's peers stay on its loop unless cross-loop sharding is explicitly enabled for that torrent.

---

Scope table

Item	REVIEW id	Decision
Reactor / loop-pool restructure (de-thread-per-peer)	P2a (reframed)	Keystone — implement
Cross-peer rarest-first picker	P2a follow-on	Implement
Request timeout + in-flight table	C1	Implement (engine core)
Unsolicited-block validation	C2	Implement (engine core)
Endgame mode + CANCEL	P2d	In scope now (engine owns cross-peer policy)
Transport abstraction (vtable)	enabler	Implement (keystone-b)
IPv6	P2e	Implement
Fast Extension (BEP-6) + Extension Protocol (BEP-10)	P2e	Implement
io_uring receive path	P2b	Implement (opt-in + epoll fallback)
In-library SHA hashing	P2c	Implement (opt-in)
MSE/PE encryption	P2e	Implement
µTP	P2e	Implement

P1 throughput items from REVIEW (event-driven slot release, don't pin SO_RCVBUF, adaptive pipeline depth, true zero-copy recv, outstanding data race) are folded into the relevant sections below rather than tracked separately.

---

§0a. Keystone: reactor restructure (the big one)

Convert from thread-per-peer to a loop pool. Do this first; it reshapes everything else.

Today: net_thread_main (net.c:125) is one thread per peer running its own epoll_wait + scheduler_tick + flush_out, pulling bytes via recv() inside proto.c.

Change:

1. Loop object: one thread, one epoll (later io_uring) instance, a set of connections, a per-loop arena, and a command-queue eventfd. The loop body is today's net_thread_main body generalized to iterate over all ready connections per wakeup instead of one fd.
2. Push-model parser: change proto.c from pull (rd_refill/rd_take_body calling recv) to push: a proto_feed(conn, buf, len) entry point the loop calls with received bytes. This is also a prerequisite for io_uring (§6) and removes the per-body direct-recv syscall pattern. The parser state machine (RS_*) is unchanged; only its byte source changes.
3. Per-loop arena: one slab shared by the loop's connections; slot allocation is single-threaded ⇒ lock-free. Memory now scales with loops, not peers (kills the ×N arena blowup of per-peer arenas).
4. Event-driven slot release (REVIEW P1a): returned credit wakes the owning loop via its command queue (coalesced), removing the 2 ms-tick latency floor.
5. Don't pin SO_RCVBUF (REVIEW P1c): drop the fixed 8 MiB SO_RCVBUF (net.c:96) so the kernel autotunes on high-BDP links; keep it an opt-in tunable.
6. Fix the outstanding data race (REVIEW P1f): make per-connection counters that status reads touch atomic.

• Effort: ~1–1.5 weeks. Risk: medium-high (core rewrite of the I/O layer).
• Tests: existing end-to-end test passes with a 1-loop / 1-peer config; add a test with multiple peers of one torrent on one loop, and multiple torrents across loops.

§0b. Keystone: transport abstraction

Decouple the parser from the raw socket so io_uring/MSE/µTP can slot in. Small; do alongside §0a (the push-model parser is the shared piece).

typedef struct {
    ssize_t (*recv)(void *ctx, void *buf, size_t n);   /* -1/EAGAIN, 0 closed */
    ssize_t (*send)(void *ctx, const void *buf, size_t n);
    int     (*want_fds)(void *ctx, int *fds, int max); /* fds the loop polls  */
    void    (*close)(void *ctx);
    void     *ctx;
} transport;

Implementations: transport_tcp (current epoll path), later transport_uring, transport_mse (wraps an inner transport), transport_utp. The loop drives t->recv/t->send; encryption and µTP compose by wrapping an inner transport.

• Effort: ~1 day on top of §0a. Risk: low.

---

§1. Cross-peer piece picker + in-flight tracking + timeout (correctness core)

Replaces the single-peer select_next_piece scan (scheduler.c:25) with a per-torrent picker shared across that torrent's peers (lock-free, since one loop owns the torrent).

Design:

• Availability counts: maintain per-piece rarity from every peer's bitfield/have (and HAVE_ALL/HAVE_NONE from §5). Picker chooses highest-priority, then rarest, then lowest index, among pieces some connected peer has and that aren't fully in-flight.

• In-flight map (REVIEW C1): track outstanding (piece, begin) per request with the issuing connection and a deadline. On timeout: drop the entry, decrement that connection's outstanding, and re-arm the block for any peer that has it. This closes the "silent drop → permanent stall" hole.

• Unsolicited-block validation (REVIEW C2): on an inbound piece message, look it up in the in-flight map; if absent, drop the payload without consuming a slot or decrementing outstanding.

• Block-level requests: the picker now hands out blocks (not whole pieces), enabling the same block to be re-issued to another peer (timeout/endgame).

• Adaptive pipeline depth (REVIEW P1d): size per-connection in-flight to the measured BDP (RTT from request→first-byte × rate from update_rate), clamped by arena credit.

• Effort: ~1 week. Risk: medium. Tests: multi-peer seed where one peer stalls a block — assert it's re-requested from another and the download completes; negative test for unsolicited blocks.

§2. Endgame mode + CANCEL

Now in scope because the engine owns cross-peer policy.

Design: when remaining blocks < threshold, allow a block to be requested from multiple peers simultaneously; when one arrives, send CANCEL to the others and drop their in-flight entries. Reuses the §1 in-flight map and block-level requests. CANCEL send path lives in proto.c (currently CANCEL is only recognized inbound).

• Effort: ~2–3 days. Risk: low-medium (depends on §1). Tests: seed with one deliberately slow peer near completion; assert endgame races finish the tail and losers receive CANCEL.

§3. IPv6 (small; do early)

AF_INET-only (net.c:90) roughly halves reachable peers. Replace the hard-coded sockaddr_in/inet_pton with sockaddr_storage + getaddrinfo-style handling; detect a literal : to choose AF_INET6. No conceptual ABI change (peer addresses are already strings).

• Effort: ~0.5 day. Risk: low. Tests: local seed bound to [::1].

§4. Fast Extension (BEP-6) + Extension Protocol (BEP-10)

Why: Reject Request turns §1's timeout-based recovery into instant recovery — a peer that won't serve a block says so explicitly. HAVE_ALL/ HAVE_NONE populate availability in one message. BEP-10 is the negotiation base real peers expect (and the prerequisite for PEX/metadata later).

Design:

• Handshake reserved bits: Fast-Ext (reserved[7] |= 0x04), Ext-Protocol (reserved[5] |= 0x10); use a feature only if the peer also set its bit.

• Inbound in handle_control: HAVE_ALL (0x0E)/HAVE_NONE (0x0F) → availability; REJECT_REQUEST (0x10) → look up in the §1 in-flight map, decrement, re-arm immediately; SUGGEST_PIECE/ALLOWED_FAST → optional selection bias (Allowed-Fast lets us request while choked).

• Outbound: send HAVE_NONE on connect when both support Fast; add a minimal bencode reader for the BEP-10 extended handshake.

• Effort: ~1.5 days (incl. minimal bencode). Risk: low-medium. Tests: mock peer advertises Fast, sends HAVE_ALL then REJECT_REQUEST; assert instant re-request (faster than the timeout) and correct completion.

§5. io_uring receive path (REVIEW P2b)

Why: near-syscall-free, genuine zero-copy receive at multi-GB/s. Reasons it's not first: kernel-version gating (multishot recv + provided buffers ≈ 5.19+), seccomp/sandbox policies that disable io_uring, and that it's a throughput-only win whose payoff appears only after the other bottlenecks are gone. So: opt-in, with the epoll path as permanent fallback.

Design: a transport_uring selected per loop at runtime (probe at startup; fall back to transport_tcp). Multishot IORING_OP_RECV with a provided-buffer ring feeds the §0a push-model parser. Stretch: register the per-loop arena with io_uring_register_buffers and use fixed-buffer reads into the destination slot, removing the residual memcpy (REVIEW P1b). Batch request writes via IORING_OP_SEND.

• Dependency: liburing, optional + feature-gated. Effort: ~3–4 days. Risk: medium (kernel variance). Tests: full suite against both backends via a config knob; CI selects by kernel capability.

§6. In-library SHA hashing (REVIEW P2c)

Why: with credit backpressure, wire speed is hostage to hash speed, and hashing currently runs in Python. Make it an opt-in in-library mode (the harness path stays default).

Design:

• Engine-level HashPool of N worker threads (shared across loops).

• Config/API: verify_in_library flag, worker_threads, torrent_set_piece_hashes(...), and a results channel ({piece, ok} ring the consumer drains; verified bytes via in-lib piece buffer / write-fd / callback).

• Flow: the loop copies each block from its slot into a pooled per-piece assembly buffer and releases the slot immediately (wire stays full; slots never leave the loop). When a piece is complete, enqueue a hash job; a worker runs SHA-1 and reports the result.

• SHA-1 with SHA-NI: runtime-dispatched (__builtin_cpu_supports("sha")) with a portable fallback. Reused by MSE key derivation (§7).

• Effort: ~3 days. Risk: medium (new threading/memory subsystem). Tests: enable in-lib verification against a local seed; assert all results ok and bytes match; negative test with a wrong expected hash (retry path).

§7. MSE / PE encryption

Why: many real-swarm peers refuse plaintext. A transport_mse wrapping an inner transport (§0b) performs the MSE handshake as initiator before BT bytes flow: DH key exchange (768-bit MSE prime) → obfuscated handshake (HASH('req1',S), HASH('req2',SKEY) xor HASH('req3',S), SKEY=info_hash, random pad) → cipher negotiation (crypto_provide/select, RC4 or plaintext, keys from HASH('keyA'/'keyB',S,SKEY), RC4-drop-1024) → transparent encrypt/decrypt so proto.c is unchanged. Primitives: RC4 (~15 lines), SHA-1 (reuse §6), bignum modexp (vendored small bigint or libcrypto BN — decision below). Config: enc_policy (plaintext-only / prefer / require).

• Effort: ~4–5 days. Risk: high (security-sensitive). Tests: libtorrent seed with encryption forced; assert completion; cross-check RC4 and plaintext selection; run under ASan.

§8. µTP (largest item)

Why: a large fraction of real peers are µTP-only (BitTorrent over UDP, LEDBAT). A transport_utp presents a reliable in-order byte stream to the parser over a UDP socket: packet types ST_SYN/DATA/STATE/FIN/RESET with seq_nr/ack_nr/timestamps + selective-ack; LEDBAT congestion control (one-way delay target ~100 ms, AIMD-style window — the core complexity); retransmission timers, send/recv windows, in-order reassembly; UDP fd registered via the transport's want_fds, timers driven off the loop tick; connect as initiator.

• Effort: ~1–2 weeks. Risk: high (timing/congestion correctness). Tests: libtorrent seed forced to µTP-only over 127.0.0.1; verify bytes and LEDBAT back-off under induced delay.

---

Suggested ordering & milestones

1. §0a reactor restructure + §0b transport abstraction (keystones; share the push-model parser). Land with multi-peer/multi-torrent tests green.
2. §1 picker + in-flight + timeout and §2 endgame/CANCEL — the core that makes multi-peer actually correct and fast.
3. §3 IPv6 + §4 Fast Extension/BEP-10 — quick wins; Fast-Ext upgrades §1 recovery from timeout to instant.
4. §6 in-library hashing — removes the Python hash ceiling (independent).
5. §5 io_uring — throughput, on the transport abstraction.
6. §7 MSE — reach; reuses SHA-1 from §6.
7. §8 µTP — biggest; reach; last.

§5/§6 are independent and can be parallelized; §7 and §8 both sit on §0b.

Decisions needed before starting

• Loop count / sizing: fixed (e.g. min(ncpu, N)) vs. configurable; pinning loop threads to cores? Default torrent→loop assignment policy (least-loaded)?
• ABI shape: engine-centric API + thin single-peer compat wrapper — confirm the new surface and how the Python harness/consumer drains completions across many connections.
• Cross-loop sharding for mega-torrents: ship affinity-only first and add sharded picker later, or design the picker for optional sharding up front?
• Crypto dependency (§7): vendor minimal bigint + RC4 + SHA-1, or link libcrypto (build dep + RC4 legacy-provider wrinkle)?
• liburing (§5): optional + runtime probe + epoll fallback (recommended).
• In-library hashing output (§6): results-ring + in-lib piece buffers only, or also a write-to-fd path so the C side can persist without the harness?
• Minimum kernel/libc baseline for io_uring and µTP timers.