Naut-Peer/PLAN.md

# torrent-peer — Forward Work Plan

Forward plan for evolving this from a single-peer, leech-only, TCP block fetcher
into a **unified multi-peer download engine**. Derived from REVIEW.md and the
architecture discussion that followed it.

> **Pivot from the previous plan.** Earlier this library treated one
> `peer_handle` as one connection and pushed all concurrency to the parent
> ("more peers = more handles"). We are reversing that: the engine now owns
> *all* peer connections itself, via an event-loop pool. This brings cross-peer
> piece picking, endgame, and request-timeout handling **in scope**, and
> removes the thread-per-peer model. If the real target were only a few fast
> peers (seedbox-to-seedbox), the old single-peer model would have been fine;
> this plan assumes a general client over real swarms (many peers, many
> torrents, heavy churn) — the workload a reactor is built for.

---

## Architectural decision: unified multi-peer engine

**Why restructure.** BitTorrent's connection profile is "many peers, most slow
or idle, with churn" — the classic c10k reactor workload. Thread-per-peer (the
current `peer_start` → one net thread per fd, `peer.c:115`) costs MBs of stack
per thread × thousands, context-switch overhead, and the `RUNNING_TICK_MS`
2 ms-tick-×-N-threads waste. Every production client (libtorrent, rtorrent,
transmission) multiplexes many sockets over a few threads instead.

**Shape: a fixed pool of event loops, with torrents pinned to a loop
(affinity).** Not one-loop-per-torrent (thread explosion + bad balance), and not
a fully global pool with a torrent's peers scattered across loops (which would
make the per-torrent piece picker contended). Instead: a small pool of loops
(≈ the cores spent on networking), each hosting *many* torrents, but **all of a
given torrent's peers live on its one assigned loop**. This yields a lock-free,
single-threaded piece picker per torrent, natural load balancing across loops,
and thread count decoupled from torrent count.

**Escape hatch:** a single torrent hot enough to need more than one core
(saturating 10GbE from one swarm) is the only case wanting its peers sharded
across loops — and the only case that pays for a sharded/locked picker. Default
to affinity; treat cross-loop sharding as an opt-in for the rare mega-torrent.

**What's reusable (most of the existing code).** The wire parser (`proto.c`),
the per-piece scheduling logic (`scheduler.c`), the arena handoff, and the SPSC
rings (`ring.h`) survive as a per-*connection* state machine. The restructure is
concentrated in `net.c` (the I/O driver and threading) and the public ABI.

**What comes into scope because of the pivot:**
- Cross-peer / swarm-wide piece picking (rarest-first across all peers of a
  torrent), replacing the single-peer priority scan.
- Endgame mode + `CANCEL` (was REVIEW P2d "parent's job" — now ours).
- Per-request in-flight tracking + timeout/re-request (REVIEW C1) and
  unsolicited-block validation (REVIEW C2) — now first-class engine concerns.

---

## Target object model

```
Engine                          process-global; owns the loop pool + worker pools
 ├─ Loop[0..L)                   one OS thread each; owns an epoll/io_uring instance,
 │    ├─ arena slab              its set of fds, and ONE per-loop arena slab
 │    ├─ command queue (eventfd) cross-thread control in (add torrent, set priority…)
 │    └─ Connection[*]           every conn on this loop is single-thread-owned
 ├─ Torrent[*]                   pinned to one Loop; owns the piece picker,
 │    ├─ piece picker            availability counts, requested[], in-flight map —
 │    ├─ priority vector         touched ONLY by its loop ⇒ no lock
 │    └─ peers ──────────────►   (its Connections, all on the same loop)
 └─ HashPool (optional)          shared worker threads for in-library verification
```

- **Loop** is the sole owner of its connections, their `reader`/`outbuf`, and its
  arena. Hot path (recv → parse → handoff → scheduler tick) is lock-free because
  one thread touches all of it.
- **Torrent** state (picker, `have` counts, `requested`, in-flight) is touched
  only by the torrent's loop ⇒ lock-free picking. A torrent is assigned to a
  loop at add time (least-loaded loop), and all its peers are opened on that loop.
- **Control plane** (add/remove torrent, set priorities, add peer, stop) is
  cross-thread: enqueue a command on the target loop's command queue and poke its
  eventfd — the same pattern as today's `ctrl_efd` (`peer.c:124 poke`).
- **Data plane / consumer handoff:** two modes (see §7):
  - *Harness/zero-copy mode (default, today's behavior):* per-connection SPSC
    `ready_ring` + `free_ring` over the per-loop arena; the consumer drains many
    rings. Slot lifetime spans the consumer.
  - *In-library hashing mode:* the loop copies each block out of its slot into a
    per-piece assembly buffer and **releases the slot immediately**, so slots
    never leave the loop and no cross-thread free ring is needed.

**ABI impact (decision needed).** The public surface moves from `peer_handle`
to an engine-centric API: `engine_create/destroy`, `engine_add_torrent`,
`torrent_set_priorities`, `engine_add_peer(torrent, ip, port)`, plus a
completion-draining call that spans connections. The Python ctypes layer
(`peer_ffi.py`) and harness change accordingly. Keep a thin single-peer
convenience wrapper so existing tests/usage migrate incrementally.

---

## Guiding constraints (do not violate)

- **One thread owns a connection end-to-end.** No connection state is shared
  across loop threads; all cross-thread interaction is via command queues. This
  is what keeps the hot path and the piece picker lock-free.
- **Keep the zero-copy harness path working.** In-library hashing, io_uring,
  encryption, and µTP are additive/opt-in. The default (epoll + zero-copy arena
  handoff) must not regress; the existing end-to-end test must keep passing.
- **Affinity by default.** A torrent's peers stay on its loop unless cross-loop
  sharding is explicitly enabled for that torrent.

---

## Scope table

| Item | REVIEW id | Decision |
|---|---|---|
| Reactor / loop-pool restructure (de-thread-per-peer) | P2a (reframed) | **Keystone — implement** |
| Cross-peer rarest-first picker | P2a follow-on | Implement |
| Request timeout + in-flight table | C1 | Implement (engine core) |
| Unsolicited-block validation | C2 | Implement (engine core) |
| Endgame mode + CANCEL | P2d | **In scope now** (engine owns cross-peer policy) |
| Transport abstraction (vtable) | enabler | Implement (keystone-b) |
| IPv6 | P2e | Implement |
| Fast Extension (BEP-6) + Extension Protocol (BEP-10) | P2e | Implement |
| io_uring receive path | P2b | Implement (opt-in + epoll fallback) |
| In-library SHA hashing | P2c | Implement (opt-in) |
| MSE/PE encryption | P2e | Implement |
| µTP | P2e | Implement |

P1 throughput items from REVIEW (event-driven slot release, don't pin
`SO_RCVBUF`, adaptive pipeline depth, true zero-copy recv, `outstanding` data
race) are folded into the relevant sections below rather than tracked
separately.

---

## §0a. Keystone: reactor restructure (the big one)

Convert from thread-per-peer to a loop pool. Do this first; it reshapes
everything else.

**Today:** `net_thread_main` (`net.c:125`) is one thread per peer running its own
`epoll_wait` + `scheduler_tick` + `flush_out`, pulling bytes via `recv()` inside
`proto.c`.

**Change:**
1. **Loop object:** one thread, one epoll (later io_uring) instance, a set of
   connections, a per-loop arena, and a command-queue eventfd. The loop body is
   today's `net_thread_main` body generalized to iterate over *all* ready
   connections per wakeup instead of one fd.
2. **Push-model parser:** change `proto.c` from *pull* (`rd_refill`/`rd_take_body`
   calling `recv`) to *push*: a `proto_feed(conn, buf, len)` entry point the loop
   calls with received bytes. This is also a prerequisite for io_uring (§6) and
   removes the per-body direct-`recv` syscall pattern. The parser state machine
   (`RS_*`) is unchanged; only its byte source changes.
3. **Per-loop arena:** one slab shared by the loop's connections; slot allocation
   is single-threaded ⇒ lock-free. Memory now scales with loops, not peers
   (kills the ×N arena blowup of per-peer arenas).
4. **Event-driven slot release (REVIEW P1a):** returned credit wakes the owning
   loop via its command queue (coalesced), removing the 2 ms-tick latency floor.
5. **Don't pin `SO_RCVBUF` (REVIEW P1c):** drop the fixed 8 MiB `SO_RCVBUF`
   (`net.c:96`) so the kernel autotunes on high-BDP links; keep it an opt-in
   tunable.
6. **Fix the `outstanding` data race (REVIEW P1f):** make per-connection
   counters that status reads touch atomic.

- **Effort:** ~1–1.5 weeks. **Risk:** medium-high (core rewrite of the I/O layer).
- **Tests:** existing end-to-end test passes with a 1-loop / 1-peer config;
  add a test with multiple peers of one torrent on one loop, and multiple
  torrents across loops.

## §0b. Keystone: transport abstraction

Decouple the parser from the raw socket so io_uring/MSE/µTP can slot in. Small;
do alongside §0a (the push-model parser is the shared piece).

```c
typedef struct {
    ssize_t (*recv)(void *ctx, void *buf, size_t n);   /* -1/EAGAIN, 0 closed */
    ssize_t (*send)(void *ctx, const void *buf, size_t n);
    int     (*want_fds)(void *ctx, int *fds, int max); /* fds the loop polls  */
    void    (*close)(void *ctx);
    void     *ctx;
} transport;
```

Implementations: `transport_tcp` (current epoll path), later `transport_uring`,
`transport_mse` (wraps an inner transport), `transport_utp`. The loop drives
`t->recv`/`t->send`; encryption and µTP compose by wrapping an inner transport.

- **Effort:** ~1 day on top of §0a. **Risk:** low.

---

## §1. Cross-peer piece picker + in-flight tracking + timeout (correctness core)

Replaces the single-peer `select_next_piece` scan (`scheduler.c:25`) with a
per-torrent picker shared across that torrent's peers (lock-free, since one loop
owns the torrent).

**Design:**
- **Availability counts:** maintain per-piece rarity from every peer's
  `bitfield`/`have` (and `HAVE_ALL`/`HAVE_NONE` from §5). Picker chooses
  highest-priority, then rarest, then lowest index, among pieces some connected
  peer has and that aren't fully in-flight.
- **In-flight map (REVIEW C1):** track outstanding `(piece, begin)` per request
  with the issuing connection and a deadline. On timeout: drop the entry,
  decrement that connection's `outstanding`, and re-arm the block for any peer
  that has it. This closes the "silent drop → permanent stall" hole.
- **Unsolicited-block validation (REVIEW C2):** on an inbound piece message, look
  it up in the in-flight map; if absent, drop the payload **without** consuming a
  slot or decrementing `outstanding`.
- **Block-level requests:** the picker now hands out blocks (not whole pieces),
  enabling the same block to be re-issued to another peer (timeout/endgame).
- **Adaptive pipeline depth (REVIEW P1d):** size per-connection in-flight to the
  measured BDP (RTT from request→first-byte × rate from `update_rate`),
  clamped by arena credit.

- **Effort:** ~1 week. **Risk:** medium. **Tests:** multi-peer seed where one
  peer stalls a block — assert it's re-requested from another and the download
  completes; negative test for unsolicited blocks.

## §2. Endgame mode + CANCEL

Now in scope because the engine owns cross-peer policy.

**Design:** when remaining blocks < threshold, allow a block to be requested from
multiple peers simultaneously; when one arrives, send `CANCEL` to the others and
drop their in-flight entries. Reuses the §1 in-flight map and block-level
requests. `CANCEL` send path lives in `proto.c` (currently CANCEL is only
recognized inbound).

- **Effort:** ~2–3 days. **Risk:** low-medium (depends on §1). **Tests:** seed
  with one deliberately slow peer near completion; assert endgame races finish
  the tail and losers receive `CANCEL`.

## §3. IPv6 (small; do early)

`AF_INET`-only (`net.c:90`) roughly halves reachable peers. Replace the
hard-coded `sockaddr_in`/`inet_pton` with `sockaddr_storage` + `getaddrinfo`-style
handling; detect a literal `:` to choose `AF_INET6`. No conceptual ABI change
(peer addresses are already strings).

- **Effort:** ~0.5 day. **Risk:** low. **Tests:** local seed bound to `[::1]`.

## §4. Fast Extension (BEP-6) + Extension Protocol (BEP-10)

**Why:** **Reject Request** turns §1's timeout-based recovery into *instant*
recovery — a peer that won't serve a block says so explicitly. `HAVE_ALL`/
`HAVE_NONE` populate availability in one message. BEP-10 is the negotiation base
real peers expect (and the prerequisite for PEX/metadata later).

**Design:**
- Handshake reserved bits: Fast-Ext (`reserved[7] |= 0x04`), Ext-Protocol
  (`reserved[5] |= 0x10`); use a feature only if the peer also set its bit.
- Inbound in `handle_control`: `HAVE_ALL (0x0E)`/`HAVE_NONE (0x0F)` →
  availability; `REJECT_REQUEST (0x10)` → look up in the §1 in-flight map,
  decrement, re-arm immediately; `SUGGEST_PIECE`/`ALLOWED_FAST` → optional
  selection bias (Allowed-Fast lets us request while choked).
- Outbound: send `HAVE_NONE` on connect when both support Fast; add a minimal
  bencode reader for the BEP-10 extended handshake.

- **Effort:** ~1.5 days (incl. minimal bencode). **Risk:** low-medium. **Tests:**
  mock peer advertises Fast, sends `HAVE_ALL` then `REJECT_REQUEST`; assert
  instant re-request (faster than the timeout) and correct completion.

## §5. io_uring receive path (REVIEW P2b)

**Why:** near-syscall-free, genuine zero-copy receive at multi-GB/s. **Reasons
it's not first:** kernel-version gating (multishot recv + provided buffers
≈ 5.19+), seccomp/sandbox policies that disable io_uring, and that it's a
throughput-only win whose payoff appears only after the other bottlenecks are
gone. So: opt-in, with the epoll path as permanent fallback.

**Design:** a `transport_uring` selected per loop at runtime (probe at startup;
fall back to `transport_tcp`). Multishot `IORING_OP_RECV` with a provided-buffer
ring feeds the §0a push-model parser. Stretch: register the per-loop arena with
`io_uring_register_buffers` and use fixed-buffer reads into the destination slot,
removing the residual `memcpy` (REVIEW P1b). Batch request writes via
`IORING_OP_SEND`.

- **Dependency:** `liburing`, optional + feature-gated. **Effort:** ~3–4 days.
  **Risk:** medium (kernel variance). **Tests:** full suite against both backends
  via a config knob; CI selects by kernel capability.

## §6. In-library SHA hashing (REVIEW P2c)

**Why:** with credit backpressure, **wire speed is hostage to hash speed**, and
hashing currently runs in Python. Make it an opt-in in-library mode (the harness
path stays default).

**Design:**
- **Engine-level `HashPool`** of N worker threads (shared across loops).
- **Config/API:** `verify_in_library` flag, `worker_threads`,
  `torrent_set_piece_hashes(...)`, and a results channel (`{piece, ok}` ring the
  consumer drains; verified bytes via in-lib piece buffer / write-fd / callback).
- **Flow:** the loop copies each block from its slot into a pooled per-piece
  assembly buffer and **releases the slot immediately** (wire stays full; slots
  never leave the loop). When a piece is complete, enqueue a hash job; a worker
  runs SHA-1 and reports the result.
- **SHA-1 with SHA-NI:** runtime-dispatched (`__builtin_cpu_supports("sha")`)
  with a portable fallback. Reused by MSE key derivation (§7).

- **Effort:** ~3 days. **Risk:** medium (new threading/memory subsystem).
  **Tests:** enable in-lib verification against a local seed; assert all results
  `ok` and bytes match; negative test with a wrong expected hash (retry path).

## §7. MSE / PE encryption

**Why:** many real-swarm peers refuse plaintext. A `transport_mse` wrapping an
inner transport (§0b) performs the MSE handshake as initiator before BT bytes
flow: DH key exchange (768-bit MSE prime) → obfuscated handshake
(`HASH('req1',S)`, `HASH('req2',SKEY) xor HASH('req3',S)`, `SKEY=info_hash`,
random pad) → cipher negotiation (`crypto_provide`/`select`, RC4 or plaintext,
keys from `HASH('keyA'/'keyB',S,SKEY)`, RC4-drop-1024) → transparent
encrypt/decrypt so `proto.c` is unchanged. Primitives: RC4 (~15 lines), SHA-1
(reuse §6), bignum modexp (vendored small bigint or `libcrypto` BN — decision
below). Config: `enc_policy` (plaintext-only / prefer / require).

- **Effort:** ~4–5 days. **Risk:** high (security-sensitive). **Tests:**
  libtorrent seed with encryption forced; assert completion; cross-check RC4 and
  plaintext selection; run under ASan.

## §8. µTP (largest item)

**Why:** a large fraction of real peers are µTP-only (BitTorrent over UDP,
LEDBAT). A `transport_utp` presents a reliable in-order byte stream to the parser
over a UDP socket: packet types `ST_SYN/DATA/STATE/FIN/RESET` with
`seq_nr`/`ack_nr`/timestamps + selective-ack; LEDBAT congestion control (one-way
delay target ~100 ms, AIMD-style window — the core complexity); retransmission
timers, send/recv windows, in-order reassembly; UDP fd registered via the
transport's `want_fds`, timers driven off the loop tick; connect as initiator.

- **Effort:** ~1–2 weeks. **Risk:** high (timing/congestion correctness).
  **Tests:** libtorrent seed forced to µTP-only over `127.0.0.1`; verify bytes
  and LEDBAT back-off under induced delay.

---

## Suggested ordering & milestones

1. **§0a reactor restructure + §0b transport abstraction** (keystones; share the
   push-model parser). Land with multi-peer/multi-torrent tests green.
2. **§1 picker + in-flight + timeout** and **§2 endgame/CANCEL** — the core that
   makes multi-peer actually correct and fast.
3. **§3 IPv6** + **§4 Fast Extension/BEP-10** — quick wins; Fast-Ext upgrades §1
   recovery from timeout to instant.
4. **§6 in-library hashing** — removes the Python hash ceiling (independent).
5. **§5 io_uring** — throughput, on the transport abstraction.
6. **§7 MSE** — reach; reuses SHA-1 from §6.
7. **§8 µTP** — biggest; reach; last.

§5/§6 are independent and can be parallelized; §7 and §8 both sit on §0b.

## Decisions needed before starting

- **Loop count / sizing:** fixed (e.g. `min(ncpu, N)`) vs. configurable; pinning
  loop threads to cores? Default torrent→loop assignment policy (least-loaded)?
- **ABI shape:** engine-centric API + thin single-peer compat wrapper — confirm
  the new surface and how the Python harness/consumer drains completions across
  many connections.
- **Cross-loop sharding for mega-torrents:** ship affinity-only first and add
  sharded picker later, or design the picker for optional sharding up front?
- **Crypto dependency (§7):** vendor minimal bigint + RC4 + SHA-1, or link
  `libcrypto` (build dep + RC4 legacy-provider wrinkle)?
- **liburing (§5):** optional + runtime probe + epoll fallback (recommended).
- **In-library hashing output (§6):** results-ring + in-lib piece buffers only,
  or also a write-to-fd path so the C side can persist without the harness?
- **Minimum kernel/libc baseline** for io_uring and µTP timers.