Initial commit: Naut-Torrent — from-scratch 10 GbE BitTorrent client

A maintainable, extensible BitTorrent client (C11, Linux/io_uring) targeting 10 GbE saturation. All torrent functionality is built from scratch; liburing is the only linked third-party dependency on the data path. Implements Phases 1-7 of the roadmap: - core: page-aligned buffer pool, MPMC/Treiber queues, bitfields, worker pool - crypto: SHA-1/256 (SHA-NI + scalar), Merkle (BEP-52), RC4 (MSE) - bencode/metainfo: zero-copy parser, v1/v2/hybrid .torrent + magnet - peer: sans-IO wire codec, MSE/PE handshake state machine, BEP-10, ut_metadata, PEX - piece/storage: block-level multi-peer engine, rarest-first + endgame, per-file completion events + single-file relocate (move-as-you-finish) - tracker/dht: HTTP + UDP (BEP-15) trackers, BEP-5 KRPC iterative lookup - platform: io_uring reactor (SQPOLL, registered buffers, SEND_ZC) - surface: versioned RPC, native plugin ABI, sandboxed Lua scripting, nautd/nautctl Verified against libtorrent (single/multi/hybrid, MSE, magnet-via-DHT, swarm); unit + interop tests green; ASan/UBSan/TSan clean. Scripting reference in docs/scripting.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 12:12:00 -04:00 · 2026-06-15 12:12:00 -04:00 · 2178d6a70c
commit 2178d6a70c
121 changed files with 12644 additions and 0 deletions
--- a/plan.md
+++ b/plan.md
@ -0,0 +1,168 @@
+# Naut-Torrent — Architecture & Implementation Plan
+
+## Context
+
+This is a greenfield project (`Naut-Torrent/` is empty). The goal is a **maintainable, extensible BitTorrent client** capable of **saturating a 10 Gigabit Ethernet link (~1.25 GB/s) in both directions**. The implementation language is C-equivalent, so the plan is expressed in C terms (structs, function-pointer vtables, manual memory management, no exceptions/RAII).
+
+Decisions confirmed with the user:
+
+- **Platform:** Linux-only, optimized hard around **io_uring** (network *and* disk).
+- **Protocol:** BitTorrent **v1 + v2 hybrid** (BEP-3 SHA-1 pieces *and* BEP-52 SHA-256 Merkle trees).
+- **Workload:** Saturate **both download and upload**, with **MSE/PE encryption (RC4)** in the hot path.
+- **Extensibility:** Clean module boundaries + control **RPC** + full **BEP-10 extension protocol** + a versioned **native plugin ABI** + an **embedded scripting** runtime.
+
+The architecture is driven first by the throughput budget below — every structural decision (shared-nothing reactors, work offload, zero-copy buffers) exists to hit 1.25 GB/s on commodity multicore hardware.
+
+---
+
+## 1. Throughput Budget (why the architecture looks the way it does)
+
+At **1.25 GB/s sustained**, the per-byte costs that must be parallelized:
+
+| Work item | Cost (per core, conservative) | Cores @ 1.25 GB/s |
+|---|---|---|
+| MSE RC4 encrypt/decrypt | ~500 MB/s/core | ~2.5 per active direction |
+| SHA-256 verify (v2, SHA-NI) | ~1.5 GB/s/core | ~1 (download) |
+| SHA-1 verify (v1, SHA-NI) | ~2.5 GB/s/core | <1 |
+| Network recv/send + framing | high, but DMA-bound | spread across reactors |
+| Disk write/read (O_DIRECT NVMe) | 5–7 GB/s/device | I/O-bound, not CPU |
+| memcpy | **eliminate** via registered buffers | ~0 |
+
+**Conclusions that shape the design:**
+
+1. RC4 is the surprise cost — encryption alone wants several cores. The hot path **must scale linearly across cores** (shared-nothing reactors, no global locks on the data path).
+2. Hashing and crypto are **offloadable, embarrassingly parallel** units of work → dedicated **worker pools** fed by lock-free queues, never run inline on a reactor.
+3. **memcpy must be designed out**: kernel→userspace via io_uring registered buffers, hash/crypto operate in place, disk writes come straight from the same buffers (O_DIRECT, page-aligned).
+4. A ~8–16 core box with one Gen4 NVMe and a 10 GbE NIC should saturate the link; the software just has to not get in the way (syscalls, locks, copies, allocator churn).
+
+---
+
+## 2. High-Level Architecture: Shared-Nothing Thread-Per-Core Reactors + Offload Pools
+
+```
+              ┌──────────────────────── Control plane (1 thread) ───────────────────────┐
+              │  RPC server · plugin host · script VM · session/torrent registry · stats │
+              └───────┬─────────────────────────────────────────────────────────────────┘
+                      │ message passing (MPSC command queues, no shared locks on hot path)
+   ┌──────────────┬───┴──────────┬──────────────┐
+   │ Reactor 0    │ Reactor 1    │ … Reactor N-1 │   (one pinned thread per I/O core)
+   │ own io_uring │ own io_uring │ own io_uring  │   network + disk SQ/CQ, shared-nothing
+   │ owns a shard │ owns a shard │ owns a shard  │   of peer connections (SO_REUSEPORT)
+   └──────┬───────┴──────┬───────┴───────┬───────┘
+          │ submit jobs   │               │  job results returned via io_uring msg_ring / eventfd
+   ┌──────┴───────────────┴───────────────┴───────┐
+   │  Hash worker pool        Crypto-assist pool   │  (pinned to remaining cores,
+   │  (SHA-1 / SHA-256 / Merkle, MPMC job queue)   │   pull jobs, post completions)
+   └───────────────────────────────────────────────┘
+```
+
+**Sharding model (the central decision):**
+
+- **Connections are sharded across reactors.** A `SO_REUSEPORT` listen socket per reactor lets the kernel hash inbound peers across cores; outbound peers are assigned by hash. Each reactor **exclusively owns** its connections → all per-connection state (recv/send buffers, MSE keystream, message parser) is **lock-free**.
+- **Torrent piece-state is shared** (peers of one torrent live on many reactors). It is guarded by a **per-torrent lock that protects only bookkeeping** (the picker's rarity counts, request map, have-bitfield). The lock is held for microseconds; all heavy work (decrypt, hash, disk) happens **outside** it. Rationale: the expensive bytes never touch a lock; only the tiny "which block next / mark block received" decisions do.
+- **Heavy work is offloaded**, not run on reactors: completed piece buffers go to the **hash pool**; crypto can run inline on the reactor (cheap per-message) or be batched to a crypto-assist pool under load. Results return to the **owning reactor** via `io_uring` `msg_ring` (cross-ring wakeup) or eventfd.
+- **Control plane is off the hot path entirely.** RPC, plugins, and scripts run on their own thread(s) and communicate with reactors only via per-reactor MPSC command queues + event fan-out. A misbehaving plugin/script can never stall the data path.
+
+**Single-torrent scaling note:** because crypto/hash/disk for one torrent fan out to all cores via the pools, even a lone large torrent uses the whole machine. Only the per-torrent picker lock is single-point; if it ever becomes hot, upgrade the picker to the lock-free variant (§6, optimization).
+
+**io_uring features exploited:** multishot `accept`/`recv`, registered buffers (`PROVIDE_BUFFERS` ring) for zero-copy recv, registered files, `SEND_ZC` (zero-copy send) for seeding, `SQPOLL` (kernel-side submission polling to cut syscalls under load), `msg_ring` for cross-thread completions, linked SQEs for read→hash chaining. NUMA-aware + hugepage buffer pools per reactor.
+
+---
+
+## 3. Module / Layer Breakdown
+
+Layers are bottom-up; each is independently unit-testable and has explicit extension seams. Suggested repo layout under `src/`.
+
+### Foundation
+
+- **`platform/`** — the *only* code that touches the kernel. io_uring lifecycle (ring setup, SQE build, CQE reaping), socket setup (`SO_REUSEPORT`, `TCP_NODELAY`, large `SO_RCV/SNDBUF`, `TCP_FASTOPEN`), file ops (`O_DIRECT`, `fallocate`, `fadvise`), `eventfd`/`timerfd`, CPU pinning, hugepage/NUMA allocation, monotonic clock. This is the seam that keeps "Linux-only now" from becoming "Linux-only forever."
+- **`core/`** — data-structure toolbox, zero dependencies:
+  - **Buffer pool**: slab allocator of fixed-size (16 KiB block + piece-sized) page-aligned blocks, per-reactor freelists, refcounted so one buffer flows recv→decrypt→hash→disk without copy.
+  - Intrusive doubly-linked lists, open-addressing hash maps, dynamic arrays, **bitfields with popcount** (have/interested/request maps), object pools, **SPSC + MPSC + MPMC lock-free queues**, per-thread lock-free logging ring, config parser, lock-free stats counters.
+- **`crypto/`** — SHA-1, SHA-256 with **runtime SHA-NI dispatch** (scalar fallback), **Merkle tree** builder/verifier (v2 piece layers), **RC4** (MSE), **Diffie-Hellman** (MSE handshake, RFC 2631 768-bit group), CSPRNG. Designed for in-place operation on pooled buffers.
+
+### Protocol
+
+- **`bencode/`** — streaming, allocation-light parser producing **zero-copy slices** into the source buffer; encoder. Hardened against malicious input (depth/size limits) — primary fuzz target.
+- **`metainfo/`** — `.torrent` parse for v1, v2, and hybrid; file tree + piece layers; **magnet URI** parsing; info-hash computation (both v1 SHA-1 and v2 SHA-256 truncated).
+- **`tracker/`** — interface `tracker_backend` with built-in **HTTP(S)** (BEP-3/BEP-23 compact) and **UDP** (BEP-15) backends; announce scheduling, scrape, multi-tier (BEP-12). *Extension seam: register custom backends.*
+- **`dht/`** — Kademlia routing table, `get_peers`/`announce_peer`/`find_node`, bootstrap, token management, BEP-32 IPv6, BEP-51 infohash indexing. Runs as its own UDP endpoint on a reactor.
+- **`peer/`** — wire codec (handshake, all BEP-3 messages), **MSE/PE layer** (DH handshake, RC4 keystream, plaintext fallback), framing, and a **BEP-10 extension registry** (`extension_handler` vtable) with built-ins: **PEX (BEP-11)**, **ut_metadata (BEP-9)**, **LTEP** negotiation. *Extension seam: register custom extension message handlers.*
+
+### Engine
+
+- **`piece/`** — `piece_picker` interface (vtable) with built-in **rarest-first**, **sequential/streaming**, **priority**, and **endgame** strategies; per-peer **adaptive request pipelining** (depth auto-tuned from observed throughput × RTT, not a fixed window — essential for filling a 10 GbE BDP); block accounting; per-torrent lock holder. *Extension seam: pluggable picker strategy.*
+- **`storage/`** — `storage_backend` interface with built-in **file backend**: maps (piece,offset)→(file,offset) across multi-file torrents, **O_DIRECT** aligned read/write via the reactor's io_uring, write coalescing, bounded read cache, configurable fsync policy, **fast-resume** (persisted bitfield + partial-piece state), preallocation. *Extension seam: register custom storage (memory, network, object-store).*
+- **`verify/`** — hashing job dispatcher: full-piece SHA-1 (v1) and incremental **Merkle leaf/branch** hashing (v2) submitted to the hash pool; on completion marks piece valid/invalid on the owning reactor.
+- **`scheduler/`** — token-bucket **rate limiting** (global + per-torrent + per-peer), **choking** algorithm (tit-for-tat + optimistic unchoke, BEP-3), connection budget, bandwidth fairness across torrents, super-seeding (BEP-16) option.
+- **`session/`** — torrent lifecycle state machine, peer-set management, alert/event bus, aggregate stats. The single registry the control plane talks to.
+
+### Surface (the "extensible" layer)
+
+- **`rpc/`** — control protocol over a UNIX domain socket (and optional TCP): command/response + **streaming event subscription**. Versioned, length-prefixed binary frames (compact) with an optional JSON mode for tooling. A stable **command registry** so plugins can add RPC verbs. This is how UIs/automation drive the client.
+- **`plugin/`** — **versioned C ABI**: host passes a `naut_host_api` struct of function pointers; the plugin (`.so`) exports `naut_plugin_register(host)`. Plugins hook the same vtables the core uses: `tracker_backend`, `storage_backend`, `piece_picker`, `extension_handler`, RPC commands, and the event bus. ABI version checked at load; plugins run on the control thread, never the data path.
+- **`script/`** — embedded **Lua-style VM** bound to the event bus (`on_torrent_added`, `on_piece_complete`, **`on_file_complete`**, `on_torrent_finished`, `on_peer_connected`, `on_alert`) and a sandboxed control API. Runs on a dedicated thread with a bounded work queue; cannot block reactors.
+  - **User feature — move files as they finish (libtorrent can't):** the engine seam is already built (Phase 3): `naut_download` fires `on_file_complete(file_index, path)` the moment a file's last covering piece verifies, and `naut_storage_relocate()` moves a single completed file safely (even mid-download). Phase 7 forwards the event to the `on_file_complete` script hook and exposes a `move_file` API; the relocate must be marshalled onto the owning reactor thread via the command queue (scripts run on their own thread).
+
+---
+
+## 4. Concurrency & Memory Model (invariants)
+
+- **Data path is lock-free.** A connection is touched by exactly one reactor. The only data-path lock is the per-torrent picker mutex, held only for O(1)–O(log n) bookkeeping.
+- **One buffer, one lifetime, zero copies.** A pooled, refcounted, page-aligned buffer is filled by `recv` (registered buffer), decrypted in place, hashed in place by a worker, then written by `SEND_ZC`/`O_DIRECT` write — same physical pages throughout.
+- **Cross-thread communication is message passing**, never shared mutable state: MPSC command queues into reactors, MPMC job queue into the hash pool, `msg_ring`/eventfd for completions back out.
+- **No allocation on the hot path.** All buffers, connection objects, request objects come from preallocated per-reactor pools sized from `peers × pipeline_depth × block_size`. Allocator is only touched at torrent add/remove.
+- **Backpressure is explicit.** Bounded queues everywhere; when the hash pool or disk falls behind, reactors stop issuing `recv` (flow control) rather than growing memory unboundedly.
+
+---
+
+## 5. Build, Repo & Quality
+
+```
+Naut-Torrent/
+├── src/{platform,core,crypto,bencode,metainfo,tracker,dht,peer,piece,storage,verify,scheduler,session,rpc,plugin,script}/
+├── include/naut/         # public headers incl. versioned plugin ABI (naut_plugin.h)
+├── apps/{nautd,nautctl}/  # daemon + CLI client over RPC
+├── plugins/example/       # reference plugin against the ABI
+├── tests/{unit,fuzz,integration,bench}/
+└── build/                 # build-system outputs
+```
+
+- **Daemon/CLI split**: `nautd` (engine) + `nautctl` (thin RPC client). UIs are just RPC consumers — keeps the core headless and embeddable.
+- **Tooling**: AddressSanitizer/UBSan/ThreadSanitizer builds; `perf`/eBPF-friendly; built-in per-core stats exported over RPC.
+
+---
+
+## 6. Implementation Phases (milestones, each independently demoable)
+
+1. **Foundation** — `platform/` io_uring echo server + `core/` buffer pool, queues, bitfields, hash map. *Gate: loopback echo saturates a core with zero per-op allocation.*
+2. **Crypto + parsers** — SHA-1/256 (+SHA-NI), Merkle, RC4, DH; `bencode/`, `metainfo/` (v1/v2/hybrid + magnet). *Gate: parse real torrents; hash throughput benchmark ≥1.5 GB/s/core.*
+3. **Single-peer transfer** — `peer/` handshake + messages (plaintext), `piece/` basic picker, `storage/` file backend, `verify/`. Download a real torrent from one peer to disk, verified. *Gate: byte-correct file from a known seed.*
+4. **Trackers + swarm** — HTTP/UDP `tracker/`, choking `scheduler/`, multi-peer, rarest-first, endgame. *Gate: download from a public/local swarm; interop with libtorrent/Transmission.*
+5. **MSE + DHT + extensions** — RC4 MSE handshake, `dht/`, PEX, ut_metadata, magnet-only start. *Gate: magnet link with no trackers completes via DHT; encrypted peers work.*
+6. **Scale to 10 GbE** — adaptive pipelining, `SEND_ZC`, registered buffers, `SQPOLL`, hash/crypto pools, NUMA/hugepages, O_DIRECT tuning. *Gate: two boxes (or two NICs) sustain ≥9.4 Gbit/s both directions.* Optional: lock-free picker if the per-torrent lock shows contention in `perf`.
+7. **Extensibility surface** — `rpc/`, `plugin/` ABI + reference plugin, `script/` VM + event hooks, `nautctl`. *Gate: a sample plugin adds a storage backend and a script reacts to `on_torrent_finished`.*
+
+---
+
+## 7. Verification (end-to-end)
+
+- **Microbenchmarks** (`tests/bench/`): hash GB/s/core, RC4 GB/s/core, bencode parse MB/s, buffer-pool alloc/free ns, picker decisions/s. Each has a regression threshold tied to the §1 budget.
+- **Correctness**: unit tests per module; **fuzzers** (`tests/fuzz/`) on bencode, peer-message, tracker-response, and metainfo parsers (the attack surface); byte-for-byte file verification after download; resume-from-partial test.
+- **Interop**: run against **libtorrent/qBittorrent** and **Transmission** as both seed and leech, plaintext and MSE; magnet + DHT-only bootstrap test against the public DHT.
+- **Throughput (the headline test)**:
+  1. Baseline the link with `iperf3` to confirm ~9.4 Gbit/s is achievable end-to-end.
+  2. Seed a large (≥50 GB) torrent from box A, download on box B over the 10 GbE link (or two NICs on one box via loopback-to-NIC). Measure with the client's own per-core RPC stats + `nstat`/`ifstat`.
+  3. Confirm sustained throughput is link-bound (not CPU/disk/lock-bound) via `perf top` — no single thread pinned at 100% on locks/copies, hash & crypto spread across pool cores.
+  4. Reverse roles to validate upload saturation (`SEND_ZC` path).
+- **Soak**: 24 h multi-torrent run under ASan-off release build; assert flat memory (pools, no leaks), no descriptor growth, stable throughput.
+
+---
+
+## 8. Key Risks & Mitigations
+
+- **Per-torrent picker lock contention** at 10 GbE → keep heavy work outside the lock; escalate to lock-free claim-by-CAS picker (already an interface, so it's a swap not a rewrite).
+- **RC4/MSE CPU cost** underestimated → crypto-assist pool + prefer plaintext when both peers allow; measure early in Phase 5.
+- **io_uring portability lock-in** → all kernel calls behind `platform/`; a future epoll backend is a new file, not a refactor.
+- **O_DIRECT alignment complexity** → enforce page-aligned pooled buffers from day one (Phase 1), so storage never has to bounce-buffer.
+- **Hybrid v1/v2 data-model complexity** → model the piece/file/Merkle layout once in `metainfo/` + `storage/` and treat v1 as the degenerate single-layer case.