# Naut-Torrent — Architecture & Implementation Plan ## Context This is a greenfield project (`Naut-Torrent/` is empty). The goal is a **maintainable, extensible BitTorrent client** capable of **saturating a 10 Gigabit Ethernet link (~1.25 GB/s) in both directions**. The implementation language is C-equivalent, so the plan is expressed in C terms (structs, function-pointer vtables, manual memory management, no exceptions/RAII). Decisions confirmed with the user: - **Platform:** Linux-only, optimized hard around **io_uring** (network *and* disk). - **Protocol:** BitTorrent **v1 + v2 hybrid** (BEP-3 SHA-1 pieces *and* BEP-52 SHA-256 Merkle trees). - **Workload:** Saturate **both download and upload**, with **MSE/PE encryption (RC4)** in the hot path. - **Extensibility:** Clean module boundaries + control **RPC** + full **BEP-10 extension protocol** + a versioned **native plugin ABI** + an **embedded scripting** runtime. The architecture is driven first by the throughput budget below — every structural decision (shared-nothing reactors, work offload, zero-copy buffers) exists to hit 1.25 GB/s on commodity multicore hardware. --- ## 1. Throughput Budget (why the architecture looks the way it does) At **1.25 GB/s sustained**, the per-byte costs that must be parallelized: | Work item | Cost (per core, conservative) | Cores @ 1.25 GB/s | |---|---|---| | MSE RC4 encrypt/decrypt | ~500 MB/s/core | ~2.5 per active direction | | SHA-256 verify (v2, SHA-NI) | ~1.5 GB/s/core | ~1 (download) | | SHA-1 verify (v1, SHA-NI) | ~2.5 GB/s/core | <1 | | Network recv/send + framing | high, but DMA-bound | spread across reactors | | Disk write/read (O_DIRECT NVMe) | 5–7 GB/s/device | I/O-bound, not CPU | | memcpy | **eliminate** via registered buffers | ~0 | **Conclusions that shape the design:** 1. RC4 is the surprise cost — encryption alone wants several cores. The hot path **must scale linearly across cores** (shared-nothing reactors, no global locks on the data path). 2. Hashing and crypto are **offloadable, embarrassingly parallel** units of work → dedicated **worker pools** fed by lock-free queues, never run inline on a reactor. 3. **memcpy must be designed out**: kernel→userspace via io_uring registered buffers, hash/crypto operate in place, disk writes come straight from the same buffers (O_DIRECT, page-aligned). 4. A ~8–16 core box with one Gen4 NVMe and a 10 GbE NIC should saturate the link; the software just has to not get in the way (syscalls, locks, copies, allocator churn). --- ## 2. High-Level Architecture: Shared-Nothing Thread-Per-Core Reactors + Offload Pools ``` ┌──────────────────────── Control plane (1 thread) ───────────────────────┐ │ RPC server · plugin host · script VM · session/torrent registry · stats │ └───────┬─────────────────────────────────────────────────────────────────┘ │ message passing (MPSC command queues, no shared locks on hot path) ┌──────────────┬───┴──────────┬──────────────┐ │ Reactor 0 │ Reactor 1 │ … Reactor N-1 │ (one pinned thread per I/O core) │ own io_uring │ own io_uring │ own io_uring │ network + disk SQ/CQ, shared-nothing │ owns a shard │ owns a shard │ owns a shard │ of peer connections (SO_REUSEPORT) └──────┬───────┴──────┬───────┴───────┬───────┘ │ submit jobs │ │ job results returned via io_uring msg_ring / eventfd ┌──────┴───────────────┴───────────────┴───────┐ │ Hash worker pool Crypto-assist pool │ (pinned to remaining cores, │ (SHA-1 / SHA-256 / Merkle, MPMC job queue) │ pull jobs, post completions) └───────────────────────────────────────────────┘ ``` **Sharding model (the central decision):** - **Connections are sharded across reactors.** A `SO_REUSEPORT` listen socket per reactor lets the kernel hash inbound peers across cores; outbound peers are assigned by hash. Each reactor **exclusively owns** its connections → all per-connection state (recv/send buffers, MSE keystream, message parser) is **lock-free**. - **Torrent piece-state is shared** (peers of one torrent live on many reactors). It is guarded by a **per-torrent lock that protects only bookkeeping** (the picker's rarity counts, request map, have-bitfield). The lock is held for microseconds; all heavy work (decrypt, hash, disk) happens **outside** it. Rationale: the expensive bytes never touch a lock; only the tiny "which block next / mark block received" decisions do. - **Heavy work is offloaded**, not run on reactors: completed piece buffers go to the **hash pool**; crypto can run inline on the reactor (cheap per-message) or be batched to a crypto-assist pool under load. Results return to the **owning reactor** via `io_uring` `msg_ring` (cross-ring wakeup) or eventfd. - **Control plane is off the hot path entirely.** RPC, plugins, and scripts run on their own thread(s) and communicate with reactors only via per-reactor MPSC command queues + event fan-out. A misbehaving plugin/script can never stall the data path. **Single-torrent scaling note:** because crypto/hash/disk for one torrent fan out to all cores via the pools, even a lone large torrent uses the whole machine. Only the per-torrent picker lock is single-point; if it ever becomes hot, upgrade the picker to the lock-free variant (§6, optimization). **io_uring features exploited:** multishot `accept`/`recv`, registered buffers (`PROVIDE_BUFFERS` ring) for zero-copy recv, registered files, `SEND_ZC` (zero-copy send) for seeding, `SQPOLL` (kernel-side submission polling to cut syscalls under load), `msg_ring` for cross-thread completions, linked SQEs for read→hash chaining. NUMA-aware + hugepage buffer pools per reactor. --- ## 3. Module / Layer Breakdown Layers are bottom-up; each is independently unit-testable and has explicit extension seams. Suggested repo layout under `src/`. ### Foundation - **`platform/`** — the *only* code that touches the kernel. io_uring lifecycle (ring setup, SQE build, CQE reaping), socket setup (`SO_REUSEPORT`, `TCP_NODELAY`, large `SO_RCV/SNDBUF`, `TCP_FASTOPEN`), file ops (`O_DIRECT`, `fallocate`, `fadvise`), `eventfd`/`timerfd`, CPU pinning, hugepage/NUMA allocation, monotonic clock. This is the seam that keeps "Linux-only now" from becoming "Linux-only forever." - **`core/`** — data-structure toolbox, zero dependencies: - **Buffer pool**: slab allocator of fixed-size (16 KiB block + piece-sized) page-aligned blocks, per-reactor freelists, refcounted so one buffer flows recv→decrypt→hash→disk without copy. - Intrusive doubly-linked lists, open-addressing hash maps, dynamic arrays, **bitfields with popcount** (have/interested/request maps), object pools, **SPSC + MPSC + MPMC lock-free queues**, per-thread lock-free logging ring, config parser, lock-free stats counters. - **`crypto/`** — SHA-1, SHA-256 with **runtime SHA-NI dispatch** (scalar fallback), **Merkle tree** builder/verifier (v2 piece layers), **RC4** (MSE), **Diffie-Hellman** (MSE handshake, RFC 2631 768-bit group), CSPRNG. Designed for in-place operation on pooled buffers. ### Protocol - **`bencode/`** — streaming, allocation-light parser producing **zero-copy slices** into the source buffer; encoder. Hardened against malicious input (depth/size limits) — primary fuzz target. - **`metainfo/`** — `.torrent` parse for v1, v2, and hybrid; file tree + piece layers; **magnet URI** parsing; info-hash computation (both v1 SHA-1 and v2 SHA-256 truncated). - **`tracker/`** — interface `tracker_backend` with built-in **HTTP(S)** (BEP-3/BEP-23 compact) and **UDP** (BEP-15) backends; announce scheduling, scrape, multi-tier (BEP-12). *Extension seam: register custom backends.* - **`dht/`** — Kademlia routing table, `get_peers`/`announce_peer`/`find_node`, bootstrap, token management, BEP-32 IPv6, BEP-51 infohash indexing. Runs as its own UDP endpoint on a reactor. - **`peer/`** — wire codec (handshake, all BEP-3 messages), **MSE/PE layer** (DH handshake, RC4 keystream, plaintext fallback), framing, and a **BEP-10 extension registry** (`extension_handler` vtable) with built-ins: **PEX (BEP-11)**, **ut_metadata (BEP-9)**, **LTEP** negotiation. *Extension seam: register custom extension message handlers.* ### Engine - **`piece/`** — `piece_picker` interface (vtable) with built-in **rarest-first**, **sequential/streaming**, **priority**, and **endgame** strategies; per-peer **adaptive request pipelining** (depth auto-tuned from observed throughput × RTT, not a fixed window — essential for filling a 10 GbE BDP); block accounting; per-torrent lock holder. *Extension seam: pluggable picker strategy.* - **`storage/`** — `storage_backend` interface with built-in **file backend**: maps (piece,offset)→(file,offset) across multi-file torrents, **O_DIRECT** aligned read/write via the reactor's io_uring, write coalescing, bounded read cache, configurable fsync policy, **fast-resume** (persisted bitfield + partial-piece state), preallocation. *Extension seam: register custom storage (memory, network, object-store).* - **`verify/`** — hashing job dispatcher: full-piece SHA-1 (v1) and incremental **Merkle leaf/branch** hashing (v2) submitted to the hash pool; on completion marks piece valid/invalid on the owning reactor. - **`scheduler/`** — token-bucket **rate limiting** (global + per-torrent + per-peer), **choking** algorithm (tit-for-tat + optimistic unchoke, BEP-3), connection budget, bandwidth fairness across torrents, super-seeding (BEP-16) option. - **`session/`** — torrent lifecycle state machine, peer-set management, alert/event bus, aggregate stats. The single registry the control plane talks to. ### Surface (the "extensible" layer) - **`rpc/`** — control protocol over a UNIX domain socket (and optional TCP): command/response + **streaming event subscription**. Versioned, length-prefixed binary frames (compact) with an optional JSON mode for tooling. A stable **command registry** so plugins can add RPC verbs. This is how UIs/automation drive the client. - **`plugin/`** — **versioned C ABI**: host passes a `naut_host_api` struct of function pointers; the plugin (`.so`) exports `naut_plugin_register(host)`. Plugins hook the same vtables the core uses: `tracker_backend`, `storage_backend`, `piece_picker`, `extension_handler`, RPC commands, and the event bus. ABI version checked at load; plugins run on the control thread, never the data path. - **`script/`** — embedded **Lua-style VM** bound to the event bus (`on_torrent_added`, `on_piece_complete`, **`on_file_complete`**, `on_torrent_finished`, `on_peer_connected`, `on_alert`) and a sandboxed control API. Runs on a dedicated thread with a bounded work queue; cannot block reactors. - **User feature — move files as they finish (libtorrent can't):** the engine seam is already built (Phase 3): `naut_download` fires `on_file_complete(file_index, path)` the moment a file's last covering piece verifies, and `naut_storage_relocate()` moves a single completed file safely (even mid-download). Phase 7 forwards the event to the `on_file_complete` script hook and exposes a `move_file` API; the relocate must be marshalled onto the owning reactor thread via the command queue (scripts run on their own thread). --- ## 4. Concurrency & Memory Model (invariants) - **Data path is lock-free.** A connection is touched by exactly one reactor. The only data-path lock is the per-torrent picker mutex, held only for O(1)–O(log n) bookkeeping. - **One buffer, one lifetime, zero copies.** A pooled, refcounted, page-aligned buffer is filled by `recv` (registered buffer), decrypted in place, hashed in place by a worker, then written by `SEND_ZC`/`O_DIRECT` write — same physical pages throughout. - **Cross-thread communication is message passing**, never shared mutable state: MPSC command queues into reactors, MPMC job queue into the hash pool, `msg_ring`/eventfd for completions back out. - **No allocation on the hot path.** All buffers, connection objects, request objects come from preallocated per-reactor pools sized from `peers × pipeline_depth × block_size`. Allocator is only touched at torrent add/remove. - **Backpressure is explicit.** Bounded queues everywhere; when the hash pool or disk falls behind, reactors stop issuing `recv` (flow control) rather than growing memory unboundedly. --- ## 5. Build, Repo & Quality ``` Naut-Torrent/ ├── src/{platform,core,crypto,bencode,metainfo,tracker,dht,peer,piece,storage,verify,scheduler,session,rpc,plugin,script}/ ├── include/naut/ # public headers incl. versioned plugin ABI (naut_plugin.h) ├── apps/{nautd,nautctl}/ # daemon + CLI client over RPC ├── plugins/example/ # reference plugin against the ABI ├── tests/{unit,fuzz,integration,bench}/ └── build/ # build-system outputs ``` - **Daemon/CLI split**: `nautd` (engine) + `nautctl` (thin RPC client). UIs are just RPC consumers — keeps the core headless and embeddable. - **Tooling**: AddressSanitizer/UBSan/ThreadSanitizer builds; `perf`/eBPF-friendly; built-in per-core stats exported over RPC. --- ## 6. Implementation Phases (milestones, each independently demoable) 1. **Foundation** — `platform/` io_uring echo server + `core/` buffer pool, queues, bitfields, hash map. *Gate: loopback echo saturates a core with zero per-op allocation.* 2. **Crypto + parsers** — SHA-1/256 (+SHA-NI), Merkle, RC4, DH; `bencode/`, `metainfo/` (v1/v2/hybrid + magnet). *Gate: parse real torrents; hash throughput benchmark ≥1.5 GB/s/core.* 3. **Single-peer transfer** — `peer/` handshake + messages (plaintext), `piece/` basic picker, `storage/` file backend, `verify/`. Download a real torrent from one peer to disk, verified. *Gate: byte-correct file from a known seed.* 4. **Trackers + swarm** — HTTP/UDP `tracker/`, choking `scheduler/`, multi-peer, rarest-first, endgame. *Gate: download from a public/local swarm; interop with libtorrent/Transmission.* 5. **MSE + DHT + extensions** — RC4 MSE handshake, `dht/`, PEX, ut_metadata, magnet-only start. *Gate: magnet link with no trackers completes via DHT; encrypted peers work.* 6. **Scale to 10 GbE** — adaptive pipelining, `SEND_ZC`, registered buffers, `SQPOLL`, hash/crypto pools, NUMA/hugepages, O_DIRECT tuning. *Gate: two boxes (or two NICs) sustain ≥9.4 Gbit/s both directions.* Optional: lock-free picker if the per-torrent lock shows contention in `perf`. 7. **Extensibility surface** — `rpc/`, `plugin/` ABI + reference plugin, `script/` VM + event hooks, `nautctl`. *Gate: a sample plugin adds a storage backend and a script reacts to `on_torrent_finished`.* --- ## 7. Verification (end-to-end) - **Microbenchmarks** (`tests/bench/`): hash GB/s/core, RC4 GB/s/core, bencode parse MB/s, buffer-pool alloc/free ns, picker decisions/s. Each has a regression threshold tied to the §1 budget. - **Correctness**: unit tests per module; **fuzzers** (`tests/fuzz/`) on bencode, peer-message, tracker-response, and metainfo parsers (the attack surface); byte-for-byte file verification after download; resume-from-partial test. - **Interop**: run against **libtorrent/qBittorrent** and **Transmission** as both seed and leech, plaintext and MSE; magnet + DHT-only bootstrap test against the public DHT. - **Throughput (the headline test)**: 1. Baseline the link with `iperf3` to confirm ~9.4 Gbit/s is achievable end-to-end. 2. Seed a large (≥50 GB) torrent from box A, download on box B over the 10 GbE link (or two NICs on one box via loopback-to-NIC). Measure with the client's own per-core RPC stats + `nstat`/`ifstat`. 3. Confirm sustained throughput is link-bound (not CPU/disk/lock-bound) via `perf top` — no single thread pinned at 100% on locks/copies, hash & crypto spread across pool cores. 4. Reverse roles to validate upload saturation (`SEND_ZC` path). - **Soak**: 24 h multi-torrent run under ASan-off release build; assert flat memory (pools, no leaks), no descriptor growth, stable throughput. --- ## 8. Key Risks & Mitigations - **Per-torrent picker lock contention** at 10 GbE → keep heavy work outside the lock; escalate to lock-free claim-by-CAS picker (already an interface, so it's a swap not a rewrite). - **RC4/MSE CPU cost** underestimated → crypto-assist pool + prefer plaintext when both peers allow; measure early in Phase 5. - **io_uring portability lock-in** → all kernel calls behind `platform/`; a future epoll backend is a new file, not a refactor. - **O_DIRECT alignment complexity** → enforce page-aligned pooled buffers from day one (Phase 1), so storage never has to bounce-buffer. - **Hybrid v1/v2 data-model complexity** → model the piece/file/Merkle layout once in `metainfo/` + `storage/` and treat v1 as the degenerate single-layer case.