Naut/plan.md

Naut-Torrent — Architecture & Implementation Plan

Context

This is a greenfield project (Naut-Torrent/ is empty). The goal is a maintainable, extensible BitTorrent client capable of saturating a 10 Gigabit Ethernet link (~1.25 GB/s) in both directions. The implementation language is C-equivalent, so the plan is expressed in C terms (structs, function-pointer vtables, manual memory management, no exceptions/RAII).

Decisions confirmed with the user:

• Platform: Linux-only, optimized hard around io_uring (network and disk).
• Protocol: BitTorrent v1 + v2 hybrid (BEP-3 SHA-1 pieces and BEP-52 SHA-256 Merkle trees).
• Workload: Saturate both download and upload, with MSE/PE encryption (RC4) in the hot path.
• Extensibility: Clean module boundaries + control RPC + full BEP-10 extension protocol + a versioned native plugin ABI + an embedded scripting runtime.

The architecture is driven first by the throughput budget below — every structural decision (shared-nothing reactors, work offload, zero-copy buffers) exists to hit 1.25 GB/s on commodity multicore hardware.

---

1. Throughput Budget (why the architecture looks the way it does)

At 1.25 GB/s sustained, the per-byte costs that must be parallelized:

Work item	Cost (per core, conservative)	Cores @ 1.25 GB/s
MSE RC4 encrypt/decrypt	~500 MB/s/core	~2.5 per active direction
SHA-256 verify (v2, SHA-NI)	~1.5 GB/s/core	~1 (download)
SHA-1 verify (v1, SHA-NI)	~2.5 GB/s/core	<1
Network recv/send + framing	high, but DMA-bound	spread across reactors
Disk write/read (O_DIRECT NVMe)	5–7 GB/s/device	I/O-bound, not CPU
memcpy	eliminate via registered buffers	~0

Conclusions that shape the design:

1. RC4 is the surprise cost — encryption alone wants several cores. The hot path must scale linearly across cores (shared-nothing reactors, no global locks on the data path).
2. Hashing and crypto are offloadable, embarrassingly parallel units of work → dedicated worker pools fed by lock-free queues, never run inline on a reactor.
3. memcpy must be designed out: kernel→userspace via io_uring registered buffers, hash/crypto operate in place, disk writes come straight from the same buffers (O_DIRECT, page-aligned).
4. A ~8–16 core box with one Gen4 NVMe and a 10 GbE NIC should saturate the link; the software just has to not get in the way (syscalls, locks, copies, allocator churn).

---

2. High-Level Architecture: Shared-Nothing Thread-Per-Core Reactors + Offload Pools

              ┌──────────────────────── Control plane (1 thread) ───────────────────────┐
              │  RPC server · plugin host · script VM · session/torrent registry · stats │
              └───────┬─────────────────────────────────────────────────────────────────┘
                      │ message passing (MPSC command queues, no shared locks on hot path)
   ┌──────────────┬───┴──────────┬──────────────┐
   │ Reactor 0    │ Reactor 1    │ … Reactor N-1 │   (one pinned thread per I/O core)
   │ own io_uring │ own io_uring │ own io_uring  │   network + disk SQ/CQ, shared-nothing
   │ owns a shard │ owns a shard │ owns a shard  │   of peer connections (SO_REUSEPORT)
   └──────┬───────┴──────┬───────┴───────┬───────┘
          │ submit jobs   │               │  job results returned via io_uring msg_ring / eventfd
   ┌──────┴───────────────┴───────────────┴───────┐
   │  Hash worker pool        Crypto-assist pool   │  (pinned to remaining cores,
   │  (SHA-1 / SHA-256 / Merkle, MPMC job queue)   │   pull jobs, post completions)
   └───────────────────────────────────────────────┘

Sharding model (the central decision):

• Connections are sharded across reactors. A SO_REUSEPORT listen socket per reactor lets the kernel hash inbound peers across cores; outbound peers are assigned by hash. Each reactor exclusively owns its connections → all per-connection state (recv/send buffers, MSE keystream, message parser) is lock-free.
• Torrent piece-state is shared (peers of one torrent live on many reactors). It is guarded by a per-torrent lock that protects only bookkeeping (the picker's rarity counts, request map, have-bitfield). The lock is held for microseconds; all heavy work (decrypt, hash, disk) happens outside it. Rationale: the expensive bytes never touch a lock; only the tiny "which block next / mark block received" decisions do.
• Heavy work is offloaded, not run on reactors: completed piece buffers go to the hash pool; crypto can run inline on the reactor (cheap per-message) or be batched to a crypto-assist pool under load. Results return to the owning reactor via io_uring msg_ring (cross-ring wakeup) or eventfd.
• Control plane is off the hot path entirely. RPC, plugins, and scripts run on their own thread(s) and communicate with reactors only via per-reactor MPSC command queues + event fan-out. A misbehaving plugin/script can never stall the data path.

Single-torrent scaling note: because crypto/hash/disk for one torrent fan out to all cores via the pools, even a lone large torrent uses the whole machine. Only the per-torrent picker lock is single-point; if it ever becomes hot, upgrade the picker to the lock-free variant (§6, optimization).

io_uring features exploited: multishot accept/recv, registered buffers (PROVIDE_BUFFERS ring) for zero-copy recv, registered files, SEND_ZC (zero-copy send) for seeding, SQPOLL (kernel-side submission polling to cut syscalls under load), msg_ring for cross-thread completions, linked SQEs for read→hash chaining. NUMA-aware + hugepage buffer pools per reactor.

---

3. Module / Layer Breakdown

Layers are bottom-up; each is independently unit-testable and has explicit extension seams. Suggested repo layout under src/.

Foundation

• platform/ — the only code that touches the kernel. io_uring lifecycle (ring setup, SQE build, CQE reaping), socket setup (SO_REUSEPORT, TCP_NODELAY, large SO_RCV/SNDBUF, TCP_FASTOPEN), file ops (O_DIRECT, fallocate, fadvise), eventfd/timerfd, CPU pinning, hugepage/NUMA allocation, monotonic clock. This is the seam that keeps "Linux-only now" from becoming "Linux-only forever."
• core/ — data-structure toolbox, zero dependencies:
  • Buffer pool: slab allocator of fixed-size (16 KiB block + piece-sized) page-aligned blocks, per-reactor freelists, refcounted so one buffer flows recv→decrypt→hash→disk without copy.
  • Intrusive doubly-linked lists, open-addressing hash maps, dynamic arrays, bitfields with popcount (have/interested/request maps), object pools, SPSC + MPSC + MPMC lock-free queues, per-thread lock-free logging ring, config parser, lock-free stats counters.
• crypto/ — SHA-1, SHA-256 with runtime SHA-NI dispatch (scalar fallback), Merkle tree builder/verifier (v2 piece layers), RC4 (MSE), Diffie-Hellman (MSE handshake, RFC 2631 768-bit group), CSPRNG. Designed for in-place operation on pooled buffers.

Protocol

• bencode/ — streaming, allocation-light parser producing zero-copy slices into the source buffer; encoder. Hardened against malicious input (depth/size limits) — primary fuzz target.
• metainfo/ — .torrent parse for v1, v2, and hybrid; file tree + piece layers; magnet URI parsing; info-hash computation (both v1 SHA-1 and v2 SHA-256 truncated).
• tracker/ — interface tracker_backend with built-in HTTP(S) (BEP-3/BEP-23 compact) and UDP (BEP-15) backends; announce scheduling, scrape, multi-tier (BEP-12). Extension seam: register custom backends.
• dht/ — Kademlia routing table, get_peers/announce_peer/find_node, bootstrap, token management, BEP-32 IPv6, BEP-51 infohash indexing. Runs as its own UDP endpoint on a reactor.
• peer/ — wire codec (handshake, all BEP-3 messages), MSE/PE layer (DH handshake, RC4 keystream, plaintext fallback), framing, and a BEP-10 extension registry (extension_handler vtable) with built-ins: PEX (BEP-11), ut_metadata (BEP-9), LTEP negotiation. Extension seam: register custom extension message handlers.

Engine

• piece/ — piece_picker interface (vtable) with built-in rarest-first, sequential/streaming, priority, and endgame strategies; per-peer adaptive request pipelining (depth auto-tuned from observed throughput × RTT, not a fixed window — essential for filling a 10 GbE BDP); block accounting; per-torrent lock holder. Extension seam: pluggable picker strategy.
• storage/ — storage_backend interface with built-in file backend: maps (piece,offset)→(file,offset) across multi-file torrents, O_DIRECT aligned read/write via the reactor's io_uring, write coalescing, bounded read cache, configurable fsync policy, fast-resume (persisted bitfield + partial-piece state), preallocation. Extension seam: register custom storage (memory, network, object-store).
• verify/ — hashing job dispatcher: full-piece SHA-1 (v1) and incremental Merkle leaf/branch hashing (v2) submitted to the hash pool; on completion marks piece valid/invalid on the owning reactor.
• scheduler/ — token-bucket rate limiting (global + per-torrent + per-peer), choking algorithm (tit-for-tat + optimistic unchoke, BEP-3), connection budget, bandwidth fairness across torrents, super-seeding (BEP-16) option.
• session/ — torrent lifecycle state machine, peer-set management, alert/event bus, aggregate stats. The single registry the control plane talks to.

Surface (the "extensible" layer)

• rpc/ — control protocol over a UNIX domain socket (and optional TCP): command/response + streaming event subscription. Versioned, length-prefixed binary frames (compact) with an optional JSON mode for tooling. A stable command registry so plugins can add RPC verbs. This is how UIs/automation drive the client.
• plugin/ — versioned C ABI: host passes a naut_host_api struct of function pointers; the plugin (.so) exports naut_plugin_register(host). Plugins hook the same vtables the core uses: tracker_backend, storage_backend, piece_picker, extension_handler, RPC commands, and the event bus. ABI version checked at load; plugins run on the control thread, never the data path.
• script/ — embedded Lua-style VM bound to the event bus (on_torrent_added, on_piece_complete, on_file_complete, on_torrent_finished, on_peer_connected, on_alert) and a sandboxed control API. Runs on a dedicated thread with a bounded work queue; cannot block reactors.
  • User feature — move files as they finish (libtorrent can't): the engine seam is already built (Phase 3): naut_download fires on_file_complete(file_index, path) the moment a file's last covering piece verifies, and naut_storage_relocate() moves a single completed file safely (even mid-download). Phase 7 forwards the event to the on_file_complete script hook and exposes a move_file API; the relocate must be marshalled onto the owning reactor thread via the command queue (scripts run on their own thread).

---

4. Concurrency & Memory Model (invariants)

• Data path is lock-free. A connection is touched by exactly one reactor. The only data-path lock is the per-torrent picker mutex, held only for O(1)–O(log n) bookkeeping.
• One buffer, one lifetime, zero copies. A pooled, refcounted, page-aligned buffer is filled by recv (registered buffer), decrypted in place, hashed in place by a worker, then written by SEND_ZC/O_DIRECT write — same physical pages throughout.
• Cross-thread communication is message passing, never shared mutable state: MPSC command queues into reactors, MPMC job queue into the hash pool, msg_ring/eventfd for completions back out.
• No allocation on the hot path. All buffers, connection objects, request objects come from preallocated per-reactor pools sized from peers × pipeline_depth × block_size. Allocator is only touched at torrent add/remove.
• Backpressure is explicit. Bounded queues everywhere; when the hash pool or disk falls behind, reactors stop issuing recv (flow control) rather than growing memory unboundedly.

---

5. Build, Repo & Quality

Naut-Torrent/
├── src/{platform,core,crypto,bencode,metainfo,tracker,dht,peer,piece,storage,verify,scheduler,session,rpc,plugin,script}/
├── include/naut/         # public headers incl. versioned plugin ABI (naut_plugin.h)
├── apps/{nautd,nautctl}/  # daemon + CLI client over RPC
├── plugins/example/       # reference plugin against the ABI
├── tests/{unit,fuzz,integration,bench}/
└── build/                 # build-system outputs

• Daemon/CLI split: nautd (engine) + nautctl (thin RPC client). UIs are just RPC consumers — keeps the core headless and embeddable.
• Tooling: AddressSanitizer/UBSan/ThreadSanitizer builds; perf/eBPF-friendly; built-in per-core stats exported over RPC.

---

6. Implementation Phases (milestones, each independently demoable)

1. Foundation — platform/ io_uring echo server + core/ buffer pool, queues, bitfields, hash map. Gate: loopback echo saturates a core with zero per-op allocation.
2. Crypto + parsers — SHA-1/256 (+SHA-NI), Merkle, RC4, DH; bencode/, metainfo/ (v1/v2/hybrid + magnet). Gate: parse real torrents; hash throughput benchmark ≥1.5 GB/s/core.
3. Single-peer transfer — peer/ handshake + messages (plaintext), piece/ basic picker, storage/ file backend, verify/. Download a real torrent from one peer to disk, verified. Gate: byte-correct file from a known seed.
4. Trackers + swarm — HTTP/UDP tracker/, choking scheduler/, multi-peer, rarest-first, endgame. Gate: download from a public/local swarm; interop with libtorrent/Transmission.
5. MSE + DHT + extensions — RC4 MSE handshake, dht/, PEX, ut_metadata, magnet-only start. Gate: magnet link with no trackers completes via DHT; encrypted peers work.
6. Scale to 10 GbE — adaptive pipelining, SEND_ZC, registered buffers, SQPOLL, hash/crypto pools, NUMA/hugepages, O_DIRECT tuning. Gate: two boxes (or two NICs) sustain ≥9.4 Gbit/s both directions. Optional: lock-free picker if the per-torrent lock shows contention in perf.
7. Extensibility surface — rpc/, plugin/ ABI + reference plugin, script/ VM + event hooks, nautctl. Gate: a sample plugin adds a storage backend and a script reacts to on_torrent_finished.

---

7. Verification (end-to-end)

• Microbenchmarks (tests/bench/): hash GB/s/core, RC4 GB/s/core, bencode parse MB/s, buffer-pool alloc/free ns, picker decisions/s. Each has a regression threshold tied to the §1 budget.
• Correctness: unit tests per module; fuzzers (tests/fuzz/) on bencode, peer-message, tracker-response, and metainfo parsers (the attack surface); byte-for-byte file verification after download; resume-from-partial test.
• Interop: run against libtorrent/qBittorrent and Transmission as both seed and leech, plaintext and MSE; magnet + DHT-only bootstrap test against the public DHT.
• Throughput (the headline test):
  1. Baseline the link with iperf3 to confirm ~9.4 Gbit/s is achievable end-to-end.
  2. Seed a large (≥50 GB) torrent from box A, download on box B over the 10 GbE link (or two NICs on one box via loopback-to-NIC). Measure with the client's own per-core RPC stats + nstat/ifstat.
  3. Confirm sustained throughput is link-bound (not CPU/disk/lock-bound) via perf top — no single thread pinned at 100% on locks/copies, hash & crypto spread across pool cores.
  4. Reverse roles to validate upload saturation (SEND_ZC path).
• Soak: 24 h multi-torrent run under ASan-off release build; assert flat memory (pools, no leaks), no descriptor growth, stable throughput.

---

8. Key Risks & Mitigations

• Per-torrent picker lock contention at 10 GbE → keep heavy work outside the lock; escalate to lock-free claim-by-CAS picker (already an interface, so it's a swap not a rewrite).
• RC4/MSE CPU cost underestimated → crypto-assist pool + prefer plaintext when both peers allow; measure early in Phase 5.
• io_uring portability lock-in → all kernel calls behind platform/; a future epoll backend is a new file, not a refactor.
• O_DIRECT alignment complexity → enforce page-aligned pooled buffers from day one (Phase 1), so storage never has to bounce-buffer.
• Hybrid v1/v2 data-model complexity → model the piece/file/Merkle layout once in metainfo/ + storage/ and treat v1 as the degenerate single-layer case.