Naut/README.md

# Naut-Torrent

A maintainable, extensible BitTorrent client engineered to **saturate a 10 GbE
link (~1.25 GB/s) in both directions** on Linux. Written in C11.

- **Platform:** Linux-only, built hard around **io_uring** (network *and* disk).
- **Protocol:** BitTorrent **v1 + v2 hybrid** (SHA-1 pieces + SHA-256 Merkle).
- **Workload:** saturate download *and* upload, with **MSE/PE encryption** in the hot path.
- **Extensible:** clean module boundaries + control **RPC** + **BEP-10** extensions
  + a versioned native **plugin ABI** + embedded **scripting**.

The full architecture rationale lives in [`plan.md`](plan.md). The short version: a
**shared-nothing, thread-per-core reactor** model with **offload pools** for
hashing/crypto, and a **one-buffer-zero-copy** data path so bytes flow
recv → decrypt → hash → disk/send without a single `memcpy`.

## Layout

```
include/naut/   public headers
src/core/       zero-dependency foundation (buffers, queues, bitfields, log)
src/platform/   the ONLY code that touches the kernel (io_uring, sockets)
src/dht/        BEP-5 KRPC codec + bounded iterative peer discovery
src/peer/       wire protocol, MSE/RC4, BEP-10, ut_metadata, and PEX
apps/echo/      Phase 1 gate: io_uring echo server on the buffer pool
apps/leech/     Phase 3 gate: verified single-peer download
apps/swarm/     tracker/DHT discovery, magnets, and concurrent peers
tests/unit/     unit + concurrency tests
```

## Build, test, run

```sh
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
ninja -C build
ctest --test-dir build --output-on-failure

# sanitizer build (address|thread|undefined)
cmake -S . -B build-tsan -G Ninja -DCMAKE_BUILD_TYPE=Debug -DNAUT_SAN=thread
ninja -C build-tsan && ./build-tsan/test_buf

# run the Phase 1 echo gate
./build/naut_echo 9000

# download from explicit peers, or omit them to use the torrent's trackers
./build/naut_swarm file.torrent output/ 192.0.2.10:6881 192.0.2.11:6881
./build/naut_swarm file.torrent output/

# trackerless magnet start through DHT (override bootstraps when needed)
./build/naut_swarm 'magnet:?xt=urn:btih:...' output/
NAUT_DHT_BOOTSTRAP=127.0.0.1:6881 ./build/naut_swarm 'magnet:?xt=urn:btih:...' output/

# force an encrypted single-peer MSE/RC4 connection
./build/naut_leech --mse file.torrent output/ 192.0.2.10 6881

# Phase 6 CPU and reactor benchmarks
./build/bench_hash
./build/bench_scale 8 8
bash tests/integration/run_echo_scale.sh ./build/naut_echo

# optional data-path tuning
NAUT_DIRECT_IO=1 NAUT_WORKERS=8 ./build/naut_swarm file.torrent output/
NAUT_CPU=2 NAUT_SQPOLL=1 NAUT_HUGEPAGES=1 NAUT_NUMA_NODE=0 ./build/naut_echo 9000
```

Requirements: Linux ≥ 6.0, `liburing` (≥ 2.x), OpenSSL `libcrypto`, Jansson,
Lua, CMake ≥ 3.20, gcc/clang, Ninja.

## Daemon, RPC, plugins, and scripts

Phase 7 adds a headless control process and thin CLI over a versioned,
length-prefixed JSON protocol on a Unix socket:

```sh
./build/nautd \
  --socket /tmp/nautd.sock \
  --plugin ./build/naut_example.so \
  --script ./tests/fixtures/phase7.lua

./build/nautctl ping
./build/nautctl plugins
./build/nautctl status
./build/nautctl events
```

`nautctl` accepts an optional JSON value after the method:

```sh
# register a torrent's storage so a move command can resolve + relocate its files
./build/nautctl add_torrent \
  '{"torrent_id":7,"torrent":"file.torrent","root":"output/"}'
./build/nautctl emit \
  '{"type":"torrent_finished","torrent_id":7}'
./build/nautctl shutdown
```

The native ABI is declared in `include/naut/naut_plugin.h`. Plugins export
`naut_plugin_register()`, receive the versioned host API, and may register RPC
methods, storage backends, and event handlers. `plugins/example/example.c`
provides the reference in-memory storage backend.

Lua scripts run on a dedicated thread behind a bounded event queue. Supported
hooks are `on_torrent_added`, `on_piece_complete`, `on_file_complete`,
`on_torrent_finished`, `on_peer_connected`, and `on_alert`. The sandbox removes
filesystem, process, package-loading, debug, and raw chunk-loading globals
(`os`, `io`, `package`/`require`, `debug`, `dofile`/`loadfile`, and
`load`/`loadstring` — the bytecode loaders are denied so a crafted binary chunk
can't escape the VM). `naut.move_file()` submits a bounded command from the
script thread to the daemon owner thread; the owner resolves it through the
torrent registry (`naut_session`) and performs the relocate with
`naut_storage_relocate()`. Register a torrent's storage first with the
`add_torrent` RPC so the id resolves. `phase7_extensibility` drives this
end to end and asserts the file actually moves on disk.

The full script-visible surface — every event hook, the `event` object's
fields, and the `naut` API table — is documented in
[`docs/scripting.md`](docs/scripting.md).

## Roadmap (status)

| Phase | Scope | State |
|------|-------|-------|
| 1 | Foundation: `platform/` io_uring + `core/` buffer pool/queues/bitfields | **done — built, tested, TSan-clean; echo ~30 Gbit/s on one core** |
| 2 | `crypto/` (SHA-1/256 + SHA-NI, Merkle, RC4) · `bencode/` · `metainfo/` (v1/v2/hybrid + magnet) | **done — FIPS/RC4 vectors pass, SHA-256 2.27 GB/s/core, info-hashes verified vs libtorrent, parsers fuzz-clean (3M iters, ASan+UBSan)** |
| 3 | Single-peer transfer: `peer/` · `piece/` · `storage/` · `verify/` | **done — `naut_leech` downloads single/multi/hybrid from a libtorrent seed, byte-identical + SHA-1 verified (`interop_leech` test)** |
| 4 | Trackers + swarm: HTTP/UDP trackers, choking, rarest-first, endgame | **done — `naut_swarm` discovers peers or accepts explicit endpoints; two-peer and live HTTP/UDP tracker interop gates pass against libtorrent** |
| 5 | MSE encryption · DHT · PEX · ut_metadata · magnet-only start | **done — forced RC4 interop passes against libtorrent; trackerless magnet gate discovers a peer through DHT, verifies BEP-9 metadata, and completes byte-identically** |
| 6 | Scale to 10 GbE: adaptive pipelining, SEND_ZC, registered bufs, SQPOLL, NUMA | **implementation complete; local 8-connection echo gate measured 15.11 Gbit/s with byte verification, while the plan's two-machine ≥9.4 Gbit/s NIC gate remains external hardware validation** |
| 7 | Extensibility surface: RPC · plugin ABI · scripting · `nautctl` | **done — versioned Unix RPC with event streaming, reference storage plugin, sandboxed Lua hooks, bounded move command marshalling, and an end-to-end integration gate** |

## Foundation design notes

- **`naut_buf` pool** (`src/core/buf.c`): page-aligned, refcounted blocks from
  one mmap'd slab. `get()` is single-consumer (owning reactor); `put()`/`ref()`
  are multi-producer. Because only the owner pops, the Treiber-stack freelist is
  ABA-free without tagging. The whole slab can be registered with io_uring as a
  fixed-buffer region.
- **`naut_mpmc`** (`src/core/mpmc.c`): Vyukov bounded queue — one implementation
  serves every cross-thread hand-off (control→reactor, reactor→hash-pool, back).
- **`naut_bitfield`** (`src/core/bitfield.c`): popcount/ctz-based; includes the
  BEP-3 MSB-first wire conversion that the peer protocol needs.
- **`platform/`** wraps every syscall so "Linux-only now" stays "portable later":
  a future epoll backend is a new file, not a refactor.
- **`crypto/`**: SHA-256 picks a SHA-NI or scalar backend at startup
  (`NAUT_NO_SHANI=1` forces scalar); both are cross-checked against FIPS vectors
  in CI. RC4 carries the MSE 1024-byte keystream drop. Merkle implements BEP-52
  zero-hash padding. *Deferred by design:* MSE Diffie-Hellman lands in Phase 5
  (where it's used); a SHA-NI SHA-1 path is a Phase 6 optimization for
  v1-heavy swarms (scalar SHA-1 is ~0.27 GB/s, fine behind the hash pool).
- **`metainfo/`**: info-hashes are computed over the raw `info` bytes and were
  verified against libtorrent for v1, v2, and hybrid. Fixtures are regenerated
  with `python3 tests/fixtures/generate.py` (needs python `libtorrent`).
- **Fuzzing**: `fuzz_lite <bencode|metainfo> <iters> [seeds...]` is a mutational
  fuzzer; build with `-DNAUT_SAN=address` and seed from `tests/fixtures/*.torrent`.
- **`peer/`**: a *sans-IO* wire codec — pure functions over byte buffers, no
  sockets — so the same code is driven by the blocking `naut_leech` now and the
  io_uring reactor in Phase 6. `naut_leech <file.torrent> <dir> <ip> <port>`
  downloads from one peer; a piece is assembled in RAM, SHA-1 verified, then
  written, so a corrupt piece never reaches disk.
- **`tracker/` + `naut_swarm`**: BEP-3 compact HTTP and BEP-15 UDP announces
  feed a deduplicated peer set. The swarm driver tracks HAVE/BITFIELD
  availability, respects choke/unchoke, expires stalled requests, schedules
  rarest-first, and bounds endgame races to two distinct peers per block.
  Redundant requests are canceled as soon as one copy arrives. `interop_swarm`
  requires two independent libtorrent seeds to contribute, while
  `interop_tracker_swarm` and `interop_udp_tracker_swarm` prove live discovery.
- **Phase 5 peer discovery and transport**: outgoing MSE is a *sans-IO* state
  machine (`naut_mse_handshake_*`) — feed bytes, pull bytes, no sockets — so the
  io_uring reactor can drive an encrypted handshake without blocking a core; the
  blocking `naut_mse_client_handshake` is a thin wrapper over it. It performs the
  768-bit Diffie-Hellman exchange, offers RC4-only PE, drops the first 1024
  keystream bytes, and keeps independent connection-owned send/receive states.
  `naut_dht` builds and validates BEP-5 KRPC and performs a bounded iterative
  IPv4 `get_peers` lookup. BEP-10 handshakes advertise `ut_metadata` and PEX;
  metadata is assembled in 16 KiB blocks and rejected unless its raw SHA-1
  matches the magnet's `btih`. `interop_mse` and `interop_magnet_dht` are the
  deterministic local gates for both required Phase 5 outcomes.
- **Phase 6 scaling path**: each swarm peer adjusts its request window from
  smoothed throughput × RTT rather than a fixed depth. Completed pieces submit
  SHA-1 jobs to a bounded MPMC worker pool and return to the owner through
  eventfd; storage writes and callbacks remain owner-thread operations.
  Piece buffers are page-aligned, and `NAUT_DIRECT_IO=1` uses O_DIRECT for
  aligned bulk regions with buffered edge fallback. The io_uring seam supports
  SQPOLL CPU affinity, registered slabs, fixed-buffer receive where the kernel
  accepts it, and SEND_ZC with notification-lifetime tracking. Two
  compatibility notes baked in: `IORING_SETUP_SQPOLL` is mutually exclusive with
  `COOP_TASKRUN` (the kernel `-EINVAL`s the combo), so the ring pairs SQPOLL with
  `SINGLE_ISSUER` only; and a kernel that rejects `IORING_RECVSEND_FIXED_BUF` on
  plain recv is detected per-operation and every affected connection retries
  unfixed (not just the first), so fixed-buffer fallback never tears a peer down.
  `run_echo_scale.sh` asserts the server echoed *every* byte across all
  connections, turning any such drop into a hard failure. If a transport
  repeatedly reports copied SEND_ZC operations (as loopback does), the ring
  degrades to normal sends instead of paying useless notification overhead.
  Hugepage and NUMA slab placement are controlled by `NAUT_HUGEPAGES` and
  `NAUT_NUMA_NODE`; worker count/affinity use `NAUT_WORKERS` and
  `NAUT_WORKER_CPU_BASE`.
- **Measured Phase 6 CPU budget on this host** (8 workers): SHA-1 1.62 GB/s,
  SHA-256 14.75 GB/s, and RC4 3.21 GB/s. The single-core SHA-256 gate measured
  1.87 GB/s. These clear the 1.25 GB/s per-direction processing budget, but do
  not replace the physical 10 GbE two-host test required by `plan.md`.
- **`storage/`** maps the torrent's flat byte space across files (a write may
  straddle a file boundary) and recognises BEP-47 padding files in hybrid
  torrents, routing them out of the content tree. Interop is proven against
  libtorrent via `tests/integration/run_interop.sh` (also run by ctest).
- **Per-file completion / move-as-you-go** (a headline feature): `naut_download`
  fires `on_file_complete(file_index, path)` the instant a file's last covering
  piece verifies — before the torrent finishes — and `naut_storage_relocate()`
  moves that file out safely (even mid-download, while other files' pieces are
  still arriving). `test_filemove` proves a file is relocated mid-download with
  no corruption. The scripting layer (Phase 7) forwards the event to an
  `on_file_complete` hook and exposes `move_file`; the daemon resolves the
  command through the `naut_session` torrent registry (`src/session/session.c`)
  and calls `naut_storage_relocate()` on its owner thread. `phase7_extensibility`
  exercises the whole chain — script thread → bounded queue → owner thread →
  storage — and asserts the file moves on disk.