Naut/README.md

Naut-Torrent

A maintainable, extensible BitTorrent client engineered to saturate a 10 GbE link (~1.25 GB/s) in both directions on Linux. Written in C11.

• Platform: Linux-only, built hard around io_uring (network and disk).
• Protocol: BitTorrent v1 + v2 hybrid (SHA-1 pieces + SHA-256 Merkle).
• Workload: saturate download and upload, with MSE/PE encryption in the hot path.
• Extensible: clean module boundaries + control RPC + BEP-10 extensions
  • a versioned native plugin ABI + embedded scripting.

The full architecture rationale lives in plan.md. The short version: a shared-nothing, thread-per-core reactor model with offload pools for hashing/crypto, and a one-buffer-zero-copy data path so bytes flow recv → decrypt → hash → disk/send without a single memcpy.

Layout

include/naut/   public headers
src/core/       zero-dependency foundation (buffers, queues, bitfields, log)
src/platform/   the ONLY code that touches the kernel (io_uring, sockets)
src/dht/        BEP-5 KRPC codec + bounded iterative peer discovery
src/peer/       wire protocol, MSE/RC4, BEP-10, ut_metadata, and PEX
apps/echo/      Phase 1 gate: io_uring echo server on the buffer pool
apps/leech/     Phase 3 gate: verified single-peer download
apps/swarm/     tracker/DHT discovery, magnets, and concurrent peers
tests/unit/     unit + concurrency tests

Build, test, run

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
ninja -C build
ctest --test-dir build --output-on-failure

# sanitizer build (address|thread|undefined)
cmake -S . -B build-tsan -G Ninja -DCMAKE_BUILD_TYPE=Debug -DNAUT_SAN=thread
ninja -C build-tsan && ./build-tsan/test_buf

# run the Phase 1 echo gate
./build/naut_echo 9000

# download from explicit peers, or omit them to use the torrent's trackers
./build/naut_swarm file.torrent output/ 192.0.2.10:6881 192.0.2.11:6881
./build/naut_swarm file.torrent output/

# trackerless magnet start through DHT (override bootstraps when needed)
./build/naut_swarm 'magnet:?xt=urn:btih:...' output/
NAUT_DHT_BOOTSTRAP=127.0.0.1:6881 ./build/naut_swarm 'magnet:?xt=urn:btih:...' output/

# force an encrypted single-peer MSE/RC4 connection
./build/naut_leech --mse file.torrent output/ 192.0.2.10 6881

# Phase 6 CPU and reactor benchmarks
./build/bench_hash
./build/bench_scale 8 8
bash tests/integration/run_echo_scale.sh ./build/naut_echo

# optional data-path tuning
NAUT_DIRECT_IO=1 NAUT_WORKERS=8 ./build/naut_swarm file.torrent output/
NAUT_CPU=2 NAUT_SQPOLL=1 NAUT_HUGEPAGES=1 NAUT_NUMA_NODE=0 ./build/naut_echo 9000

Requirements: Linux ≥ 6.0, liburing (≥ 2.x), OpenSSL libcrypto, Jansson, Lua, CMake ≥ 3.20, gcc/clang, Ninja.

Daemon, RPC, plugins, and scripts

Phase 7 adds a headless control process and thin CLI over a versioned, length-prefixed JSON protocol on a Unix socket:

./build/nautd \
  --socket /tmp/nautd.sock \
  --plugin ./build/naut_example.so \
  --script ./tests/fixtures/phase7.lua

./build/nautctl ping
./build/nautctl plugins
./build/nautctl status
./build/nautctl events

nautctl accepts an optional JSON value after the method:

# register a torrent's storage so a move command can resolve + relocate its files
./build/nautctl add_torrent \
  '{"torrent_id":7,"torrent":"file.torrent","root":"output/"}'
./build/nautctl emit \
  '{"type":"torrent_finished","torrent_id":7}'
./build/nautctl shutdown

The native ABI is declared in include/naut/naut_plugin.h. Plugins export naut_plugin_register(), receive the versioned host API, and may register RPC methods, storage backends, and event handlers. plugins/example/example.c provides the reference in-memory storage backend.

Lua scripts run on a dedicated thread behind a bounded event queue. Supported hooks are on_torrent_added, on_piece_complete, on_file_complete, on_torrent_finished, on_peer_connected, and on_alert. The sandbox removes filesystem, process, package-loading, debug, and raw chunk-loading globals (os, io, package/require, debug, dofile/loadfile, and load/loadstring — the bytecode loaders are denied so a crafted binary chunk can't escape the VM). naut.move_file() submits a bounded command from the script thread to the daemon owner thread; the owner resolves it through the torrent registry (naut_session) and performs the relocate with naut_storage_relocate(). Register a torrent's storage first with the add_torrent RPC so the id resolves. phase7_extensibility drives this end to end and asserts the file actually moves on disk.

The full script-visible surface — every event hook, the event object's fields, and the naut API table — is documented in docs/scripting.md. A complete worked example, examples/anime_sort.lua, sorts anime into a Title/Season NN/Title - SNNENN.ext library as each episode finishes, using an embedded Anitomy-style filename parser (examples/).

Roadmap (status)

Phase	Scope	State
1	Foundation: platform/ io_uring + core/ buffer pool/queues/bitfields	done — built, tested, TSan-clean; echo ~30 Gbit/s on one core
2	crypto/ (SHA-1/256 + SHA-NI, Merkle, RC4) · bencode/ · metainfo/ (v1/v2/hybrid + magnet)	done — FIPS/RC4 vectors pass, SHA-256 2.27 GB/s/core, info-hashes verified vs libtorrent, parsers fuzz-clean (3M iters, ASan+UBSan)
3	Single-peer transfer: peer/ · piece/ · storage/ · verify/	done — naut_leech downloads single/multi/hybrid from a libtorrent seed, byte-identical + SHA-1 verified (interop_leech test)
4	Trackers + swarm: HTTP/UDP trackers, choking, rarest-first, endgame	done — naut_swarm discovers peers or accepts explicit endpoints; two-peer and live HTTP/UDP tracker interop gates pass against libtorrent
5	MSE encryption · DHT · PEX · ut_metadata · magnet-only start	done — forced RC4 interop passes against libtorrent; trackerless magnet gate discovers a peer through DHT, verifies BEP-9 metadata, and completes byte-identically
6	Scale to 10 GbE: adaptive pipelining, SEND_ZC, registered bufs, SQPOLL, NUMA	implementation complete; local 8-connection echo gate measured 15.11 Gbit/s with byte verification, while the plan's two-machine ≥9.4 Gbit/s NIC gate remains external hardware validation
7	Extensibility surface: RPC · plugin ABI · scripting · nautctl	done — versioned Unix RPC with event streaming, reference storage plugin, sandboxed Lua hooks, bounded move command marshalling, and an end-to-end integration gate

Foundation design notes

• naut_buf pool (src/core/buf.c): page-aligned, refcounted blocks from one mmap'd slab. get() is single-consumer (owning reactor); put()/ref() are multi-producer. Because only the owner pops, the Treiber-stack freelist is ABA-free without tagging. The whole slab can be registered with io_uring as a fixed-buffer region.
• naut_mpmc (src/core/mpmc.c): Vyukov bounded queue — one implementation serves every cross-thread hand-off (control→reactor, reactor→hash-pool, back).
• naut_bitfield (src/core/bitfield.c): popcount/ctz-based; includes the BEP-3 MSB-first wire conversion that the peer protocol needs.
• platform/ wraps every syscall so "Linux-only now" stays "portable later": a future epoll backend is a new file, not a refactor.
• crypto/: SHA-256 picks a SHA-NI or scalar backend at startup (NAUT_NO_SHANI=1 forces scalar); both are cross-checked against FIPS vectors in CI. RC4 carries the MSE 1024-byte keystream drop. Merkle implements BEP-52 zero-hash padding. Deferred by design: MSE Diffie-Hellman lands in Phase 5 (where it's used); a SHA-NI SHA-1 path is a Phase 6 optimization for v1-heavy swarms (scalar SHA-1 is ~0.27 GB/s, fine behind the hash pool).
• metainfo/: info-hashes are computed over the raw info bytes and were verified against libtorrent for v1, v2, and hybrid. Fixtures are regenerated with python3 tests/fixtures/generate.py (needs python libtorrent).
• Fuzzing: fuzz_lite <bencode|metainfo> <iters> [seeds...] is a mutational fuzzer; build with -DNAUT_SAN=address and seed from tests/fixtures/*.torrent.
• peer/: a sans-IO wire codec — pure functions over byte buffers, no sockets — so the same code is driven by the blocking naut_leech now and the io_uring reactor in Phase 6. naut_leech <file.torrent> <dir> <ip> <port> downloads from one peer; a piece is assembled in RAM, SHA-1 verified, then written, so a corrupt piece never reaches disk.
• tracker/ + naut_swarm: BEP-3 compact HTTP and BEP-15 UDP announces feed a deduplicated peer set. The swarm driver tracks HAVE/BITFIELD availability, respects choke/unchoke, expires stalled requests, schedules rarest-first, and bounds endgame races to two distinct peers per block. Redundant requests are canceled as soon as one copy arrives. interop_swarm requires two independent libtorrent seeds to contribute, while interop_tracker_swarm and interop_udp_tracker_swarm prove live discovery.
• Phase 5 peer discovery and transport: outgoing MSE is a sans-IO state machine (naut_mse_handshake_*) — feed bytes, pull bytes, no sockets — so the io_uring reactor can drive an encrypted handshake without blocking a core; the blocking naut_mse_client_handshake is a thin wrapper over it. It performs the 768-bit Diffie-Hellman exchange, offers RC4-only PE, drops the first 1024 keystream bytes, and keeps independent connection-owned send/receive states. naut_dht builds and validates BEP-5 KRPC and performs a bounded iterative IPv4 get_peers lookup. BEP-10 handshakes advertise ut_metadata and PEX; metadata is assembled in 16 KiB blocks and rejected unless its raw SHA-1 matches the magnet's btih. interop_mse and interop_magnet_dht are the deterministic local gates for both required Phase 5 outcomes.
• Phase 6 scaling path: each swarm peer adjusts its request window from smoothed throughput × RTT rather than a fixed depth. Completed pieces submit SHA-1 jobs to a bounded MPMC worker pool and return to the owner through eventfd; storage writes and callbacks remain owner-thread operations. Piece buffers are page-aligned, and NAUT_DIRECT_IO=1 uses O_DIRECT for aligned bulk regions with buffered edge fallback. The io_uring seam supports SQPOLL CPU affinity, registered slabs, fixed-buffer receive where the kernel accepts it, and SEND_ZC with notification-lifetime tracking. Two compatibility notes baked in: IORING_SETUP_SQPOLL is mutually exclusive with COOP_TASKRUN (the kernel -EINVALs the combo), so the ring pairs SQPOLL with SINGLE_ISSUER only; and a kernel that rejects IORING_RECVSEND_FIXED_BUF on plain recv is detected per-operation and every affected connection retries unfixed (not just the first), so fixed-buffer fallback never tears a peer down. run_echo_scale.sh asserts the server echoed every byte across all connections, turning any such drop into a hard failure. If a transport repeatedly reports copied SEND_ZC operations (as loopback does), the ring degrades to normal sends instead of paying useless notification overhead. Hugepage and NUMA slab placement are controlled by NAUT_HUGEPAGES and NAUT_NUMA_NODE; worker count/affinity use NAUT_WORKERS and NAUT_WORKER_CPU_BASE.
• Measured Phase 6 CPU budget on this host (8 workers): SHA-1 1.62 GB/s, SHA-256 14.75 GB/s, and RC4 3.21 GB/s. The single-core SHA-256 gate measured 1.87 GB/s. These clear the 1.25 GB/s per-direction processing budget, but do not replace the physical 10 GbE two-host test required by plan.md.
• storage/ maps the torrent's flat byte space across files (a write may straddle a file boundary) and recognises BEP-47 padding files in hybrid torrents, routing them out of the content tree. Interop is proven against libtorrent via tests/integration/run_interop.sh (also run by ctest).
• Per-file completion / move-as-you-go (a headline feature): naut_download fires on_file_complete(file_index, path) the instant a file's last covering piece verifies — before the torrent finishes — and naut_storage_relocate() moves that file out safely (even mid-download, while other files' pieces are still arriving). test_filemove proves a file is relocated mid-download with no corruption. The scripting layer (Phase 7) forwards the event to an on_file_complete hook and exposes move_file; the daemon resolves the command through the naut_session torrent registry (src/session/session.c) and calls naut_storage_relocate() on its owner thread. phase7_extensibility exercises the whole chain — script thread → bounded queue → owner thread → storage — and asserts the file moves on disk.