Cieric/Naut
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions
Naut/README.md
ookami125 bc1eb5a08e build: vendor dependencies as submodules
2026-06-24 23:51:21 -04:00

15 KiB
Raw Permalink Blame History

Naut-Torrent

A maintainable, extensible BitTorrent client engineered to saturate a 10 GbE link (~1.25 GB/s) in both directions on Linux. Written in C11.

  • Platform: Linux-only, built hard around io_uring (network and disk).
  • Protocol: BitTorrent v1 + v2 hybrid (SHA-1 pieces + SHA-256 Merkle).
  • Workload: saturate download and upload, with MSE/PE encryption in the hot path.
  • Extensible: clean module boundaries + control RPC + BEP-10 extensions
    • a versioned native plugin ABI + embedded scripting.

The full architecture rationale lives in plan.md. The short version: a shared-nothing, thread-per-core reactor model with offload pools for hashing/crypto, and a one-buffer-zero-copy data path so bytes flow recv → decrypt → hash → disk/send without a single memcpy.

Layout

include/naut/   public headers
src/core/       zero-dependency foundation (buffers, queues, bitfields, log)
src/platform/   the ONLY code that touches the kernel (io_uring, sockets)
src/dht/        BEP-5 KRPC codec + bounded iterative peer discovery
src/peer/       wire protocol, MSE/RC4, BEP-10, ut_metadata, and PEX
apps/echo/      Phase 1 gate: io_uring echo server on the buffer pool
apps/leech/     Phase 3 gate: verified single-peer download
apps/swarm/     tracker/DHT discovery, magnets, and concurrent peers
apps/nautctl/   thin CLI frontend over daemon RPC
plugins/webui/  daemon plugin that serves web/torrent-ui as the web panel
tests/unit/     unit + concurrency tests

Build, test, run

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
ninja -C build
ctest --test-dir build --output-on-failure

# portable daemon/client build with Jansson and Lua embedded
cmake -S . -B build-standalone -G Ninja \
  -DCMAKE_BUILD_TYPE=Release -DNAUT_STANDALONE=ON
ninja -C build-standalone nautd nautctl
ldd build-standalone/nautd
ldd build-standalone/nautctl

# sanitizer build (address|thread|undefined)
cmake -S . -B build-tsan -G Ninja -DCMAKE_BUILD_TYPE=Debug -DNAUT_SAN=thread
ninja -C build-tsan && ./build-tsan/test_buf

# run the Phase 1 echo gate
./build/naut_echo 9000

# start the engine, then add and inspect downloads through its RPC frontend
./build/nautd
./build/nautctl add file.torrent output/
./build/nautctl list
./build/nautctl show 1
./build/nautctl events

# explicit peers and trackerless magnets use the same daemon workflow
./build/nautctl add file.torrent output/ 192.0.2.10:6881
./build/nautctl add 'magnet:?xt=urn:btih:...' output/

# force an encrypted single-peer MSE/RC4 connection
./build/naut_leech --mse file.torrent output/ 192.0.2.10 6881

# Phase 6 CPU and reactor benchmarks
./build/bench_hash
./build/bench_scale 8 8
bash tests/integration/run_echo_scale.sh ./build/naut_echo

# optional data-path tuning
NAUT_DIRECT_IO=1 NAUT_WORKERS=8 ./build/nautd
NAUT_CPU=2 NAUT_SQPOLL=1 NAUT_HUGEPAGES=1 NAUT_NUMA_NODE=0 ./build/naut_echo 9000

Requirements: Linux ≥ 6.0, liburing (≥ 2.x), OpenSSL libcrypto, Jansson, Lua, CMake ≥ 3.20, gcc/clang, Ninja.

NAUT_STANDALONE=ON downloads hash-pinned Jansson 2.14.1 and Lua 5.4.8 sources at configure time and statically embeds them in nautd and nautctl. The resulting executables still use the host's glibc/ELF loader intentionally: fully static glibc breaks normal DNS/NSS behavior and native .so plugins. Release builds target a portable CPU baseline. Use -DNAUT_NATIVE=ON only for a local build that will run on the same CPU family as the build machine.

Daemon, RPC, plugins, and scripts

nautd is the application engine: it owns torrent workers, storage, scripts, plugins, progress, and lifecycle. nautctl is one thin frontend over a versioned, length-prefixed JSON protocol on a Unix socket; a desktop or web panel can use the same RPC surface.

./build/nautd \
  --socket /tmp/nautd.sock \
  --plugin ./build/naut_example.so

./build/nautctl ping
./build/nautctl plugins
./build/nautctl status
./build/nautctl script ./examples/anime_sort.lua
./build/nautctl add show.torrent /downloads/show
./build/nautctl list
./build/nautctl events

The convenience commands cover normal operation:

./build/nautctl add file.torrent output/ [IP:PORT ...]
./build/nautctl list
./build/nautctl show 1
./build/nautctl remove 1
./build/nautctl script rules.lua
./build/nautctl unscript
./build/nautctl shutdown

For tooling and plugin methods, the generic form remains nautctl METHOD [PARAMS_JSON].

Web panel

The web panel is a daemon plugin, not part of nautctl. It serves the static frontend from web/torrent-ui/public by default and adapts that UI's /api/* contract to Naut's daemon RPC surface:

NAUT_WEBUI_ROOT=web/torrent-ui/public \
NAUT_AUTH_PASSWORD='change-me' \
./build/nautd --socket /tmp/nautd.sock --plugin ./build/naut_webui.so
# open http://127.0.0.1:8080

Configuration:

NAUT_WEBUI_HOST=127.0.0.1       # default
NAUT_WEBUI_PORT=8080            # default
NAUT_WEBUI_ROOT=web/torrent-ui/public
NAUT_AUTH_USER=admin            # default
NAUT_AUTH_PASSWORD=change-me    # generated and logged if omitted
NAUT_WEBUI_SAVE_PATH=/downloads # default add-torrent destination

The plugin implements the stable torrent-ui API surface: cookie login, /api/snapshot, /api/stream Server-Sent Events, /api/meta, torrent detail tabs, add/remove, and /api/plugins loading ES modules from public/plugins/plugins.json. Some advanced qBittorrent-style controls in the UI are accepted as no-ops until Naut grows matching daemon RPC methods.

The native ABI is declared in include/naut/naut_plugin.h. Plugins export naut_plugin_register(), receive the versioned host API, and may register RPC methods, storage backends, and event handlers. plugins/example/example.c provides the reference in-memory storage backend.

Lua scripts run on a dedicated thread behind a bounded event queue. Supported hooks are on_torrent_added, on_piece_complete, on_file_complete, on_torrent_finished, on_peer_connected, and on_alert. The sandbox removes filesystem, process, package-loading, debug, and raw chunk-loading globals (os, io, package/require, debug, dofile/loadfile, and load/loadstring — the bytecode loaders are denied so a crafted binary chunk can't escape the VM). naut.move_file() submits a bounded command to the worker that owns the torrent. That worker performs naut_storage_relocate() and keeps tracking the file at its new path. phase7_extensibility drives a real daemon-owned download end to end and asserts the moved file byte-for-byte.

The full script-visible surface — every event hook, the event object's fields, and the naut API table — is documented in docs/scripting.md. A complete worked example, examples/anime_sort.lua, sorts anime into a Title/Season NN/Title - SNNENN.ext library as each episode finishes, using an embedded Anitomy-style filename parser (examples/).

Roadmap (status)

Phase Scope State
1 Foundation: platform/ io_uring + core/ buffer pool/queues/bitfields done — built, tested, TSan-clean; echo ~30 Gbit/s on one core
2 crypto/ (SHA-1/256 + SHA-NI, Merkle, RC4) · bencode/ · metainfo/ (v1/v2/hybrid + magnet) done — FIPS/RC4 vectors pass, SHA-256 2.27 GB/s/core, info-hashes verified vs libtorrent, parsers fuzz-clean (3M iters, ASan+UBSan)
3 Single-peer transfer: peer/ · piece/ · storage/ · verify/ done — naut_leech downloads single/multi/hybrid from a libtorrent seed, byte-identical + SHA-1 verified (interop_leech test)
4 Trackers + swarm: HTTP/UDP trackers, choking, rarest-first, endgame done — naut_swarm discovers peers or accepts explicit endpoints; two-peer and live HTTP/UDP tracker interop gates pass against libtorrent
5 MSE encryption · DHT · PEX · ut_metadata · magnet-only start done — forced RC4 interop passes against libtorrent; trackerless magnet gate discovers a peer through DHT, verifies BEP-9 metadata, and completes byte-identically
6 Scale to 10 GbE: adaptive pipelining, SEND_ZC, registered bufs, SQPOLL, NUMA implementation complete; local 8-connection echo gate measured 15.11 Gbit/s with byte verification, while the plan's two-machine ≥9.4 Gbit/s NIC gate remains external hardware validation
7 Extensibility surface: RPC · plugin ABI · scripting · nautctl done — versioned Unix RPC with event streaming, reference storage plugin, sandboxed Lua hooks, bounded move command marshalling, and an end-to-end integration gate

Foundation design notes

  • naut_buf pool (src/core/buf.c): page-aligned, refcounted blocks from one mmap'd slab. get() is single-consumer (owning reactor); put()/ref() are multi-producer. Because only the owner pops, the Treiber-stack freelist is ABA-free without tagging. The whole slab can be registered with io_uring as a fixed-buffer region.
  • naut_mpmc (src/core/mpmc.c): Vyukov bounded queue — one implementation serves every cross-thread hand-off (control→reactor, reactor→hash-pool, back).
  • naut_bitfield (src/core/bitfield.c): popcount/ctz-based; includes the BEP-3 MSB-first wire conversion that the peer protocol needs.
  • platform/ wraps every syscall so "Linux-only now" stays "portable later": a future epoll backend is a new file, not a refactor.
  • crypto/: SHA-256 picks a SHA-NI or scalar backend at startup (NAUT_NO_SHANI=1 forces scalar); both are cross-checked against FIPS vectors in CI. RC4 carries the MSE 1024-byte keystream drop. Merkle implements BEP-52 zero-hash padding. Deferred by design: MSE Diffie-Hellman lands in Phase 5 (where it's used); a SHA-NI SHA-1 path is a Phase 6 optimization for v1-heavy swarms (scalar SHA-1 is ~0.27 GB/s, fine behind the hash pool).
  • metainfo/: info-hashes are computed over the raw info bytes and were verified against libtorrent for v1, v2, and hybrid. Fixtures are regenerated with python3 tests/fixtures/generate.py (needs python libtorrent).
  • Fuzzing: fuzz_lite <bencode|metainfo> <iters> [seeds...] is a mutational fuzzer; build with -DNAUT_SAN=address and seed from tests/fixtures/*.torrent.
  • peer/: a sans-IO wire codec — pure functions over byte buffers, no sockets — so the same code is driven by the blocking naut_leech now and the io_uring reactor in Phase 6. naut_leech <file.torrent> <dir> <ip> <port> downloads from one peer; a piece is assembled in RAM, SHA-1 verified, then written, so a corrupt piece never reaches disk.
  • tracker/ + naut_swarm: BEP-3 compact HTTP and BEP-15 UDP announces feed a deduplicated peer set. The swarm driver tracks HAVE/BITFIELD availability, respects choke/unchoke, expires stalled requests, schedules rarest-first, and bounds endgame races to two distinct peers per block. Redundant requests are canceled as soon as one copy arrives. interop_swarm requires two independent libtorrent seeds to contribute, while interop_tracker_swarm and interop_udp_tracker_swarm prove live discovery.
  • Phase 5 peer discovery and transport: outgoing MSE is a sans-IO state machine (naut_mse_handshake_*) — feed bytes, pull bytes, no sockets — so the io_uring reactor can drive an encrypted handshake without blocking a core; the blocking naut_mse_client_handshake is a thin wrapper over it. It performs the 768-bit Diffie-Hellman exchange, offers RC4-only PE, drops the first 1024 keystream bytes, and keeps independent connection-owned send/receive states. naut_dht builds and validates BEP-5 KRPC and performs a bounded iterative IPv4 get_peers lookup. BEP-10 handshakes advertise ut_metadata and PEX; metadata is assembled in 16 KiB blocks and rejected unless its raw SHA-1 matches the magnet's btih. interop_mse and interop_magnet_dht are the deterministic local gates for both required Phase 5 outcomes.
  • Phase 6 scaling path: each swarm peer adjusts its request window from smoothed throughput × RTT rather than a fixed depth. Completed pieces submit SHA-1 jobs to a bounded MPMC worker pool and return to the owner through eventfd; storage writes and callbacks remain owner-thread operations. Piece buffers are page-aligned, and NAUT_DIRECT_IO=1 uses O_DIRECT for aligned bulk regions with buffered edge fallback. The io_uring seam supports SQPOLL CPU affinity, registered slabs, fixed-buffer receive where the kernel accepts it, and SEND_ZC with notification-lifetime tracking. Two compatibility notes baked in: IORING_SETUP_SQPOLL is mutually exclusive with COOP_TASKRUN (the kernel -EINVALs the combo), so the ring pairs SQPOLL with SINGLE_ISSUER only; and a kernel that rejects IORING_RECVSEND_FIXED_BUF on plain recv is detected per-operation and every affected connection retries unfixed (not just the first), so fixed-buffer fallback never tears a peer down. run_echo_scale.sh asserts the server echoed every byte across all connections, turning any such drop into a hard failure. If a transport repeatedly reports copied SEND_ZC operations (as loopback does), the ring degrades to normal sends instead of paying useless notification overhead. Hugepage and NUMA slab placement are controlled by NAUT_HUGEPAGES and NAUT_NUMA_NODE; worker count/affinity use NAUT_WORKERS and NAUT_WORKER_CPU_BASE.
  • Measured Phase 6 CPU budget on this host (8 workers): SHA-1 1.62 GB/s, SHA-256 14.75 GB/s, and RC4 3.21 GB/s. The single-core SHA-256 gate measured 1.87 GB/s. These clear the 1.25 GB/s per-direction processing budget, but do not replace the physical 10 GbE two-host test required by plan.md.
  • storage/ maps the torrent's flat byte space across files (a write may straddle a file boundary) and recognises BEP-47 padding files in hybrid torrents, routing them out of the content tree. Interop is proven against libtorrent via tests/integration/run_interop.sh (also run by ctest).
  • Per-file completion / move-as-you-go (a headline feature): naut_download fires on_file_complete(file_index, path) the instant a file's last covering piece verifies — before the torrent finishes — and naut_storage_relocate() moves that file out safely (even mid-download, while other files' pieces are still arriving). test_filemove proves a file is relocated mid-download with no corruption. The scripting layer forwards the event to an on_file_complete hook and exposes move_file; the daemon queues the command back to the worker that owns the torrent's storage. phase7_extensibility exercises the whole chain — download worker → script thread → bounded command queue → download worker — and checks the moved bytes.