Initial commit: Naut-Torrent — from-scratch 10 GbE BitTorrent client
A maintainable, extensible BitTorrent client (C11, Linux/io_uring) targeting 10 GbE saturation. All torrent functionality is built from scratch; liburing is the only linked third-party dependency on the data path. Implements Phases 1-7 of the roadmap: - core: page-aligned buffer pool, MPMC/Treiber queues, bitfields, worker pool - crypto: SHA-1/256 (SHA-NI + scalar), Merkle (BEP-52), RC4 (MSE) - bencode/metainfo: zero-copy parser, v1/v2/hybrid .torrent + magnet - peer: sans-IO wire codec, MSE/PE handshake state machine, BEP-10, ut_metadata, PEX - piece/storage: block-level multi-peer engine, rarest-first + endgame, per-file completion events + single-file relocate (move-as-you-finish) - tracker/dht: HTTP + UDP (BEP-15) trackers, BEP-5 KRPC iterative lookup - platform: io_uring reactor (SQPOLL, registered buffers, SEND_ZC) - surface: versioned RPC, native plugin ABI, sandboxed Lua scripting, nautd/nautctl Verified against libtorrent (single/multi/hybrid, MSE, magnet-via-DHT, swarm); unit + interop tests green; ASan/UBSan/TSan clean. Scripting reference in docs/scripting.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
ookami125
commit
2178d6a70c
121 changed files with 12644 additions and 0 deletions
216
README.md
Normal file
216
README.md
Normal file
|
|
@ -0,0 +1,216 @@
|
|||
# Naut-Torrent
|
||||
|
||||
A maintainable, extensible BitTorrent client engineered to **saturate a 10 GbE
|
||||
link (~1.25 GB/s) in both directions** on Linux. Written in C11.
|
||||
|
||||
- **Platform:** Linux-only, built hard around **io_uring** (network *and* disk).
|
||||
- **Protocol:** BitTorrent **v1 + v2 hybrid** (SHA-1 pieces + SHA-256 Merkle).
|
||||
- **Workload:** saturate download *and* upload, with **MSE/PE encryption** in the hot path.
|
||||
- **Extensible:** clean module boundaries + control **RPC** + **BEP-10** extensions
|
||||
+ a versioned native **plugin ABI** + embedded **scripting**.
|
||||
|
||||
The full architecture rationale lives in [`plan.md`](plan.md). The short version: a
|
||||
**shared-nothing, thread-per-core reactor** model with **offload pools** for
|
||||
hashing/crypto, and a **one-buffer-zero-copy** data path so bytes flow
|
||||
recv → decrypt → hash → disk/send without a single `memcpy`.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
include/naut/ public headers
|
||||
src/core/ zero-dependency foundation (buffers, queues, bitfields, log)
|
||||
src/platform/ the ONLY code that touches the kernel (io_uring, sockets)
|
||||
src/dht/ BEP-5 KRPC codec + bounded iterative peer discovery
|
||||
src/peer/ wire protocol, MSE/RC4, BEP-10, ut_metadata, and PEX
|
||||
apps/echo/ Phase 1 gate: io_uring echo server on the buffer pool
|
||||
apps/leech/ Phase 3 gate: verified single-peer download
|
||||
apps/swarm/ tracker/DHT discovery, magnets, and concurrent peers
|
||||
tests/unit/ unit + concurrency tests
|
||||
```
|
||||
|
||||
## Build, test, run
|
||||
|
||||
```sh
|
||||
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
|
||||
ninja -C build
|
||||
ctest --test-dir build --output-on-failure
|
||||
|
||||
# sanitizer build (address|thread|undefined)
|
||||
cmake -S . -B build-tsan -G Ninja -DCMAKE_BUILD_TYPE=Debug -DNAUT_SAN=thread
|
||||
ninja -C build-tsan && ./build-tsan/test_buf
|
||||
|
||||
# run the Phase 1 echo gate
|
||||
./build/naut_echo 9000
|
||||
|
||||
# download from explicit peers, or omit them to use the torrent's trackers
|
||||
./build/naut_swarm file.torrent output/ 192.0.2.10:6881 192.0.2.11:6881
|
||||
./build/naut_swarm file.torrent output/
|
||||
|
||||
# trackerless magnet start through DHT (override bootstraps when needed)
|
||||
./build/naut_swarm 'magnet:?xt=urn:btih:...' output/
|
||||
NAUT_DHT_BOOTSTRAP=127.0.0.1:6881 ./build/naut_swarm 'magnet:?xt=urn:btih:...' output/
|
||||
|
||||
# force an encrypted single-peer MSE/RC4 connection
|
||||
./build/naut_leech --mse file.torrent output/ 192.0.2.10 6881
|
||||
|
||||
# Phase 6 CPU and reactor benchmarks
|
||||
./build/bench_hash
|
||||
./build/bench_scale 8 8
|
||||
bash tests/integration/run_echo_scale.sh ./build/naut_echo
|
||||
|
||||
# optional data-path tuning
|
||||
NAUT_DIRECT_IO=1 NAUT_WORKERS=8 ./build/naut_swarm file.torrent output/
|
||||
NAUT_CPU=2 NAUT_SQPOLL=1 NAUT_HUGEPAGES=1 NAUT_NUMA_NODE=0 ./build/naut_echo 9000
|
||||
```
|
||||
|
||||
Requirements: Linux ≥ 6.0, `liburing` (≥ 2.x), OpenSSL `libcrypto`, Jansson,
|
||||
Lua, CMake ≥ 3.20, gcc/clang, Ninja.
|
||||
|
||||
## Daemon, RPC, plugins, and scripts
|
||||
|
||||
Phase 7 adds a headless control process and thin CLI over a versioned,
|
||||
length-prefixed JSON protocol on a Unix socket:
|
||||
|
||||
```sh
|
||||
./build/nautd \
|
||||
--socket /tmp/nautd.sock \
|
||||
--plugin ./build/naut_example.so \
|
||||
--script ./tests/fixtures/phase7.lua
|
||||
|
||||
./build/nautctl ping
|
||||
./build/nautctl plugins
|
||||
./build/nautctl status
|
||||
./build/nautctl events
|
||||
```
|
||||
|
||||
`nautctl` accepts an optional JSON value after the method:
|
||||
|
||||
```sh
|
||||
# register a torrent's storage so a move command can resolve + relocate its files
|
||||
./build/nautctl add_torrent \
|
||||
'{"torrent_id":7,"torrent":"file.torrent","root":"output/"}'
|
||||
./build/nautctl emit \
|
||||
'{"type":"torrent_finished","torrent_id":7}'
|
||||
./build/nautctl shutdown
|
||||
```
|
||||
|
||||
The native ABI is declared in `include/naut/naut_plugin.h`. Plugins export
|
||||
`naut_plugin_register()`, receive the versioned host API, and may register RPC
|
||||
methods, storage backends, and event handlers. `plugins/example/example.c`
|
||||
provides the reference in-memory storage backend.
|
||||
|
||||
Lua scripts run on a dedicated thread behind a bounded event queue. Supported
|
||||
hooks are `on_torrent_added`, `on_piece_complete`, `on_file_complete`,
|
||||
`on_torrent_finished`, `on_peer_connected`, and `on_alert`. The sandbox removes
|
||||
filesystem, process, package-loading, debug, and raw chunk-loading globals
|
||||
(`os`, `io`, `package`/`require`, `debug`, `dofile`/`loadfile`, and
|
||||
`load`/`loadstring` — the bytecode loaders are denied so a crafted binary chunk
|
||||
can't escape the VM). `naut.move_file()` submits a bounded command from the
|
||||
script thread to the daemon owner thread; the owner resolves it through the
|
||||
torrent registry (`naut_session`) and performs the relocate with
|
||||
`naut_storage_relocate()`. Register a torrent's storage first with the
|
||||
`add_torrent` RPC so the id resolves. `phase7_extensibility` drives this
|
||||
end to end and asserts the file actually moves on disk.
|
||||
|
||||
The full script-visible surface — every event hook, the `event` object's
|
||||
fields, and the `naut` API table — is documented in
|
||||
[`docs/scripting.md`](docs/scripting.md).
|
||||
|
||||
## Roadmap (status)
|
||||
|
||||
| Phase | Scope | State |
|
||||
|------|-------|-------|
|
||||
| 1 | Foundation: `platform/` io_uring + `core/` buffer pool/queues/bitfields | **done — built, tested, TSan-clean; echo ~30 Gbit/s on one core** |
|
||||
| 2 | `crypto/` (SHA-1/256 + SHA-NI, Merkle, RC4) · `bencode/` · `metainfo/` (v1/v2/hybrid + magnet) | **done — FIPS/RC4 vectors pass, SHA-256 2.27 GB/s/core, info-hashes verified vs libtorrent, parsers fuzz-clean (3M iters, ASan+UBSan)** |
|
||||
| 3 | Single-peer transfer: `peer/` · `piece/` · `storage/` · `verify/` | **done — `naut_leech` downloads single/multi/hybrid from a libtorrent seed, byte-identical + SHA-1 verified (`interop_leech` test)** |
|
||||
| 4 | Trackers + swarm: HTTP/UDP trackers, choking, rarest-first, endgame | **done — `naut_swarm` discovers peers or accepts explicit endpoints; two-peer and live HTTP/UDP tracker interop gates pass against libtorrent** |
|
||||
| 5 | MSE encryption · DHT · PEX · ut_metadata · magnet-only start | **done — forced RC4 interop passes against libtorrent; trackerless magnet gate discovers a peer through DHT, verifies BEP-9 metadata, and completes byte-identically** |
|
||||
| 6 | Scale to 10 GbE: adaptive pipelining, SEND_ZC, registered bufs, SQPOLL, NUMA | **implementation complete; local 8-connection echo gate measured 15.11 Gbit/s with byte verification, while the plan's two-machine ≥9.4 Gbit/s NIC gate remains external hardware validation** |
|
||||
| 7 | Extensibility surface: RPC · plugin ABI · scripting · `nautctl` | **done — versioned Unix RPC with event streaming, reference storage plugin, sandboxed Lua hooks, bounded move command marshalling, and an end-to-end integration gate** |
|
||||
|
||||
## Foundation design notes
|
||||
|
||||
- **`naut_buf` pool** (`src/core/buf.c`): page-aligned, refcounted blocks from
|
||||
one mmap'd slab. `get()` is single-consumer (owning reactor); `put()`/`ref()`
|
||||
are multi-producer. Because only the owner pops, the Treiber-stack freelist is
|
||||
ABA-free without tagging. The whole slab can be registered with io_uring as a
|
||||
fixed-buffer region.
|
||||
- **`naut_mpmc`** (`src/core/mpmc.c`): Vyukov bounded queue — one implementation
|
||||
serves every cross-thread hand-off (control→reactor, reactor→hash-pool, back).
|
||||
- **`naut_bitfield`** (`src/core/bitfield.c`): popcount/ctz-based; includes the
|
||||
BEP-3 MSB-first wire conversion that the peer protocol needs.
|
||||
- **`platform/`** wraps every syscall so "Linux-only now" stays "portable later":
|
||||
a future epoll backend is a new file, not a refactor.
|
||||
- **`crypto/`**: SHA-256 picks a SHA-NI or scalar backend at startup
|
||||
(`NAUT_NO_SHANI=1` forces scalar); both are cross-checked against FIPS vectors
|
||||
in CI. RC4 carries the MSE 1024-byte keystream drop. Merkle implements BEP-52
|
||||
zero-hash padding. *Deferred by design:* MSE Diffie-Hellman lands in Phase 5
|
||||
(where it's used); a SHA-NI SHA-1 path is a Phase 6 optimization for
|
||||
v1-heavy swarms (scalar SHA-1 is ~0.27 GB/s, fine behind the hash pool).
|
||||
- **`metainfo/`**: info-hashes are computed over the raw `info` bytes and were
|
||||
verified against libtorrent for v1, v2, and hybrid. Fixtures are regenerated
|
||||
with `python3 tests/fixtures/generate.py` (needs python `libtorrent`).
|
||||
- **Fuzzing**: `fuzz_lite <bencode|metainfo> <iters> [seeds...]` is a mutational
|
||||
fuzzer; build with `-DNAUT_SAN=address` and seed from `tests/fixtures/*.torrent`.
|
||||
- **`peer/`**: a *sans-IO* wire codec — pure functions over byte buffers, no
|
||||
sockets — so the same code is driven by the blocking `naut_leech` now and the
|
||||
io_uring reactor in Phase 6. `naut_leech <file.torrent> <dir> <ip> <port>`
|
||||
downloads from one peer; a piece is assembled in RAM, SHA-1 verified, then
|
||||
written, so a corrupt piece never reaches disk.
|
||||
- **`tracker/` + `naut_swarm`**: BEP-3 compact HTTP and BEP-15 UDP announces
|
||||
feed a deduplicated peer set. The swarm driver tracks HAVE/BITFIELD
|
||||
availability, respects choke/unchoke, expires stalled requests, schedules
|
||||
rarest-first, and bounds endgame races to two distinct peers per block.
|
||||
Redundant requests are canceled as soon as one copy arrives. `interop_swarm`
|
||||
requires two independent libtorrent seeds to contribute, while
|
||||
`interop_tracker_swarm` and `interop_udp_tracker_swarm` prove live discovery.
|
||||
- **Phase 5 peer discovery and transport**: outgoing MSE is a *sans-IO* state
|
||||
machine (`naut_mse_handshake_*`) — feed bytes, pull bytes, no sockets — so the
|
||||
io_uring reactor can drive an encrypted handshake without blocking a core; the
|
||||
blocking `naut_mse_client_handshake` is a thin wrapper over it. It performs the
|
||||
768-bit Diffie-Hellman exchange, offers RC4-only PE, drops the first 1024
|
||||
keystream bytes, and keeps independent connection-owned send/receive states.
|
||||
`naut_dht` builds and validates BEP-5 KRPC and performs a bounded iterative
|
||||
IPv4 `get_peers` lookup. BEP-10 handshakes advertise `ut_metadata` and PEX;
|
||||
metadata is assembled in 16 KiB blocks and rejected unless its raw SHA-1
|
||||
matches the magnet's `btih`. `interop_mse` and `interop_magnet_dht` are the
|
||||
deterministic local gates for both required Phase 5 outcomes.
|
||||
- **Phase 6 scaling path**: each swarm peer adjusts its request window from
|
||||
smoothed throughput × RTT rather than a fixed depth. Completed pieces submit
|
||||
SHA-1 jobs to a bounded MPMC worker pool and return to the owner through
|
||||
eventfd; storage writes and callbacks remain owner-thread operations.
|
||||
Piece buffers are page-aligned, and `NAUT_DIRECT_IO=1` uses O_DIRECT for
|
||||
aligned bulk regions with buffered edge fallback. The io_uring seam supports
|
||||
SQPOLL CPU affinity, registered slabs, fixed-buffer receive where the kernel
|
||||
accepts it, and SEND_ZC with notification-lifetime tracking. Two
|
||||
compatibility notes baked in: `IORING_SETUP_SQPOLL` is mutually exclusive with
|
||||
`COOP_TASKRUN` (the kernel `-EINVAL`s the combo), so the ring pairs SQPOLL with
|
||||
`SINGLE_ISSUER` only; and a kernel that rejects `IORING_RECVSEND_FIXED_BUF` on
|
||||
plain recv is detected per-operation and every affected connection retries
|
||||
unfixed (not just the first), so fixed-buffer fallback never tears a peer down.
|
||||
`run_echo_scale.sh` asserts the server echoed *every* byte across all
|
||||
connections, turning any such drop into a hard failure. If a transport
|
||||
repeatedly reports copied SEND_ZC operations (as loopback does), the ring
|
||||
degrades to normal sends instead of paying useless notification overhead.
|
||||
Hugepage and NUMA slab placement are controlled by `NAUT_HUGEPAGES` and
|
||||
`NAUT_NUMA_NODE`; worker count/affinity use `NAUT_WORKERS` and
|
||||
`NAUT_WORKER_CPU_BASE`.
|
||||
- **Measured Phase 6 CPU budget on this host** (8 workers): SHA-1 1.62 GB/s,
|
||||
SHA-256 14.75 GB/s, and RC4 3.21 GB/s. The single-core SHA-256 gate measured
|
||||
1.87 GB/s. These clear the 1.25 GB/s per-direction processing budget, but do
|
||||
not replace the physical 10 GbE two-host test required by `plan.md`.
|
||||
- **`storage/`** maps the torrent's flat byte space across files (a write may
|
||||
straddle a file boundary) and recognises BEP-47 padding files in hybrid
|
||||
torrents, routing them out of the content tree. Interop is proven against
|
||||
libtorrent via `tests/integration/run_interop.sh` (also run by ctest).
|
||||
- **Per-file completion / move-as-you-go** (a headline feature): `naut_download`
|
||||
fires `on_file_complete(file_index, path)` the instant a file's last covering
|
||||
piece verifies — before the torrent finishes — and `naut_storage_relocate()`
|
||||
moves that file out safely (even mid-download, while other files' pieces are
|
||||
still arriving). `test_filemove` proves a file is relocated mid-download with
|
||||
no corruption. The scripting layer (Phase 7) forwards the event to an
|
||||
`on_file_complete` hook and exposes `move_file`; the daemon resolves the
|
||||
command through the `naut_session` torrent registry (`src/session/session.c`)
|
||||
and calls `naut_storage_relocate()` on its owner thread. `phase7_extensibility`
|
||||
exercises the whole chain — script thread → bounded queue → owner thread →
|
||||
storage — and asserts the file moves on disk.
|
||||
Loading…
Add table
Add a link
Reference in a new issue