Initial commit: multi-peer torrent download engine
Reactor/loop-pool engine with TCP/µTP/MSE transports, per-connection pipelining, priority-driven piece selection with endgame, and the Python FFI test harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
ookami125
commit
d8208685a2
55 changed files with 9989 additions and 0 deletions
176
README.md
Normal file
176
README.md
Normal file
|
|
@ -0,0 +1,176 @@
|
|||
# torrent-peer
|
||||
|
||||
A fast, BitTorrent-wire-compatible **leech peer** written in C, driven by a
|
||||
Python **test harness**. The normal `.torrent` path parses metainfo locally and
|
||||
uses the neighboring `torrent-tracker` library for DHT get_peers and HTTP/UDP
|
||||
tracker announces; libtorrent remains in the tests for seed fixtures and as a
|
||||
magnet-metadata fallback.
|
||||
|
||||
Milestone 1 scope: download from connected peers as fast as possible. No
|
||||
seeding, PEX, or native magnet metadata resolution yet.
|
||||
|
||||
The peer **never hashes or persists data**. It only fetches the blocks the
|
||||
harness asks for and hands the raw bytes back. The harness owns *what* to
|
||||
download, reassembles pieces, and verifies SHA-1.
|
||||
|
||||
## How data moves (and why it's fast)
|
||||
|
||||
```
|
||||
enqueue_piece(i) request pipeline (BDP-deep)
|
||||
harness ───────────────────▶ C peer ───────────────────────────▶ remote peer
|
||||
▲ ▲ │ net thread (epoll + TCP)
|
||||
│ │ ready_ring (SPSC) │ recv() steered directly into arena slots
|
||||
│ └──────────────────────────┘
|
||||
│ block_desc{piece,begin,len,slot} ← zero-copy: payload lives in arena
|
||||
│
|
||||
└────── free_ring (SPSC) ── peer_release_slot(slot) ─────────────┐
|
||||
return spent slots = credit that lets new requests go out ┘
|
||||
```
|
||||
|
||||
- **SPSC lock-free rings (LMAX Disruptor pattern).** The net thread is the sole
|
||||
producer of completed blocks (`ready_ring`) and sole consumer of returned
|
||||
slots (`free_ring`); the harness thread is the mirror. Head/tail counters sit
|
||||
on separate cache lines with acquire/release ordering — no locks on the hot
|
||||
path. See `src/ring.h`.
|
||||
- **Pre-allocated, page-faulted arena.** `num_slots × 16 KiB` of memory is
|
||||
allocated and touched once up front (`src/arena.c`); there is **zero per-block
|
||||
allocation** in steady state. The harness wraps it once as a `memoryview` and
|
||||
reads each block zero-copy (`harness/peer_ffi.py`).
|
||||
- **Payload steering.** The parser reads a piece message's 13-byte header from a
|
||||
small staging buffer, then `recv()`s the bulk payload **straight into its
|
||||
arena slot** — the only bytes copied are whatever were prefetched alongside
|
||||
the header (`src/proto.c::rd_take_body`).
|
||||
- **Credit-based flow control.** A block consumes one free slot on arrival;
|
||||
requests are only issued while `outstanding < free_slots`, so a slow consumer
|
||||
naturally throttles the wire without dropping data (`src/scheduler.c`).
|
||||
- **Adaptive pipeline depth.** The in-flight target tracks the bandwidth-delay
|
||||
product, `target ≈ 2 × rate × min_rtt / 16 KiB`, clamped to
|
||||
`[32, max_pipeline]`. It uses the *minimum* observed request→block RTT (not an
|
||||
average) so the estimate reflects the unloaded path instead of feeding back
|
||||
the pipeline's own queueing delay (`src/scheduler.c`).
|
||||
- **Request timeouts + re-queue.** Every issued block is tracked with its send
|
||||
time; a block that goes unanswered past `request_timeout_ms` is re-queued and
|
||||
re-issued ahead of fresh requests, so a silently dropped request can't stall
|
||||
the download. Incoming blocks are matched against outstanding requests — an
|
||||
unsolicited/duplicate block is drained and dropped without consuming a slot or
|
||||
disturbing the credit accounting (`src/reqtab.c`, `src/proto.c`).
|
||||
|
||||
## Piece selection
|
||||
|
||||
The harness supplies a **priority vector — one `uint8_t` per piece** — via
|
||||
`peer_set_priorities()` / `peer_set_priority()`. At each piece boundary the peer
|
||||
picks the highest-priority piece that
|
||||
|
||||
1. the **remote peer actually has** (tracked from its `bitfield`/`have`
|
||||
messages — the peer never assumes availability), and
|
||||
2. has **not already been fully requested**,
|
||||
|
||||
breaking ties toward the lowest piece index. Priority `0` means "skip". Because
|
||||
selection re-reads the vector live, the harness can implement any scheme by
|
||||
rewriting priorities over time — sequential, rarest-first, or a deadline ramp
|
||||
(e.g. fetch episode 1 first, then rarest-first, while slowly raising episode 2's
|
||||
priorities so it lands inside 24 minutes) — without any peer-side changes. A
|
||||
piece that fails verification is re-armed with `peer_request_piece()`.
|
||||
|
||||
This replaces a fixed in-order schedule, which stalled whenever the connected
|
||||
peer lacked the next piece. Selection is one O(num_pieces) scan per *piece* (not
|
||||
per block); `src/scheduler.c`.
|
||||
|
||||
io_uring zero-copy receive is the natural next upgrade and would drop into
|
||||
`src/net.c` without touching the protocol parser or the arena handoff.
|
||||
|
||||
## Layout
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `include/peer.h` | public C ABI (the only surface ctypes binds to) |
|
||||
| `src/peer.c` | lifecycle, helpers, control-plane piece queue |
|
||||
| `src/net.c` | TCP connect, socket tuning, epoll loop, outgoing buffer |
|
||||
| `src/proto.c` | handshake + wire framing + payload steering |
|
||||
| `src/scheduler.c` | piece selection, adaptive depth, timeouts |
|
||||
| `src/reqtab.c` | in-flight request table (match + timeout) |
|
||||
| `src/ring.c/.h` | SPSC rings |
|
||||
| `src/arena.c` | aligned, pre-faulted block arena |
|
||||
| `harness/peer_ffi.py` | ctypes bindings + zero-copy arena view |
|
||||
| `harness/torrent_meta.py` | local bencoded `.torrent` parser |
|
||||
| `harness/tracker_ffi.py` | ctypes bindings for `../torrent-tracker` DHT/tracker helpers |
|
||||
| `harness/harness.py` | tracker discovery + reassembly + SHA-1 verify |
|
||||
| `tests/test_localseed.py` | end-to-end test against a local libtorrent seed |
|
||||
|
||||
## Build
|
||||
|
||||
```sh
|
||||
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
|
||||
cmake --build build # -> build/libtorrentpeer.so
|
||||
```
|
||||
|
||||
Options: `-DPEER_NATIVE=OFF` (portable build, no `-march=native`),
|
||||
`-DPEER_ASAN=ON` (AddressSanitizer/UBSan).
|
||||
|
||||
## Test
|
||||
|
||||
The end-to-end test creates a random file, seeds it with libtorrent on
|
||||
localhost, downloads every piece through the C peer, and asserts the bytes match
|
||||
(it builds the library automatically if needed):
|
||||
|
||||
```sh
|
||||
python tests/test_localseed.py # standalone
|
||||
# or, if pytest is available: python -m pytest tests/
|
||||
```
|
||||
|
||||
Run the full path under ASan:
|
||||
|
||||
```sh
|
||||
cmake -S . -B build-asan -DPEER_ASAN=ON && cmake --build build-asan
|
||||
ASAN_OPTIONS=detect_leaks=0 \
|
||||
LD_PRELOAD=$(gcc -print-file-name=libasan.so) \
|
||||
python tests/test_localseed.py
|
||||
```
|
||||
|
||||
For offline interoperability against multiple seed clients, use the Docker
|
||||
Compose harness in `interop/`. It creates a private trackerless fixture torrent,
|
||||
starts libtorrent, Transmission, and aria2 seeders on an internal-only Docker
|
||||
network, then verifies that this engine can download and hash-check the fixture
|
||||
from each one:
|
||||
|
||||
```sh
|
||||
docker compose -f interop/docker-compose.yml up --build \
|
||||
--abort-on-container-exit --exit-code-from runner
|
||||
```
|
||||
|
||||
## Use against a real torrent
|
||||
|
||||
```sh
|
||||
# explicit peer (skip tracker):
|
||||
python harness/harness.py file.torrent --peer 1.2.3.4:51413 -o out.bin
|
||||
# or discover peers via DHT and the torrent's HTTP/UDP trackers:
|
||||
python harness/harness.py file.torrent -o out.bin
|
||||
```
|
||||
|
||||
Tunables: `--slots N` (arena depth, 16 KiB each), `--pipeline N` (cap on
|
||||
outstanding requests; the live depth adapts to the BDP under this cap),
|
||||
`--timeout SECONDS`. Per-block request timeout and an optional `SO_RCVBUF`
|
||||
override are `peer_config` fields (`request_timeout_ms`, `recv_buffer_bytes`).
|
||||
|
||||
## Status
|
||||
|
||||
Verified byte-for-byte on local-seed transfers up to 256 MiB; ~400+ MB/s over
|
||||
loopback with the arena (16 MiB) far smaller than the file, i.e. with
|
||||
credit-based backpressure fully engaged. Clean under ASan/UBSan. The test suite
|
||||
covers full download + byte verification, a **partial seed** (peer downloads
|
||||
only the pieces the remote has), **priority ordering**, and — via a raw-socket
|
||||
mock peer — **request-timeout recovery** and **unsolicited-block rejection**.
|
||||
|
||||
### Known gaps / next steps
|
||||
|
||||
- Real-swarm auto-discovery is still hit-or-miss: many announced peers are
|
||||
unreachable, behind NAT, or only usable with a different transport/encryption
|
||||
mode. Use `harness/swarm_download.py --encryption` and `--utp` to choose the
|
||||
dial mode.
|
||||
- Single peer / single net thread. Aggregate throughput needs a multi-peer
|
||||
connection manager with a shared piece picker (enables endgame mode + CANCEL).
|
||||
- io_uring registered buffers + multishot recv would give a true zero-copy,
|
||||
near-syscall-free receive path (drops into `src/net.c`).
|
||||
- Optional in-library SHA-NI hashing on a worker pool, so wire speed isn't held
|
||||
hostage to Python hash throughput at multi-GB/s.
|
||||
- Wider peer reach: µTP, MSE/PE encryption, IPv6, Fast Extension (BEP-6).
|
||||
Loading…
Add table
Add a link
Reference in a new issue