torrent-peer

A fast, BitTorrent-wire-compatible leech peer written in C, driven by a Python test harness. The normal .torrent path parses metainfo locally and uses the neighboring torrent-tracker library for DHT get_peers and HTTP/UDP tracker announces; libtorrent remains in the tests for seed fixtures and as a magnet-metadata fallback.

Milestone 1 scope: download from connected peers as fast as possible. No seeding, PEX, or native magnet metadata resolution yet.

The peer never hashes or persists data. It only fetches the blocks the harness asks for and hands the raw bytes back. The harness owns what to download, reassembles pieces, and verifies SHA-1.

How data moves (and why it's fast)

            enqueue_piece(i)            request pipeline (BDP-deep)
 harness  ───────────────────▶  C peer ───────────────────────────▶  remote peer
   ▲  ▲                          │  net thread (epoll + TCP)
   │  │   ready_ring (SPSC)       │  recv() steered directly into arena slots
   │  └──────────────────────────┘
   │      block_desc{piece,begin,len,slot}   ← zero-copy: payload lives in arena
   │
   └──────  free_ring (SPSC)  ── peer_release_slot(slot)  ─────────────┐
            return spent slots = credit that lets new requests go out  ┘

• SPSC lock-free rings (LMAX Disruptor pattern). The net thread is the sole producer of completed blocks (ready_ring) and sole consumer of returned slots (free_ring); the harness thread is the mirror. Head/tail counters sit on separate cache lines with acquire/release ordering — no locks on the hot path. See src/ring.h.
• Pre-allocated, page-faulted arena. num_slots × 16 KiB of memory is allocated and touched once up front (src/arena.c); there is zero per-block allocation in steady state. The harness wraps it once as a memoryview and reads each block zero-copy (harness/peer_ffi.py).
• Payload steering. The parser reads a piece message's 13-byte header from a small staging buffer, then recv()s the bulk payload straight into its arena slot — the only bytes copied are whatever were prefetched alongside the header (src/proto.c::rd_take_body).
• Credit-based flow control. A block consumes one free slot on arrival; requests are only issued while outstanding < free_slots, so a slow consumer naturally throttles the wire without dropping data (src/scheduler.c).
• Adaptive pipeline depth. The in-flight target tracks the bandwidth-delay product, target ≈ 2 × rate × min_rtt / 16 KiB, clamped to [32, max_pipeline]. It uses the minimum observed request→block RTT (not an average) so the estimate reflects the unloaded path instead of feeding back the pipeline's own queueing delay (src/scheduler.c).
• Request timeouts + re-queue. Every issued block is tracked with its send time; a block that goes unanswered past request_timeout_ms is re-queued and re-issued ahead of fresh requests, so a silently dropped request can't stall the download. Incoming blocks are matched against outstanding requests — an unsolicited/duplicate block is drained and dropped without consuming a slot or disturbing the credit accounting (src/reqtab.c, src/proto.c).

Piece selection

The harness supplies a priority vector — one uint8_t per piece — via peer_set_priorities() / peer_set_priority(). At each piece boundary the peer picks the highest-priority piece that

1. the remote peer actually has (tracked from its bitfield/have messages — the peer never assumes availability), and
2. has not already been fully requested,

breaking ties toward the lowest piece index. Priority 0 means "skip". Because selection re-reads the vector live, the harness can implement any scheme by rewriting priorities over time — sequential, rarest-first, or a deadline ramp (e.g. fetch episode 1 first, then rarest-first, while slowly raising episode 2's priorities so it lands inside 24 minutes) — without any peer-side changes. A piece that fails verification is re-armed with peer_request_piece().

This replaces a fixed in-order schedule, which stalled whenever the connected peer lacked the next piece. Selection is one O(num_pieces) scan per piece (not per block); src/scheduler.c.

io_uring zero-copy receive is the natural next upgrade and would drop into src/net.c without touching the protocol parser or the arena handoff.

Layout

Path	Role
include/peer.h	public C ABI (the only surface ctypes binds to)
src/peer.c	lifecycle, helpers, control-plane piece queue
src/net.c	TCP connect, socket tuning, epoll loop, outgoing buffer
src/proto.c	handshake + wire framing + payload steering
src/scheduler.c	piece selection, adaptive depth, timeouts
src/reqtab.c	in-flight request table (match + timeout)
src/ring.c/.h	SPSC rings
src/arena.c	aligned, pre-faulted block arena
harness/peer_ffi.py	ctypes bindings + zero-copy arena view
harness/torrent_meta.py	local bencoded .torrent parser
harness/tracker_ffi.py	ctypes bindings for ../torrent-tracker DHT/tracker helpers
harness/harness.py	tracker discovery + reassembly + SHA-1 verify
tests/test_localseed.py	end-to-end test against a local libtorrent seed

Build

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build           # -> build/libtorrentpeer.so

Options: -DPEER_NATIVE=OFF (portable build, no -march=native), -DPEER_ASAN=ON (AddressSanitizer/UBSan).

Test

The end-to-end test creates a random file, seeds it with libtorrent on localhost, downloads every piece through the C peer, and asserts the bytes match (it builds the library automatically if needed):

python tests/test_localseed.py      # standalone
# or, if pytest is available:  python -m pytest tests/

Run the full path under ASan:

cmake -S . -B build-asan -DPEER_ASAN=ON && cmake --build build-asan
ASAN_OPTIONS=detect_leaks=0 \
  LD_PRELOAD=$(gcc -print-file-name=libasan.so) \
  python tests/test_localseed.py

For offline interoperability against multiple seed clients, use the Docker Compose harness in interop/. It creates a private trackerless fixture torrent, starts libtorrent, Transmission, and aria2 seeders on an internal-only Docker network, then verifies that this engine can download and hash-check the fixture from each one:

docker compose -f interop/docker-compose.yml up --build \
  --abort-on-container-exit --exit-code-from runner

Use against a real torrent

# explicit peer (skip tracker):
python harness/harness.py file.torrent --peer 1.2.3.4:51413 -o out.bin
# or discover peers via DHT and the torrent's HTTP/UDP trackers:
python harness/harness.py file.torrent -o out.bin

Tunables: --slots N (arena depth, 16 KiB each), --pipeline N (cap on outstanding requests; the live depth adapts to the BDP under this cap), --timeout SECONDS. Per-block request timeout and an optional SO_RCVBUF override are peer_config fields (request_timeout_ms, recv_buffer_bytes).

Status

Verified byte-for-byte on local-seed transfers up to 256 MiB; ~400+ MB/s over loopback with the arena (16 MiB) far smaller than the file, i.e. with credit-based backpressure fully engaged. Clean under ASan/UBSan. The test suite covers full download + byte verification, a partial seed (peer downloads only the pieces the remote has), priority ordering, and — via a raw-socket mock peer — request-timeout recovery and unsolicited-block rejection.

Known gaps / next steps

• Real-swarm auto-discovery is still hit-or-miss: many announced peers are unreachable, behind NAT, or only usable with a different transport/encryption mode. Use harness/swarm_download.py --encryption and --utp to choose the dial mode.
• Single peer / single net thread. Aggregate throughput needs a multi-peer connection manager with a shared piece picker (enables endgame mode + CANCEL).
• io_uring registered buffers + multishot recv would give a true zero-copy, near-syscall-free receive path (drops into src/net.c).
• Optional in-library SHA-NI hashing on a worker pool, so wire speed isn't held hostage to Python hash throughput at multi-GB/s.
• Wider peer reach: µTP, MSE/PE encryption, IPv6, Fast Extension (BEP-6).