# torrent-peer

A fast, BitTorrent-wire-compatible **leech peer** written in C, driven by a
Python **test harness**. The normal `.torrent` path parses metainfo locally and
uses the neighboring `torrent-tracker` library for DHT get_peers and HTTP/UDP
tracker announces; libtorrent remains in the tests for seed fixtures and as a
magnet-metadata fallback.

Milestone 1 scope: download from connected peers as fast as possible. No
seeding, PEX, or native magnet metadata resolution yet.

The peer **never hashes or persists data**. It only fetches the blocks the
harness asks for and hands the raw bytes back. The harness owns *what* to
download, reassembles pieces, and verifies SHA-1.

## How data moves (and why it's fast)

```
            enqueue_piece(i)            request pipeline (BDP-deep)
 harness  ───────────────────▶  C peer ───────────────────────────▶  remote peer
   ▲  ▲                          │  net thread (epoll + TCP)
   │  │   ready_ring (SPSC)       │  recv() steered directly into arena slots
   │  └──────────────────────────┘
   │      block_desc{piece,begin,len,slot}   ← zero-copy: payload lives in arena
   │
   └──────  free_ring (SPSC)  ── peer_release_slot(slot)  ─────────────┐
            return spent slots = credit that lets new requests go out  ┘
```

- **SPSC lock-free rings (LMAX Disruptor pattern).** The net thread is the sole
  producer of completed blocks (`ready_ring`) and sole consumer of returned
  slots (`free_ring`); the harness thread is the mirror. Head/tail counters sit
  on separate cache lines with acquire/release ordering — no locks on the hot
  path. See `src/ring.h`.
- **Pre-allocated, page-faulted arena.** `num_slots × 16 KiB` of memory is
  allocated and touched once up front (`src/arena.c`); there is **zero per-block
  allocation** in steady state. The harness wraps it once as a `memoryview` and
  reads each block zero-copy (`harness/peer_ffi.py`).
- **Payload steering.** The parser reads a piece message's 13-byte header from a
  small staging buffer, then `recv()`s the bulk payload **straight into its
  arena slot** — the only bytes copied are whatever were prefetched alongside
  the header (`src/proto.c::rd_take_body`).
- **Credit-based flow control.** A block consumes one free slot on arrival;
  requests are only issued while `outstanding < free_slots`, so a slow consumer
  naturally throttles the wire without dropping data (`src/scheduler.c`).
- **Adaptive pipeline depth.** The in-flight target tracks the bandwidth-delay
  product, `target ≈ 2 × rate × min_rtt / 16 KiB`, clamped to
  `[32, max_pipeline]`. It uses the *minimum* observed request→block RTT (not an
  average) so the estimate reflects the unloaded path instead of feeding back
  the pipeline's own queueing delay (`src/scheduler.c`).
- **Request timeouts + re-queue.** Every issued block is tracked with its send
  time; a block that goes unanswered past `request_timeout_ms` is re-queued and
  re-issued ahead of fresh requests, so a silently dropped request can't stall
  the download. Incoming blocks are matched against outstanding requests — an
  unsolicited/duplicate block is drained and dropped without consuming a slot or
  disturbing the credit accounting (`src/reqtab.c`, `src/proto.c`).

## Piece selection

The harness supplies a **priority vector — one `uint8_t` per piece** — via
`peer_set_priorities()` / `peer_set_priority()`. At each piece boundary the peer
picks the highest-priority piece that

1. the **remote peer actually has** (tracked from its `bitfield`/`have`
   messages — the peer never assumes availability), and
2. has **not already been fully requested**,

breaking ties toward the lowest piece index. Priority `0` means "skip". Because
selection re-reads the vector live, the harness can implement any scheme by
rewriting priorities over time — sequential, rarest-first, or a deadline ramp
(e.g. fetch episode 1 first, then rarest-first, while slowly raising episode 2's
priorities so it lands inside 24 minutes) — without any peer-side changes. A
piece that fails verification is re-armed with `peer_request_piece()`.

This replaces a fixed in-order schedule, which stalled whenever the connected
peer lacked the next piece. Selection is one O(num_pieces) scan per *piece* (not
per block); `src/scheduler.c`.

io_uring zero-copy receive is the natural next upgrade and would drop into
`src/net.c` without touching the protocol parser or the arena handoff.

## Layout

| Path | Role |
|------|------|
| `include/peer.h` | public C ABI (the only surface ctypes binds to) |
| `src/peer.c` | lifecycle, helpers, control-plane piece queue |
| `src/net.c` | TCP connect, socket tuning, epoll loop, outgoing buffer |
| `src/proto.c` | handshake + wire framing + payload steering |
| `src/scheduler.c` | piece selection, adaptive depth, timeouts |
| `src/reqtab.c` | in-flight request table (match + timeout) |
| `src/ring.c/.h` | SPSC rings |
| `src/arena.c` | aligned, pre-faulted block arena |
| `harness/peer_ffi.py` | ctypes bindings + zero-copy arena view |
| `harness/torrent_meta.py` | local bencoded `.torrent` parser |
| `harness/tracker_ffi.py` | ctypes bindings for `../torrent-tracker` DHT/tracker helpers |
| `harness/harness.py` | tracker discovery + reassembly + SHA-1 verify |
| `tests/test_localseed.py` | end-to-end test against a local libtorrent seed |

## Build

```sh
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build           # -> build/libtorrentpeer.so
```

Options: `-DPEER_NATIVE=OFF` (portable build, no `-march=native`),
`-DPEER_ASAN=ON` (AddressSanitizer/UBSan).

## Test

The end-to-end test creates a random file, seeds it with libtorrent on
localhost, downloads every piece through the C peer, and asserts the bytes match
(it builds the library automatically if needed):

```sh
python tests/test_localseed.py      # standalone
# or, if pytest is available:  python -m pytest tests/
```

Run the full path under ASan:

```sh
cmake -S . -B build-asan -DPEER_ASAN=ON && cmake --build build-asan
ASAN_OPTIONS=detect_leaks=0 \
  LD_PRELOAD=$(gcc -print-file-name=libasan.so) \
  python tests/test_localseed.py
```

For offline interoperability against multiple seed clients, use the Docker
Compose harness in `interop/`. It creates a private trackerless fixture torrent,
starts libtorrent, Transmission, and aria2 seeders on an internal-only Docker
network, then verifies that this engine can download and hash-check the fixture
from each one:

```sh
docker compose -f interop/docker-compose.yml up --build \
  --abort-on-container-exit --exit-code-from runner
```

## Use against a real torrent

```sh
# explicit peer (skip tracker):
python harness/harness.py file.torrent --peer 1.2.3.4:51413 -o out.bin
# or discover peers via DHT and the torrent's HTTP/UDP trackers:
python harness/harness.py file.torrent -o out.bin
```

Tunables: `--slots N` (arena depth, 16 KiB each), `--pipeline N` (cap on
outstanding requests; the live depth adapts to the BDP under this cap),
`--timeout SECONDS`. Per-block request timeout and an optional `SO_RCVBUF`
override are `peer_config` fields (`request_timeout_ms`, `recv_buffer_bytes`).

## Status

Verified byte-for-byte on local-seed transfers up to 256 MiB; ~400+ MB/s over
loopback with the arena (16 MiB) far smaller than the file, i.e. with
credit-based backpressure fully engaged. Clean under ASan/UBSan. The test suite
covers full download + byte verification, a **partial seed** (peer downloads
only the pieces the remote has), **priority ordering**, and — via a raw-socket
mock peer — **request-timeout recovery** and **unsolicited-block rejection**.

### Known gaps / next steps

- Real-swarm auto-discovery is still hit-or-miss: many announced peers are
  unreachable, behind NAT, or only usable with a different transport/encryption
  mode. Use `harness/swarm_download.py --encryption` and `--utp` to choose the
  dial mode.
- Single peer / single net thread. Aggregate throughput needs a multi-peer
  connection manager with a shared piece picker (enables endgame mode + CANCEL).
- io_uring registered buffers + multishot recv would give a true zero-copy,
  near-syscall-free receive path (drops into `src/net.c`).
- Optional in-library SHA-NI hashing on a worker pool, so wire speed isn't held
  hostage to Python hash throughput at multi-GB/s.
- Wider peer reach: µTP, MSE/PE encryption, IPv6, Fast Extension (BEP-6).