Initial commit: tracker and DHT protocol library

PLAN.md

@@ -0,0 +1,98 @@
+# torrent-tracker Protocol Plan
+
+Goal: build a tracker-side library that can serve the tracker protocols and
+extensions seen in real torrents, with protocol parsing separated from swarm
+storage and serving policy.
+
+## Implemented Foundation
+
+- BEP-3 HTTP announce parameters: `info_hash`, `peer_id`, `port`, `uploaded`,
+  `downloaded`, `left`, `compact`, `no_peer_id`, `event`, `numwant`, `key`,
+  `ip`, and `trackerid`.
+- BEP-23 compact peer responses: IPv4 `peers` and BEP-7-style IPv6 `peers6`.
+- BEP-48 HTTP scrape parsing and bencoded scrape responses.
+- BEP-15 UDP connect, announce, scrape, and error packet handling.
+- BEP-41 UDP announce extension parser, including concatenated URLData chunks.
+- Client-side HTTP announce/scrape query builders and bencoded tracker response
+  parsers.
+- Client-side UDP connect/announce/scrape packet builders and response parsers
+  with transaction-id validation.
+- BEP-5 DHT/KRPC message builders and parser for `ping`, `find_node`,
+  `get_peers`, `announce_peer`, response, and error packets.
+- BEP-32 IPv6 DHT compact node support through `nodes6` plus `want` flags.
+- In-memory swarm table keyed by 20-byte tracker infohash.
+- Announce handling for insert/update/stop, source-address-derived endpoints,
+  seed/leecher counts, completed-download scrape counts, no-self filtering,
+  randomized peer selection, `numwant` clamping, and stale peer pruning.
+
+## Tracker-Side Roadmap
+
+## Client-Side Roadmap
+
+1. DHT node/session manager:
+   Add a UDP event loop, node ID generation, transaction table, timeout/retry
+   logic, bootstrap nodes, and request dispatch for KRPC packets.
+
+2. DHT routing table:
+   Implement BEP-5 k-buckets, XOR distance sorting, good/questionable/bad node
+   state, bucket refresh, and persistence of known nodes between runs.
+
+3. DHT peer discovery:
+   Implement iterative `get_peers` lookup, token storage, `announce_peer`, and
+   integration with the tracker client result format so callers can consume
+   central tracker and DHT peers through one path.
+
+4. Tracker session manager:
+   Add a small client state machine that stores UDP connection IDs until expiry,
+   schedules announces at `interval`, handles started/completed/stopped events,
+   retries UDP with exponential backoff, and rotates across announce-list tiers.
+
+5. Transport adapters:
+   Wrap the protocol helpers with optional HTTP(S) and UDP socket code. Keep TLS
+   outside the core ABI, but provide a CLI/harness path that exercises real
+   network announces through the C helpers.
+
+6. Magnet and v2 metadata:
+   Keep accepting 20-byte tracker hashes: v1 SHA-1 infohash and BEP-52 truncated
+   SHA-256 for v2/hybrid torrents. Add metadata utilities or bindings so callers
+   can compute the right tracker hash from metainfo without duplicating logic.
+
+7. Client interop tests:
+   Compare C helper output against libtorrent/tracker_probe behavior for HTTP,
+   UDP, compact IPv4, `peers6`, scrape, failure responses, BEP-41 URLData, and
+   DHT KRPC packets observed from real nodes.
+
+## Tracker-Side Roadmap
+
+1. Network daemons:
+   Add HTTP/HTTPS and UDP listeners around the protocol core. HTTPS should live
+   behind a TLS terminator initially; the library API should not require a TLS
+   dependency.
+
+2. IPv6 and multi-homed behavior:
+   The store accepts IPv4 and IPv6 endpoints and prefers source addresses over
+   user-supplied `ip`. Add daemon-level tests for announcing the same peer over
+   multiple listen interfaces and returning family-appropriate UDP responses.
+
+3. Tracker policy:
+   Add hooks for private torrents, whitelist/auth tokens in HTTP query strings
+   and BEP-41 UDP URLData, per-swarm limits, rate limits, and abuse controls.
+
+4. Response selection:
+   Add stronger selection policy controls: per-family caps, seed/leecher mix,
+   deterministic test hooks for sampling, and configurable `tracker id`
+   generation/validation.
+
+5. Persistence and admin:
+   Add optional durable storage or snapshot/restore for long-running trackers,
+   metrics export, registered-torrent management, and scrape-cache policy.
+
+6. Compatibility matrix:
+   Build interop tests against libtorrent, Transmission, qBittorrent, aria2,
+   rtorrent, and opentracker for HTTP, UDP, compact, IPv6, scrape, and private
+   torrent behavior.
+
+7. Optional adjacent discovery:
+   DHT, PEX, and local peer discovery are peer-discovery mechanisms rather than
+   central tracker protocols. Keep them as separate modules if this project grows
+   into a full discovery stack.