Great work! Just an FYI, you might want to limit the dynamic allocation size in the bencode decoder: since it's untrusted input (either from torrent or announce), a malicious input could DoS the client by requesting extremely large allocations during string parsing. A good upper bound could be the remaining length of the input, as a well formed torrent can't contain a string longer than the rest of the file.
Neat! There is this challenge on codecrafters that guides you through the process a little, provides tests and such, I've played around with it a bit during a free month they had, was fun:
Windows 7 support is one reason to stick to older GoLang releases. A project in Go 1.21.4 or earlier will work on every Windows release and any computer made since 2009, whereas a version bump to v1.21.5 means it will only work on more recent computers and Win10 and 11 for no benefit.
I think this is a reasonable take. Yes, people shouldn't be running Windows 7 as their daily driver. But if you can support it at basically no effort and without sacrifices that is the right thing to do. Supporting more platforms is a good thing, even if that platform is an old Windows version instead of an Amiga
The Go team isn't making new versions just for fun. Each version since 1.21 has had improvements. Especially the fix/change to for loop variables in 1.22 is very nice to have, and helps preventing to write bugs.
If there's a reasonable expectation that users will use outdated platforms, it makes sense to support them. If there is no such expectation at all, why would one forego the improvements to the language and tooling.
what did you refer for building this? did you read the protocol or did you go through other implementations? curious because it always makes me wonder how people build stuff like this from scratch
Interestingly (or maybe not) I started writing a Bittorrent client last week, also in Go. I don't really like AI/LLMs for coding, and my goal is to really learn from it. Not to just deliver some piece of code that an AI wrote...
So the first step is to figure out if there is an actual spec for the protocol. Well, in the case of Bittorrent there is a spec, but it's kinda... brief. Basically, this is the spec: https://www.bittorrent.org/beps/bep_0003.html
It's very important to break the work up in tiny steps that each give some kind of a result. E.g. I started with parsing .torrent files, for which you have to implement bencoding.
That was interesting, because I downloaded the Arch Linux .torrent and found out it was basically incorrect: it has a url-list key which isn't mentioned in the spec. Then you have to go down a rabbit hole of some research trying to find out what the heck is up with that (turned out to be https://www.bittorrent.org/beps/bep_0019.html). Finally, was able to successfully parse the Debian Linux .torrent file.
Then came implementing the tracker Announce HTTP request and the peer protocol. The peer protocol was tricky, and that's when an experimentation mindset comes in handy. I stripped the announce URL from the Debian torrent and loaded it up in KTorrent, that way it wasn't able to find any peers. I used the "Add peer" option to add my client so I could see what messages it was sending me, and alongside I would implement sending those to KTorrent messages too. A lot of trial and error and debugging.
I have to confess that from time to time I had to ask ChatGPT about some small protocol details because they just didn't show up in the spec. There's also a lot of details that clients have implemented over time but which aren't really documented well. Mostly algorithms for which pieces to download, which peers to choose, how to "choke"/"unchoke", etc, is not well-defined. But web searches help out a lot.
I'm now at a point where I can fully download and upload a torrent with my local KTorrent, so the next steps are to actually get peers from the tracker, and some algorithm to download pieces and store them in a file.
Hope this gives some insights. Let me know if you have specific questions about the process.
Cogent Core also looks neat, but I didn't have the time to play with it before I switched over to using the Odin programming language instead of Go https://www.cogentcore.org/core
I personally had nothing but issues with Fyne (especially in regard to performance, across multiple computers and operating systems), but it's probably the most popular option https://fyne.io
This is cool! I’ve been thinking about something like this as well. How hard was it, and do you have a sense for how “complete” it is? Does it handle DHT and Magnet and all the crazy NAT traversal stuff?
I’m guessing the main obstacle for me has always been that I’m not sure what the complete list of features is to have a client that will just work for the majority of torrents in the wild. It seems like there are dozens of protocols associated with torrenting and I don’t even know what the full list is much less what each does.
How hard it is depends a lot on your experience, how comfortable you are with the programming language, and if you have an experimentation mindset. To offer a perspective, I also decided to build a Bittorrent client in Go last week, and I'm about 80% of the way of the posted project in a week. But then again I have a lot of experience with Go, extensive knowledge about protocols and networking, and experimentation has become a kind of second nature for me.
There's a lot of stuff "around" Bittorrent, like protocol extensions. But in my experience so far, many clients have different levels of support. E.g. they would try communicating using Protocol Encryption (obfuscation, actually) and if it doesn't work fall back to the regular protocol (if configured in the settings).
If you just want to download a Linux distribution (e.g. Debian) using the .torrent file on their website, it's relatively straight forward. It has just one file, there's a tracker, plenty of peers using the regular protocol, and you could get away not listening for connections when you're behind a NAT.
Naturally when you get into the grey area of content, then you'd need to parse magnet links, use DHT to discover peers, and perhaps you'd want to implement UPnP to try and automatically map a port on your modem. But that is also an iterative process, you can just build out one by one. The more features, the more peers you'll be able to discover and have successful exchanges with.
it was challenging for sure. Took me almost a month to get acquainted with the protocol, how bencoding works etc, build a mental model and then eventually writing code.
In my experience, magnet is pretty straightforward. Dht is quite the rabbit hole, and it might be difficult finding clients that support dht (not everyone does)
I am not the one who created the post, but currently it doesn't seem to be usable as a library because most of the code is in a directory called "internal", which isn't accessible when using the repository as a dependency in another project.
pretty solid attempt. but no mention of crash recovery, encrypted peer handshakes, or even basic uTP support. no idea how it behaves with NAT either. no memory guardrails during parsing, feels risky in real swarm. not production safe without those. would love to see it modular too, like usable as lib not just cli. tracking roadmap would've helped too.
[0]: kaitai.io
Suggestion: Add a simple usage one liner in the README on how to actually download a .torrent file with it.
Suggestion: Bonus points if you add torrent.ParseFromUrlEveryone should do this for their own spiritual journey.
https://app.codecrafters.io/courses/bittorrent/overview
EDIT: It seems like it was deprecated 10 months ago
https://github.com/golang/go/issues/64622
There are always trade-offs/sacrifices.
The Go team isn't making new versions just for fun. Each version since 1.21 has had improvements. Especially the fix/change to for loop variables in 1.22 is very nice to have, and helps preventing to write bugs.
If there's a reasonable expectation that users will use outdated platforms, it makes sense to support them. If there is no such expectation at all, why would one forego the improvements to the language and tooling.
There's always a price to pay.
[1] https://github.com/piyushgupta53/go-torrent-client/blob/6130...
[2] https://go.dev/doc/modules/gomod-ref#go-notes
This brings me back to college. We did this as our final project for our networking class at Georgia Tech.
I've long lost the code for this, but the lessons learned have lived on :)
Projects like these are great ways to learn new languages too!
So the first step is to figure out if there is an actual spec for the protocol. Well, in the case of Bittorrent there is a spec, but it's kinda... brief. Basically, this is the spec: https://www.bittorrent.org/beps/bep_0003.html
There are some additional extensions here: https://www.bittorrent.org/beps/bep_0000.html
It's very important to break the work up in tiny steps that each give some kind of a result. E.g. I started with parsing .torrent files, for which you have to implement bencoding.
That was interesting, because I downloaded the Arch Linux .torrent and found out it was basically incorrect: it has a url-list key which isn't mentioned in the spec. Then you have to go down a rabbit hole of some research trying to find out what the heck is up with that (turned out to be https://www.bittorrent.org/beps/bep_0019.html). Finally, was able to successfully parse the Debian Linux .torrent file.
Then came implementing the tracker Announce HTTP request and the peer protocol. The peer protocol was tricky, and that's when an experimentation mindset comes in handy. I stripped the announce URL from the Debian torrent and loaded it up in KTorrent, that way it wasn't able to find any peers. I used the "Add peer" option to add my client so I could see what messages it was sending me, and alongside I would implement sending those to KTorrent messages too. A lot of trial and error and debugging.
I have to confess that from time to time I had to ask ChatGPT about some small protocol details because they just didn't show up in the spec. There's also a lot of details that clients have implemented over time but which aren't really documented well. Mostly algorithms for which pieces to download, which peers to choose, how to "choke"/"unchoke", etc, is not well-defined. But web searches help out a lot.
Also an honorable mention to https://wiki.theory.org/Main_Page which has some nice information.
I'm now at a point where I can fully download and upload a torrent with my local KTorrent, so the next steps are to actually get peers from the tracker, and some algorithm to download pieces and store them in a file.
Hope this gives some insights. Let me know if you have specific questions about the process.
Edit: ah, planned feature
My personal favourite is an ImGui wrapper https://github.com/AllenDang/giu
The most featureful is probably unison, although I'm uncertain if anyone uses it outside of their own project (https://gurpscharactersheet.com), meaning documentation will be sparse https://github.com/richardwilkes/unison
Gio uses a different way of thinking about GUIs, used by Tailscale and gotraceui https://gioui.org
Wails is great if you're familiar with development on the web https://wails.io
The GTK4 bindings also look nice https://github.com/diamondburned/gotk4
Cogent Core also looks neat, but I didn't have the time to play with it before I switched over to using the Odin programming language instead of Go https://www.cogentcore.org/core
I personally had nothing but issues with Fyne (especially in regard to performance, across multiple computers and operating systems), but it's probably the most popular option https://fyne.io
Last week I came across these Qt bindings: https://github.com/mappu/miqt
I have no personal experience with it yet, but am excited to try it out.
I’m guessing the main obstacle for me has always been that I’m not sure what the complete list of features is to have a client that will just work for the majority of torrents in the wild. It seems like there are dozens of protocols associated with torrenting and I don’t even know what the full list is much less what each does.
If you're interested, the main Bittorrent spec is actually quite small, you can easily read and understand it in an hour or so: https://www.bittorrent.org/beps/bep_0003.html
There's a lot of stuff "around" Bittorrent, like protocol extensions. But in my experience so far, many clients have different levels of support. E.g. they would try communicating using Protocol Encryption (obfuscation, actually) and if it doesn't work fall back to the regular protocol (if configured in the settings).
If you just want to download a Linux distribution (e.g. Debian) using the .torrent file on their website, it's relatively straight forward. It has just one file, there's a tracker, plenty of peers using the regular protocol, and you could get away not listening for connections when you're behind a NAT.
Naturally when you get into the grey area of content, then you'd need to parse magnet links, use DHT to discover peers, and perhaps you'd want to implement UPnP to try and automatically map a port on your modem. But that is also an iterative process, you can just build out one by one. The more features, the more peers you'll be able to discover and have successful exchanges with.
Magnet and DHT are yet to be added.
Using P2P networks in download-only mode, so called leeching or free-riding, is discouraged.