/dev/log for hypertube

Personal movie / anime streaming site where the content are obtained via bittorrent protocol built with nextjs and django. Mega thanks for laian for helping on this project

Design, Demo and screenshots

User is able to create an account with their email or a third party app

Discover screen inspired by netflix

Video streaming in action

Comment section for videos

BitTorrent protocol

This protocol is well known for peer-to-peer file sharing, and the documentation can be found here and here. This re-iteration of the procol in this writeup is to further abstract the protocol, making it easier for future reference.

Every node / machine in the network is considered a peer. A peer can be a seeder (file uploader) or a leecher (file downloader) depending on what they want to acheive in the network.

The torrent file

The .torrent file or Metainfo file is a file that contains information on the underlying file to download such as the length, filename, hash, pieces, piece length and so on.

NOTE: A piece here refers to a part of the actual files content. The pieces property in the .torrent file is the result of concatnation of the hashes of the specific parts of files, not the hash of the actual file itself. piece length is just a number of how big a piece should be

NOTE: A piece length is usually a power of 2 number, so the final piece may contain extra number of bytes from the actual file. what in trying to say here is that the bytes in the actual file is not exactly divisible by the fixed length pieces as defined in the .torrent file

The .torrent file also contains information about its announce URL of its tracker(s), which will be an important information for file acquisition.

For leechers, the .torrent file is the first step to downloading files. They can acquire this file from a public database or shared from friends or family.

For seeders, the generation of the .torrent file is the last step of sharing files to the network.

The tracker

The tracker is an HTTP over TCP/UDP service that keeps tracks of what peers have what files. Leechers will query trackers to get a list of peers to download from and seeders will query to a tracker to add itself as a peer for a certain file.

The query to trackers is also known as announce requests. Periodically, clients would make announce request to update the tracker on the state of the current upload / download. The trackers will maintain their record internally for all of the announce requests receive.

The announce request will contain information about the number of bytes uploaded / downloaded and the number of bytes left in the upload / download. For leechers, it also accepts information on how many peers would the leecher like to expect from the response of the query.

The response will contain information such as as number of seeders / leechers, as well as a list of peers the leechers can start downloading from.

The tracker will also support a /scrape endpoint which just shows the current status for any given torrent.

There are some public trackers out there, however you can also deploy your own tracker for the same purposes as long the convention is followed strictly.

Download messaging

Once the peers are determined, the leecher can initiate the download by sending messages to the peer using formatted messages.

The sequence starts with a handshake, and the leecher will send request messages requesting for data, the peer will then response with piece messages containing raw bytes of the requested file. The full protocol can be viewed here under section Peer wire protocol (TCP).

Video streaming

The previous sections described the acquisition of the video file, the following processes will involve the delivery part of the service, where we still stream the video source to the client.

Though the requirements specify that we need to use WebRTC, my persoal opinion thinks that using RTMP is a better option, considering the former is used mainly for realtime video calls optimizing processing speed with the cost of quality over the latter which is optimized for high quality streaming in exchange for slower processing speed.

Below is the video streaming flow we have implemented with a simple websocket server as the signalling server.

Mkv conversion

The mp4 files are converted to mkv on the fly using ffmpeg-python, a ffmpeg wrapper for python.

NOTE: This is not required anymore as MediaStream API is compatible for all video files within our obsevation

NOTE: The codecs supported by webRTC can be referred here, the package we use to initialize streams (aiortcs mediaplayer) conviniently converts the files to the correct codecs for us

FFmpeg is a universal media converter, it can read from a variety of media inputs such as a video file, a webcam, a live streaming service to do some processing and output it to various media output devices / sources.

By being able to create and manipulate certain ffmpeg components, the user is able to do video / audio processing within the capabilities of ffmpeg. The key components of ffmpeg are follows

• Demuxer aka "demultiplexers", they take a media source and extracts metadata and individual streams of the media source into encoded (compressed) packets for the video / audio / subtitle streams

• Decoder Takes encoded packets as inputs and decodes them into raw frames (data) to pass on to other components

• Filtergraph This component is responsible for transforming raw packets into whatever the user desires and the output of this component are the modified raw packets

• Encoder The reverse of the decoder, takes in raw packets and compress them into packets of their respective codecs / formats. The output of this component usually goes to the Muxer

• Muxer Reverse of the muxer, takes the packets outputted by the encoder to combine and send them to an output source.

External search APIs

To facilitate the discovery feature, we offloaded the search function to two external APIs, OMDB and TMDB. Both of these offer search and filter functions, however we use OMDB as our source of truth and the TMDB acts as a backup complementing the contents which OMDB is lacking. OMDB results will always be the priority in the case of a conflict.

Torrent file matching

Once a movie name is obtained, a query has to be made to an external magnet database to obtain the magnet link / metainfo. We are using yts search endpoint for this purpose. Once we have a list of magnets and their titles to go through, we used groq LLM to match the most suitable title for us from the list of titles.

i18n

i18n is used for internationalization of static web text content. It uses the nextjs curated next-intl package to manage locales and translations. All text contents and their translations are stored in frontend/messages/{language}.json.

Requirements

• Node.js: v20.10.0 or compatible
• Python: v3.10 or compatible
• ffmpeg v4.4.2

Environment

Node.js Installation

To install the correct version of Node.js on your system, use nvm (Node Version Manager) to manage your Node.js versions.

1. First, install nvm if you haven't already. You can follow the installation instructions from the official nvm repository.

2. Once nvm is installed, run the following commands to install and use Node.js v20:

   nvm install 20
   nvm use 20

Python Installation with Conda

To install and set up Python 3.11.1 (or compatible) using Conda, follow these steps:

1. Install Conda: If you don't have Conda installed, you can install it by downloading the Miniconda or Anaconda distribution, which includes Conda.

2. Create a new Conda environment with Python 3.10:

   conda create -n hypertube python=3.10

   Replace hypertube with the name you wish to give to your environment.

3. Activate the environment:

   conda activate hypertube

4. To confirm that the correct version of Python is installed, run:

   python --version

   This should display Python 3.10 (or a compatible version, depending on what you installed).

Instructions

1. Credentials setup

To make sure the required credentials are in place

cp .env.example .env

---

2. Frontend Setup

To get the frontend up and running, follow these steps:

1. Navigate to the frontend directory:

   cd frontend

2. Install the necessary dependencies and start the development server:

   npm install && npm run dev

   • npm install will install all required dependencies from the package.json file.
   • npm run dev will start the development server (make sure you have the necessary environment variables and configurations set).

   The frontend should now be available at http://localhost:3000 (or another port depending on your configuration).

---

3. Backend Setup

To set up the backend, follow these steps:

1. Navigate to the backend directory:

   cd backend

2. Install the required Python dependencies:

   If you're using a virtual environment (recommended):

   pip install -r ../requirements.txt

   This will install all necessary packages listed in requirements.txt.

3. Run the backend server:

   python manage.py migrate # run table migrations
   python manage.py createsuperuser # create super user, needed for admin access
   python manage.py runserver

   This will start the Django development server. By default, the admin page will be available at http://127.0.0.1:8000/admin.

   • If you're using a different configuration (e.g., Docker or a custom port), be sure to adjust the command accordingly.

Feature suggestions

1. Multifile torrents for tv series with multiple episodes
2. Custom title matching engine for torrent file query
3. Direct peer to peer with webrtc just like how webtorrent does it, bypassing the signalling requirement(?)
4. Optimizations for larger mkv to support multiple streams
5. More options to apply filters on the fly
6. More robust torrent state handling