This repository contains the arewecompressedyet.com website source code.

To run a local copy, you will need a copy of node and npm.

You will need node, npm, etc. For Ubuntu:

sudo apt install nodejs npm nodejs-legacy libicu-dev

You'll also need tools to build AV1 or other codecs being tested:

sudo apt install yasm libtool

First, run the ./setup.sh script. It will create directories needed for AWCY to run.

Next, create a configuration file called config.json.

Here is an example config.json:

{ "channel": "#daalatest",
  "have_aws": false,
  "port": 3000,
  "rd_server_url": "http://localhost:4000"
}

You will also need a file called secret_key which contains the key needed to use the website.

echo 'fake_password_to_compile' > secret_key

These commands will create the configuration files and install the node.js modules that get used by awcy. Open a node command line and run the following:

mkdir runs
cd www
npm install
cd ..
npm install
npm run tsc
node generate_list.js
npm run start

To run the server, execute the run_awcy.bat file or run the following in your command line:

  node awcy_server.js

Now you can open localhost:3000 with your browser to see your local version of the website.

For the website to build codecs, you need local checkouts of every codec. While the ./setup.sh script automatically adds AV1, you can add your own. For example:

git clone https://aomedia.googlesource.com/aom av1
ln -s av1 av1-rt

The AWCY web server manages the repositories and runs/ directory, and compiles and builds the codecs. Once this is done, it hands off the job of actually talking to all the AWS machines to rd_server.

To install rd_server, checkout the rd_tool repository in the same directory as awcy:

git clone https://github.com/tdaede/rd_tool.git

Then start the rd_server.py daemon:

./rd_server.py

The rd_server.py daemon listens on port 4000 by default.

More documentation on rd_server.py can be found in its README.

Workers are Linux machines accessible over ssh. They need to be the same architecture as the AWCY server.

A worker needs a AWCY work root, which has a directory structure as follows:

daalatool/
slot0/
vmaf/

The slot* directores are created by AWCY for jobs.

daalatool must be pre-populated by a clone of the daala repo, with tools built (make tools):

git clone https://github.com/xiph/daala.git daalatool
cd daalatool
./autogen.sh
./configure
make
make tools

vmaf is a clone of the Netflix VMAF repository, also built.

git clone https://github.com/Netflix/vmaf.git
cd vmaf/libvmaf
meson build --buildtype release
ninja -C build

In addition, a copy of the test media must be accessible to the worker.

The workers should be configured on the main server with a machines.json file, which contains the host, user and port (for ssh). In addition, it contains the number of cores (which controlls the number of slots), plus the work root (containing the above mentioned directores) and the test media path. For example:

[
  {
    "host": "localhost",
    "user": "xiph",
    "cores": 8,
    "port": 22,
    "work_root": "/home/xiph/awcy_temp",
    "media_path": "/home/xiph/sets/"
  }
]

Each worker must have a unique work root - usually local storage on the worker. One worker is intended to be one physical machine or VM. For testing purposes, one machine can act as multiple workers, but the work_root must be independent. The media path is read-only and can be shared, e.g. via NFS.

The machines should be accesible via passwordless SSH authentication. If needed, the SSH_PRIVKEY_FILE environment variable can inform rd_server.py of a custom private key (.pem format) to use for logging into workers.

The runs/ directory will contain all of the output files generated from a job. There is a info.json file that specifies what options were used by that particular run. Here is an example of an info.json file:

{"codec":"daala","commit":"","run_id":"2014-09-19T22-00-08.196Z","task":"video-1-short","nick":"AWCY","task_type":"video"}

There is also an output.txt file that contains the output of the rd_tool.py script.

After each run, a cache file called list.json file is generated by the generate_list.js script. This contains all of the info.json files, as an ordered list. This should probably be replaced by a "real" database at some point.

AWCY is deployed as a set of services managed by Docker Compose:

The awcy and rdtool images are large (3+ GB each) as they include build toolchains for all supported codecs.

• Docker Engine 24+ and Docker Compose v2 (docker compose)
• A conf/ directory with:
  • awcy.pem → symlink to your SSH private key (used by rdtool to reach workers)
  • machines.json → list of worker machines (see Setting up workers)
  • Caddyfile → Caddy configuration (reverse proxy to awcy:3000)
• Your SSH private key at daala.pem in the repo root (mode 600, owned by the host user)

Create a .env file in the repo root. Docker Compose reads it automatically:

# .env
AWCY_API_KEY=your_secret_api_key
IRC_CHANNEL=none                    # or e.g. #awcy-notifications
RUNS_DIR=/mnt/runs/betaruns         # host path for run output
MEDIA_DIR=/mnt/runs/sets            # host path for input video files
WORKER_SSH_PUBKEY=                  # only needed if running the worker container

Environment variables accepted by each service (with defaults):

awcy / rdtool shared:

• CONFIG_DIR=/data/conf: configuration files, SSH keys, set list, sqlite database
• CODECS_SRC_DIR=/data/src: codec git clones and build output
• RUNS_DST_DIR=/data/runs: job run output
• WORK_DIR=/data/work: per-job working directory
• MEDIAS_SRC_DIR=/data/media: input video files (from https://media.xiph.org/sets/)
• LOCAL_WORKER_ENABLED=false: set to true to use the host itself as a worker (requires openssh-server)

awcy only:

• AWCY_API_KEY=awcy_api_key: WebUI/API authentication key
• AWCY_SERVER_PORT=3000: awcy listening port
• RD_SERVER_PORT=4000: port awcy uses to reach rdtool
• IRC_CHANNEL=none: IRC notifications target channel

Build all images and start the stack:

docker compose build
docker compose up -d

On first startup, conf/config.json is generated automatically with the correct rd_server_url pointing at the rdtool container. The WebUI is available at https://arewecompressedyet.com (or http://localhost:3000 if running locally without Caddy TLS).

To follow logs:

docker compose logs -f

To stop:

docker compose down

Workers are remote Linux machines accessible over SSH. They are listed in conf/machines.json:

[
  {
    "host": "worker1.example.com",
    "user": "xiph",
    "cores": 8,
    "port": 22,
    "work_root": "/home/xiph/awcy_work",
    "media_path": "/home/xiph/sets"
  }
]

Each worker needs:

• The public key from conf/awcy.pub added to its ~/.ssh/authorized_keys
• A work root directory containing daalatool/, dump_ciede2000/, vmaf/, and slot*/ subdirectories
• A copy of (or NFS mount to) the test media at media_path

See Setting up workers for details on building the worker dependencies.

To spin up a local worker container (e.g. for testing), use the worker profile:

# Get the public key generated by rdtool on first boot
WORKER_SSH_PUBKEY=$(docker compose exec rdtool cat /data/conf/awcy.pub)

# Start one or more worker containers
WORKER_SSH_PUBKEY="$WORKER_SSH_PUBKEY" docker compose --profile worker up -d worker

Worker containers are excluded from the default docker compose up and must be started explicitly with --profile worker.

To verify the stack is working, submit a job using the awcy-builder-quicktest set from the WebUI. It completes in roughly one minute.

docker compose down

Bind-mounted data (conf/, src/, runs, media) remains on the host and is not removed.