Server Deployment

This section covers how to deploy the Parsec server.

Requirements

Preamble

The Parsec server depends on the following external components in order to work properly:

A PostgreSQL database to store the metadata.
An S3 object storage to store the data blocks.

Note

The Parsec server need access to an S3 object storage-like service, not necessarily AWS S3
An SMTP server for sending emails.
A TSL/SSL server certificate for HTTPS communication with the clients.
(Optional) A Sentry DSN for telemetry report.

Warning

For security reasons, the installation of these components is outside the scope of this guide. In order to securely configure and manage them, please refer to their official documentations.

This guide provides instructions for quickly settings up mock-ups or basic installs of those components. Keep in mind that these instructions are provided for convenience and should not be used in production.

Parsec testing infra

With Docker

Generating the required TLS certificates

For this guide, the required TLS certificates will be generated with a custom Certificate Authority (CA) created for this purpose.

#!/usr/bin/env bash

function generate_cert_conf() {
    local name=$1
    local san=$2

    echo "Generating $name.crt.conf"

    cat << EOF > $name.crt.conf
[req]
distinguished_name = req_dist_name
req_extensions = req_ext
prompt = no

[req_dist_name]
CN = $name

[req_ext]
subjectAltName = $san
EOF
}

function generate_certificate_request() {
    local name=$1
    echo "Generate certificate request $name.csr"
    openssl req -batch \
        -new -sha512 -noenc -newkey rsa:4096 \
        -config $name.crt.conf \
        -keyout $name.key -out $name.csr
}

function sign_crt_with_ca() {
    local ca_crt=$1
    local ca_key=$2
    local name=$3

    echo "Sign certificate request $name.crt"

    openssl x509 -req -in $name.csr \
        -CA $ca_crt -CAkey $ca_key \
        -extfile $name.crt.conf \
        -extensions req_ext \
        -CAcreateserial -out $name.crt \
        -days 10 -sha512
}

if [ ! -f custom-ca.key ]; then
    echo "Generate a mini Certificate Authority"
    openssl req -batch \
        -x509 -sha512 -nodes -days 10 -newkey rsa:4096 \
        -subj "/CN=Mini Certificate Authority" \
        -keyout custom-ca.key -out custom-ca.crt
fi

for service in parsec-{s3,server,proxy}; do
    if [ ! -f $service.crt.conf ]; then
        generate_cert_conf $service DNS:$service,DNS:localhost,IP:127.0.0.1
    fi

    # Generate key + csr if missing or if the key is older than the conf
    if [ ! -f $service.key ] || [ $service.key -ot $service.crt.conf ]; then
        generate_certificate_request $service
    fi

    # Generate crt if missing or if it's older than the csr or the custom CA
    if [ ! -f $service.crt ] || [ $service.crt -ot $service.csr ] || [ $service.crt -ot custom-ca.key ]; then
        sign_crt_with_ca custom-ca.{crt,key} $service
    fi
done

if [ "$(stat -c %g parsec-server.key)" -ne 1234 ]; then
    echo "Changing group id of parsec-server.key to 1234"
    sudo chown $USER:1234 parsec-server.key
fi

if [ "$(stat -c %a parsec-server.key)" -ne 640 ]; then
    echo "Changing permission of parsec-server.key to 640"
    chmod 640 parsec-server.key
fi

The script will:

Generate the CA key & self-signed certificate (custom-ca.{key,crt}).
For parsec-s3 and parsec-server services:
1. Generate the service key & Certificate Signing Request (CSR) parsec-{service}.{key,csr}.
2. Generate the certificate using the CSR and the CA.
For the service parsec-server:
1. Change the group id of the key file to 1234 (That is the GID used by the parsec-server container).
2. Change the file mode to give read permission to the group 1234.
Note

This is required because docker-compose does not allow to mount the file with the correct permissions in the container.

Warning

For production, you should use certificates issued from a trusted CA

The env files

We split the configuration of the parsec server into multiple env files so it’s simpler to understand how to configure each part.

The administration token

To be able to perform admin tasks (like creating an organization) on the server, an administration token is required. Below you will find a simple script to generate a token:

#!/usr/bin/env bash
set -euo pipefail

ENV_FILE=parsec-admin-token.env
if [ ! -f $ENV_FILE ]; then
    TOKEN=$(openssl rand 63 | base64 --wrap=86)
    echo "PARSEC_ADMINISTRATION_TOKEN=$TOKEN" > $ENV_FILE
    echo "Parsec administration token generated in: $ENV_FILE"
else
    echo "Parsec administration token already exists in: $ENV_FILE"
fi

The script will generate a random token (openssl rand 63 | base64 --wrap=86) and create the env file parsec-admin-token.env

Note

The step TOKEN=$(openssl rand 63 | base64 --wrap=86) could also be replaced by a value generated by a password-generator for example.

It doesn’t need to be encoded in base64 (we encode it in the script just to have printable characters).

Database configuration

Create the file parsec-db.env with the following content to configure the access to the PostgreSQL database:

# The Database url.
PARSEC_DB=postgresql://DB_USER:DB_PASS@parsec-postgres:5432/parsec
# The minimum number of connections to the database.
PARSEC_DB_MIN_CONNECTIONS=5
# The maximum number of connections to the database.
PARSEC_DB_MAX_CONNECTIONS=7

SMTP configuration

Create the file parsec-smtp.env to configure the access to the SMTP server (mailhog in this case).

We need to set the connection informations, the sender information, in which the default language the emails are sent:

# The SMTP host to use for sending email.
PARSEC_EMAIL_HOST=parsec-smtp
# The port to use when connecting to the SMTP server.
PARSEC_EMAIL_PORT=1025
# The username to use for the SMTP server.
PARSEC_EMAIL_HOST_USER=SMTP_USER
# The password to use for the SMTP server.
PARSEC_EMAIL_HOST_PASSWORD=SMTP_PASS
PARSEC_EMAIL_SENDER=parsec@test.xyz

# PARSEC_EMAIL_USE_SSL
# PARSEC_EMAIL_USE_TLS
PARSEC_EMAIL_LANGUAGE=en

S3 service configuration

Create the file parsec-s3.env with the following content to set the URL for the S3-like service:

# The blockstore URL.
# Can be S3, Switch or POSTGRESQL URL
PARSEC_BLOCKSTORE=s3:parsec-s3\:9000:region1:parsec:S3_ROOT_USER:S3_ROOT_PASS

Note

We need to escape the : with a \ when specifying the port of the service.

Parsec server configuration

Create the file parsec.env with the following content to configure the parsec-server service:

# Host to listen to.
PARSEC_HOST=0.0.0.0

# The SSL key file.
PARSEC_SSL_KEYFILE=/run/secrets/parsec-pem-key

# The SSL certificate file.
PARSEC_SSL_CERTFILE=/run/secrets/parsec-pem-crt

# The granularity of Error log outputs.
PARSEC_LOG_LEVEL=WARNING

# The log formatting to use (`CONSOLE` or `JSON`).
PARSEC_LOG_FORMAT=CONSOLE

# The log file to write to (default to `stderr`).
# PARSEC_LOG_FILE

# List of proxy addresses to trust
PARSEC_PROXY_TRUSTED_ADDRESS=parsec-proxy

# The URL to reach Parsec server.
PARSEC_SERVER_ADDR=parsec3://example.com

# Keep SSE connection open by sending keepalive messages to client (pass <=0 to disable).
PARSEC_SSE_KEEPALIVE=30

# Sentry Data Source Name for telemetry report.
# PARSEC_SENTRY_DSN

# Sentry environment for telemetry report.
PARSEC_SENTRY_ENVIRONMENT=production

Note

To see the full list of environment variables that you can use to configure Parsec, you can run:

python -m parsec run --help

Look for the sections [env var: VARIABLE] next to each configuration option. For example:

--administration-token TOKEN    Secret token to access the Administration API
                                [env var: PARSEC_ADMINISTRATION_TOKEN; required]

The docker-compose file

You can use the following docker-compose file (parsec-server.docker.yaml) to deploy the Parsec server for testing:

services:
  parsec-proxy:
    depends_on:
      - parsec-server
    image: nginx:1.27-alpine
    container_name: parsec-proxy
    ports:
      - 443:443
      - 80:80
    volumes:
      - ./parsec-nginx.conf:/etc/nginx/nginx.conf:ro
      - ./parsec-proxy.crt:/certs/proxy.crt:ro
      - ./parsec-proxy.key:/certs/proxy.key:ro

  parsec-postgres:
    image: postgres:14.10-alpine
    container_name: parsec-postgres
    environment:
      POSTGRES_USER: DB_USER
      POSTGRES_PASSWORD: DB_PASS
      POSTGRES_DB: parsec
    ports:
      # Expose PostgreSQL to localhost
      - 127.0.0.1:5432:5432
    volumes:
      - parsec-db-data:/var/lib/postgresql/data

  parsec-s3:
    image: quay.io/minio/minio:RELEASE.2024-09-13T20-26-02Z
    container_name: parsec-s3
    command: server --console-address ":9090" --certs-dir /opts/certs /data
    environment:
      MINIO_ROOT_USER: S3_ROOT_USER
      MINIO_ROOT_PASSWORD: S3_ROOT_PASS
    ports:
      # Admin console exposed to https://127.0.0.1:9090
      - 127.0.0.1:9090:9090
      # Expose S3 API to localhost
      - 127.0.0.1:9000:9000
    volumes:
      - parsec-object-data:/data
      - ./parsec-s3.key:/opts/certs/private.key:ro
      - ./parsec-s3.crt:/opts/certs/public.crt:ro
      - ./custom-ca.crt:/opts/certs/CAs/ca.test.crt:ro

  parsec-smtp:
    image: mailhog/mailhog:v1.0.1
    container_name: parsec-smtp
    ports:
      - 1025:1025
      # Web interface exposed to http://127.0.0.1:8025
      - 127.0.0.1:8025:8025

  parsec-server:
    depends_on:
      - parsec-smtp
      - parsec-s3
      - parsec-postgres
    image: ghcr.io/scille/parsec-cloud/parsec-server:3.2.1-a.0+dev
    container_name: parsec-server
    env_file:
      - parsec.env
      - parsec-s3.env
      - parsec-db.env
      - parsec-smtp.env
      - parsec-admin-token.env
    environment:
      AWS_CA_BUNDLE: /run/secrets/mini-ca-crt
    secrets:
      - mini-ca-crt
      - parsec-pem-crt
      - parsec-pem-key
    ports:
      - 127.0.0.1:6777:6777

volumes:
  parsec-db-data: {}
  parsec-object-data: {}

secrets:
  parsec-pem-crt:
    file: ./parsec-server.crt
  parsec-pem-key:
    file: ./parsec-server.key
  mini-ca-crt:
    file: ./custom-ca.crt

It will setup 4 services:

Service name	Description
`parsec-postgres`	The PostgreSQL database
`parsec-s3`	The Object Storage service
`parsec-smtp`	A mock SMTP server
`parsec-server`	The Parsec server
`parsec-proxy`	A Nginx proxy server, used as an example to configure a reverse proxy. Learn more about using parsec behind a reverse proxy

Starting the services

The docker containers can be started as follows:

docker compose -f parsec-server.docker.yaml up

Initial configuration

On the first start, a one-time configuration is required for the database and s3 services.

Applying the database migration

(optional) Check that the database is accessible with:

set -a
source parsec-db.env
docker exec -t parsec-postgres psql 'postgresql://DB_USER:DB_PASS@0.0.0.0:5432/parsec' -c "\conninfo"

Note

You should have something like display on your console:

You are connected to database "parsec" as user "parsec" on host "0.0.0.0" at port "5432".

To bootstrap the database we just need to apply the migrations with:

docker compose -f parsec-server.docker.yaml run parsec-server migrate

Create the S3 Bucket

Access the console at https://127.0.0.1:9090, you will need to use the credential specified in the docker-compose file at services.parsec-s3.environment.MINIO_ROOT_{USER,PASSWORD}.

Go to https://127.0.0.1:9090/buckets/add-bucket to create a new bucket named parsec with the features object locking toggled on.

After that you will need to restart the parsec-server (that likely exited because it wasn’t able to access the S3 bucket):

docker compose -f parsec-server.docker.yaml restart parsec-server

Test the SMTP configuration & server

You can test mailhog with:

#!/usr/bin/env bash

set -a
source parsec-smtp.env

curl \
    --url "smtp://127.0.0.1:$PARSEC_EMAIL_PORT" \
    --user "$PARSEC_EMAIL_HOST_USER@localhost:$PARSEC_EMAIL_HOST_PASSWORD" \
    --mail-from $PARSEC_EMAIL_SENDER \
    --mail-rcpt rcpt@test.com \
    --upload-file <(date --rfc-3339=seconds)

You can then check if the email is present in the web interface at http://127.0.0.1:8025

On Bare Metal

If you don’t want to install Parsec server via Docker, you can install it on bare metal.

Requirements

Python v3.12 with pip and venv modules
A S3 like object storage endpoint.
A postgres-14 database endpoint.
A TLS certificate & key for the server.

Configure the environment

Configure the env files, follow the env files.

Installation

Set up a virtual env:

python -m venv venv

Configure your shell to use the virtual env:

source venv/bin/activate

Install parsec-server

The parsec-server is available as a python package hosted on https://pypi.org/project/parsec-cloud/.

You need to install it with the extra backend enabled.
python -m pip install 'parsec-cloud==3.2.1-a.0+dev'

Prepare the database by applying the migrations:

source venv/bin/activate
set -a
source parsec-db.env
python -m parsec migrate

Start the server

Create a wrapper script run-parsec-server

#!/bin/bash

# Load the virtualenv.
source venv/bin/activate

# Load the env file into the environment table.
set -a
source parsec-admin-token.env
source parsec-db.env
source parsec-smtp.env
source parsec-s3.env
source parsec.env
set +a

# Start the parsec server.
python -m parsec run

Execute the wrapper script run-parsec-server

Note

To run the wrapper with only run-parsec-server you need to have set the executable mode on the script file (chmod +x run-parsec-server). Otherwise, you need to execute it with the bash shell (bash run-parsec-server).

Start using Parsec server

Create the first organization

set -a
source parsec-admin-token.env
export SSL_CAFILE=$PWD/custom-ca.crt
parsec-cli organization create --addr parsec3://127.0.0.1:6777 <orgname>

Note

Change <orgname> to the organization’s name that suit you.

Save the link after Bootstrap organization url: you will need it to create the first user (owner) of the organization.

Add the first user to the organization

First, start parsec with the custom CA:

export SSL_CAFILE=$PWD/custom-ca.crt
parsec

After that go to Menu/Join an organization (or CTRL+O) and paste the link from before (should already be filled in the text field). Follow the instructions to create the first user of the organization.

Running behind a reverse proxy

To run Parsec behind a reverse proxy you will need to add the option --proxy-trusted-address or set the environment variable PARSEC_PROXY_TRUSTED_ADDRESS to the address of the reverse proxy (e.g.: localhost).

If this option is not set, the gunicorn/uvicorn FORWARDED_ALLOW_IPS environment variable is used, defaulting to trusting only localhost if absent.

Tip

You can provide multiple addresses by separating them with a comma.

Example: Use the option --proxy-trusted-address '::1,10.0.0.42' will trust the address ::1 and 10.0.0.42

An example of a reverse proxy configuration for nginx can be found in the docker compose file:

  parsec-proxy:
    depends_on:
      - parsec-server
    image: nginx:1.27-alpine
    container_name: parsec-proxy
    ports:
      - 443:443
      - 80:80
    volumes:
      - ./parsec-nginx.conf:/etc/nginx/nginx.conf:ro
      - ./parsec-proxy.crt:/certs/proxy.crt:ro
      - ./parsec-proxy.key:/certs/proxy.key:ro

The provided configuration for nginx is:

events {
    worker_connections 128;
}


http {
    server {
        listen 80;
        listen 443 ssl;
        server_name example.com;

        ssl_certificate /certs/proxy.crt;
        ssl_certificate_key /certs/proxy.key;

        location / {
            proxy_pass https://parsec-server:6777;

            # Add X-Forwarded headers to the proxied request
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-Host $host;
            proxy_set_header X-Forwarded-Port $server_port;

            # Remove the Forwarded header
            proxy_set_header Forwarded "";

            # Overwrite the Host header
            proxy_set_header Host example.com;
        }
    }
}

It configures Nginx to serve the domain example.com by listening on port 80 and 443, and proxy the requests to the Parsec server.

The important takeaways are:

Set the headers X-Forwarded-For, X-Forwarded-Proto, X-Forwarded-Host and X-Forwarded-Port.

Note

Currently, Parsec only uses the X-Forwarded-For and X-Forwarded-Proto headers. But it better to overwrite all of them to avoid any issue.
Remove the header Forwarded.

Note

The Forwarded header (RFC-7239) is not used by Parsec, but it may be in the future.
Set the header host to the accessible address. Here we force the value to be example.com, but you can set it to $host like for X-Forwarded-Host.