Parsec Server installation
This guide covers the installation procedure for the Parsec server.
Requirements
Docker and the docker-compose plugin.
Parsec client (Linux).
Preamble
The Parsec server depends on the following external components in order to work properly:
A PostgreSQL database to store the metadata.
An S3 object storage to store the data blocks.
Note
The Parsec server need access to an
S3 object storage
-like service, not necessarilyAWS S3
An SMTP server for sending emails.
A TSL/SSL server certificate for
HTTPS
communication with the clients.(Optional) A Sentry DSN for telemetry report.
Avertissement
For security reasons, the installation of these components is outside the scope of this guide. In order to securely configure and manage them, please refer to their official documentations.
This guide provides instructions for quickly settings up mock-ups or basic installs of those components. Keep in mind that these instructions are provided for convenience and should not be used in production.
Parsec testing infra
With Docker
Generating the required TLS certificates
For this guide, the required TLS certificates will be generated with a custom Certificate Authority (CA) created for this purpose.
1#!/usr/bin/env bash
2
3function generate_cert_conf() {
4 local name=$1
5 local san=$2
6
7 echo "Generating $name.crt.conf"
8
9 cat << EOF > $name.crt.conf
10[req]
11distinguished_name = req_dist_name
12req_extensions = req_ext
13prompt = no
14
15[req_dist_name]
16CN = $name
17
18[req_ext]
19subjectAltName = $san
20EOF
21}
22
23function generate_certificate_request() {
24 local name=$1
25 echo "Generate certificate request $name.csr"
26 openssl req -batch \
27 -new -sha512 -noenc -newkey rsa:4096 \
28 -config $name.crt.conf \
29 -keyout $name.key -out $name.csr
30}
31
32function sign_crt_with_ca() {
33 local ca_crt=$1
34 local ca_key=$2
35 local name=$3
36
37 echo "Sign certificate request $name.crt"
38
39 openssl x509 -req -in $name.csr \
40 -CA $ca_crt -CAkey $ca_key \
41 -extfile $name.crt.conf \
42 -extensions req_ext \
43 -CAcreateserial -out $name.crt \
44 -days 10 -sha512
45}
46
47if [ ! -f custom-ca.key ]; then
48 echo "Generate a mini Certificate Authority"
49 openssl req -batch \
50 -x509 -sha512 -nodes -days 10 -newkey rsa:4096 \
51 -subj "/CN=Mini Certificate Authority" \
52 -keyout custom-ca.key -out custom-ca.crt
53fi
54
55for service in parsec-{s3,server}; do
56 if [ ! -f $service.crt.conf ]; then
57 generate_cert_conf $service DNS:$service,DNS:localhost,IP:127.0.0.1
58 fi
59
60 if [ ! -f $service.key ]; then
61 generate_certificate_request $service
62 fi
63
64 if [ ! -f $service.crt ]; then
65 sign_crt_with_ca custom-ca.{crt,key} $service
66 fi
67done
68
69if [ "$(stat -c %g parsec-server.key)" -ne 1234 ]; then
70 echo "Changing group id of parsec-server.key to 1234"
71 sudo chown $USER:1234 parsec-server.key
72fi
73
74if [ "$(stat -c %a parsec-server.key)" -ne 640 ]; then
75 echo "Changing permission of parsec-server.key to 640"
76 chmod 640 parsec-server.key
77fi
The script will:
Generate the CA key & self-signed certificate (
custom-ca.{key,crt}
).For
parsec-s3
andparsec-server
services:Generate the service key & Certificate Signing Request (CSR)
parsec-{service}.{key,csr}
.Generate the certificate using the CSR and the CA.
For the service
parsec-server
:Change the group id of the key file to
1234
(That is the GID used by theparsec-server
container).Change the file mode to give read permission to the group
1234
.
Note
This is required because
docker-compose
does not allow to mount the file with the correct permissions in the container.
Avertissement
For production, you should use certificates issued from a trusted CA
The env files
We split the configuration of the parsec server into multiple env files so it’s simpler to understand how to configure each part.
The administration token
To be able to perform admin tasks (like creating an organization) on the server, an administration token is required. Below you will find a simple script to generate a token:
1#!/usr/bin/env bash
2
3ENV_FILE=parsec-admin-token.env
4if [ ! -f $ENV_FILE ]; then
5 TOKEN=$(openssl rand 63 | base64 --wrap=86)
6 echo "PARSEC_ADMINISTRATION_TOKEN=$TOKEN" > ENV_FILE
7 echo "Parsec administration token generated in: $ENV_FILE"
8else
9 echo "Parsec administration token already exists in: $ENV_FILE"
10fi
The script will generate a random token (openssl rand 63 | base64 --wrap=86
) and create the env file parsec-admin-token.env
Note
The step TOKEN=$(openssl rand 63 | base64 --wrap=86)
could also be replaced by a value generated by a password-generator for example.
It doesn’t need to be encoded in base64
(we encode it in the script just to have printable characters).
Database configuration
Create the file parsec-db.env
with the following content to configure the access to the PostgreSQL database:
1# The Database url.
2PARSEC_DB=postgresql://DB_USER:DB_PASS@parsec-postgres:5432/parsec
3# The minimum number of connections to the database.
4PARSEC_DB_MIN_CONNECTIONS=5
5# The maximum number of connections to the database.
6PARSEC_DB_MAX_CONNECTIONS=7
SMTP configuration
Create the file parsec-smtp.env
to configure the access to the SMTP server (mailhog
in this case).
We need to set the connection informations, the sender information, in which the default language the emails are sent:
1# The SMTP host to use for sending email.
2PARSEC_EMAIL_HOST=parsec-smtp
3# The port to use when connecting to the SMTP server.
4PARSEC_EMAIL_PORT=1025
5# The username to use for the SMTP server.
6PARSEC_EMAIL_HOST_USER=SMTP_USER
7# The password to use for the SMTP server.
8PARSEC_EMAIL_HOST_PASSWORD=SMTP_PASS
9PARSEC_EMAIL_SENDER=parsec@test.xyz
10
11# PARSEC_EMAIL_USE_SSL
12# PARSEC_EMAIL_USE_TLS
13PARSEC_EMAIL_LANGUAGE=en
S3 service configuration
Create the file parsec-s3.env
with the following content to set the URL for the S3-like service:
1# The blockstore URL.
2# Can be S3, Switch or POSTGRESQL URL
3PARSEC_BLOCKSTORE=s3:parsec-s3\:9000:region1:parsec:S3_ROOT_USER:S3_ROOT_PASS
Note
We need to escape the :
with a \
when specifying the port of the service.
Parsec server configuration
Create the file parsec.env
with the following content to configure the parsec-server
service:
1# Host to listen to.
2PARSEC_HOST=0.0.0.0
3
4# The SSL key file.
5PARSEC_SSL_KEYFILE=/run/secrets/parsec-pem-key
6
7# The SSL certificate file.
8PARSEC_SSL_CERTFILE=/run/secrets/parsec-pem-crt
9
10# Enforce HTTPS by redirecting HTTP request.
11PARSEC_FORWARD_PROTO_ENFORCE_HTTPS=X-Forwarded-Proto:https
12
13# The granularity of Error log outputs.
14PARSEC_LOG_LEVEL=WARNING
15
16# The log formatting to use (`CONSOLE` or `JSON`).
17PARSEC_LOG_FORMAT=CONSOLE
18
19# The log file to write to (default to `stderr`).
20# PARSEC_LOG_FILE
21
22# The URL to reach Parsec server.
23PARSEC_SERVER_ADDR=parsec3://127.0.0.1:6777
24
25# Keep SSE connection open by sending keepalive messages to client (pass <=0 to disable).
26PARSEC_SSE_KEEPALIVE=30
27
28# Sentry Data Source Name for telemetry report.
29# PARSEC_SENTRY_DSN
30
31# Sentry environment for telemetry report.
32PARSEC_SENTRY_ENVIRONMENT=production
Note
To see the full list of environment variables that you can use to configure Parsec, you can run:
python -m parsec.cli backend run --help
Look for the sections [env var: VARIABLE]
next to each configuration option. For example:
--administration-token TOKEN Secret token to access the Administration API
[env var: PARSEC_ADMINISTRATION_TOKEN; required]
The docker-compose file
You can use the following docker-compose file (parsec-server.docker.yaml
) to deploy the Parsec server for testing:
1version: "3.8"
2
3services:
4 parsec-postgres:
5 image: postgres:14.10-alpine
6 container_name: parsec-postgres
7 environment:
8 POSTGRES_USER: DB_USER
9 POSTGRES_PASSWORD: DB_PASS
10 POSTGRES_DB: parsec
11 volumes:
12 - parsec-db-data:/var/lib/postgresql/data
13
14 parsec-s3:
15 image: quay.io/minio/minio:RELEASE.2023-09-20T22-49-55Z
16 container_name: parsec-s3
17 command: server --console-address ":9090" --certs-dir /opts/certs /data
18 environment:
19 MINIO_ROOT_USER: S3_ROOT_USER
20 MINIO_ROOT_PASSWORD: S3_ROOT_PASS
21 ports:
22 # Admin console exposed to https://127.0.0.1:9090
23 - 127.0.0.1:9090:9090
24 volumes:
25 - parsec-object-data:/data
26 - ./parsec-s3.key:/opts/certs/private.key:ro
27 - ./parsec-s3.crt:/opts/certs/public.crt:ro
28 - ./custom-ca.crt:/opts/certs/CAs/ca.test.crt:ro
29
30 parsec-smtp:
31 image: mailhog/mailhog:v1.0.1
32 container_name: parsec-smtp
33 ports:
34 - 1025:1025
35 # Web interface exposed to http://127.0.0.1:8025
36 - 127.0.0.1:8025:8025
37
38 parsec-server:
39 depends_on:
40 - parsec-smtp
41 - parsec-s3
42 - parsec-postgres
43 image: ghcr.io/scille/parsec-cloud/parsec-server:2023-10-11-v2.16.0-rc.5.dev-b33d909
44 container_name: parsec-server
45 env_file:
46 - parsec.env
47 - parsec-blockstore.env
48 - parsec-db.env
49 - parsec-email.env
50 - parsec-admin-token.env
51 environment:
52 AWS_CA_BUNDLE: /run/secrets/mini-ca-crt
53 secrets:
54 - mini-ca-crt
55 - parsec-pem-crt
56 - parsec-pem-key
57 ports:
58 - 127.0.0.1:6777:6777
59
60volumes:
61 parsec-db-data: {}
62 parsec-object-data: {}
63
64secrets:
65 parsec-pem-crt:
66 file: ./parsec-server.crt
67 parsec-pem-key:
68 file: ./parsec-server.key
69 mini-ca-crt:
70 file: ./custom-ca.crt
It will setup 4 services:
Service name |
Description |
---|---|
|
The PostgreSQL database |
|
The Object Storage service |
|
A mock SMTP server |
|
The Parsec server |
Starting the services
The docker containers can be started as follow:
docker compose -f parsec-server.docker.yaml up
Initial configuration
On the first start, a one-time configuration is required for the database and s3 services.
Applying the database migration
(optional) Check that the database is accessible with:
set -a
source parsec-db.env
docker exec -t parsec-postgres psql 'postgresql://DB_USER:DB_PASS@0.0.0.0:5432/parsec' -c "\conninfo"
Note
You should have something like display on your console:
You are connected to database "parsec" as user "parsec" on host "0.0.0.0" at port "5432".
To bootstrap the database we just need to apply the migrations with:
docker compose -f parsec-server.docker.yaml run parsec-server migrate
Create the S3 Bucket
Access the console at https://127.0.0.1:9090, you will need to use the credential specified in the docker-compose
file at services.parsec-s3.environment.MINIO_ROOT_{USER,PASSWORD}
.
Go to https://127.0.0.1:9090/buckets/add-bucket to create a new bucket named parsec
with the features object locking
toggled on.
After that you will need to restart the parsec-server
(that likely exited because it wasn’t able to access the S3 bucket):
docker compose -f parsec-server.docker.yaml restart parsec-server
Test the SMTP configuration & server
You can test mailhog
with:
1#!/usr/bin/env bash
2
3set -a
4source parsec-email.env
5
6curl \
7 --url "smtp://127.0.0.1:$PARSEC_EMAIL_PORT" \
8 --user "$PARSEC_EMAIL_HOST_USER@localhost:$PARSEC_EMAIL_HOST_PASSWORD" \
9 --mail-from $PARSEC_EMAIL_SENDER \
10 --mail-rcpt rcpt@test.com \
11 --upload-file <(date --rfc-3339=seconds)
You can then check if the email is present in the web interface at http://127.0.0.1:8025
On Bare Metal
If you don’t want to install Parsec server via Docker, you can install it on bare metal.
Requirements
Python v3.12 with
pip
andvenv
modulesA S3 like object storage endpoint.
A postgres-14 database endpoint.
A TLS certificate & key for the server.
Configure the environment
Configure the env files, follow the env files.
Installation
Set up a virtual env:
python -m venv venv
Configure your shell to use the virtual env:
source venv/bin/activate
Install
parsec-server
The
parsec-server
is available as a python package hosted on https://pypi.org/project/parsec-cloud/.You need to install it with the extra
backend
enabled.pip install 'parsec-cloud[backend]==3.0.0-rc.8'
Prepare the database by applying the migrations:
source venv/bin/activate set -a source parsec-db.env python -m parsec.cli backend migrate
Start the server
Create a wrapper script
run-parsec-server
#!/bin/bash # Load the virtualenv. source venv/bin/activate # Load the env file into the environment table. set -a source parsec-admin-token.env source parsec-db.env source parsec-smtp.env source parsec-s3.env source parsec.env set +a # Start the parsec server. python -m parsec.cli backend run
Execute the wrapper script
run-parsec-server
Note
To run the wrapper with only
run-parsec-server
you need to have set the executable mode on the script file (chmod +x run-parsec-server
). Otherwise, you need to execute it with thebash
shell (bash run-parsec-server
).
Start using Parsec server
Create the first organization
set -a
source parsec-admin-token.env
export SSL_CAFILE=$PWD/custom-ca.crt
parsec.cli core create_organization --addr parsec3://127.0.0.1:6777 <orgname>
Note
Change <orgname>
to the organization’s name that suit you.
Save the link after Bootstrap organization url:
you will need it to create the first user (owner) of the organization.
Add the first user to the organization
First, start parsec
with the custom CA:
export SSL_CAFILE=$PWD/custom-ca.crt
parsec
After that go to Menu
/Join an organization
(or CTRL+O
) and paste the link from before (should already be filled in the text field). Follow the instructions to create the first user of the organization.