9. Dive into real world cdist

9.1. Introduction

This walkthrough shows real world cdist configuration example.

Sample target host is named test.ungleich.ch. Just replace test.ungleich.ch with your target hostname.

Our goal is to configure python application hosting. For writing sample application we will use Bottle WSGI micro web-framework. It will use PostgreSQL database and it will list items from items table. It will be served by uWSGI server. We will also use the Nginx web server as a reverse proxy and we want HTTPS. For HTTPS we will use Let's Encrypt certificate.

For setting up hosting we want to use cdist so we will write a new type for that. This type will:

  • install required packages
  • create OS user, user home directory and application home directory
  • create PostgreSQL database
  • configure uWSGI
  • configure Let's Encrypt certificate
  • configure nginx.

Our type will not create the actual python application. Its intention is only to configure hosing for specified user and project. It is up to the user to create his/her applications.

So let's start.

9.2. Creating type layout

We will create a new custom type. Let's call it __sample_bottle_hosting.

Go to ~/.cdist/type directory (create it if it does not exist) and create new type layout:

cd ~/.cdist/type
mkdir __sample_bottle_hosting
cd __sample_bottle_hosting
touch manifest gencode-remote
mkdir parameter
touch parameter/required

9.3. Creating __sample_bottle_hosting type parameters

Our type will be configurable through the means of parameters. Let's define the following parameters:

projectname
name for the project, needed for uWSGI ini file
user
user name
domain
target host domain, needed for Let's Encrypt certificate.

We define parameters to make our type reusable for different projects, user and domain.

Define required parameters:

printf "projectname\n" >> parameter/required
printf "user\n" >> parameter/required
printf "domain\n" >> parameter/required

For details on type parameters see Defining parameters.

9.4. Creating __sample_bottle_hosting type manifest

Next step is to define manifest (~/.cdist/type/__sample_bottle_hosting/manifest). We also want our type to currently support only Devuan. So we will start by checking target host OS. We will use os global explorer:

os=$(cat "$__global/explorer/os")

case "$os" in
    devuan)
        :
    ;;
    *)
        echo "OS $os currently not supported" >&2
        exit 1
    ;;
esac

If target host OS is not Devuan then we print error message to stderr and exit. For other OS-es support we should check and change package names we should install, because packages differ in different OS-es and in different OS distributions like GNU/Linux distributions. There can also be a different configuration locations (e.g. nginx config directory could be in /usr/local tree). If we detected unsupported OS we should error out. cdist will stop configuration process and output error message.

9.4.1. Creating user and user directories

Then we create user and his/her home directory and application home directory. We will use existing cdist types __user and __directory:

user="$(cat "$__object/parameter/user")"
home="/home/$user"
apphome="$home/app"

# create user
__user "$user" --home "$home" --shell /bin/bash
# create user home dir
require="__user/$user" __directory "$home" \
    --owner "$user" --group "$user" --mode 0755
# create app home dir
require="__user/$user __directory/$home" __directory "$apphome" \
    --state present --owner "$user" --group "$user" --mode 0755

First we define user, home and apphome variables. User is defined by type's user parameter. Here we use require which is cdist's way to define dependencies. User home directory should be created after user is created. And application home directory is created after both user and user home directory are created. For details on require see Dependencies.

9.4.2. Installing packages

Install required packages using existing __package type. Before installing package we want to update apt package index using __apt_update_index:

# define packages that need to be installed
packages_to_install="nginx uwsgi-plugin-python3 python3-dev python3-pip postgresql postgresql-contrib libpq-dev python3-venv uwsgi python3-psycopg2"

# update package index
__apt_update_index
# install packages
for package in $packages_to_install
    do require="__apt_update_index" __package $package --state=present
done

Here we use shell for loop. It executes require="__apt_update_index" __package for each member in a list we define in packages_to_install variable. This is much nicer then having as many require="__apt_update_index" __package lines as there are packages we want to install.

For python packages we use __package_pip:

# install pip3 packages
for package in bottle bottle-pgsql; do
    __package_pip --pip pip3 $package
done

9.4.3. Creating PostgreSQL database

Create PostgreSQL database using __postgres_database and __postgres_role for creating database user:

#PostgreSQL db & user
postgres_server=postgresql

# create PostgreSQL db user
require="__package/postgresql" __postgres_role $user --login --createdb
# create PostgreSQL db
require="__postgres_role/$user __package/postgresql" __postgres_database $user \
    --owner $user

9.4.4. Configuring uWSGI

Configure uWSGI using __file type:

# configure uWSGI
projectname="$(cat "$__object/parameter/projectname")"
require="__package/uwsgi" __file /etc/uwsgi/apps-enabled/$user.ini \
            --owner root --group root --mode 0644 \
            --state present \
            --source - << EOF
[uwsgi]
socket = $apphome/uwsgi.sock
chdir = $apphome
wsgi-file = $projectname/wsgi.py
touch-reload = $projectname/wsgi.py
processes = 4
threads = 2
chmod-socket = 666
daemonize=true
vacuum = true
uid = $user
gid = $user
EOF

We require package uWSGI present in order to create /etc/uwsgi/apps-enabled/$user.ini file. Installation of uWSGI also creates configuration layout: /etc/uwsgi/apps-enabled. If this directory does not exist then __file type would error. We also use stdin as file content source. For details see Input from stdin. For feading stdin we use here-document (<< operator). It allows redirection of subsequent lines read by the shell to the input of a command until a line containing only the delimiter and a newline, with no blank characters in between (EOF in our case).

9.4.5. Configuring nginx for Let's Encrypt and HTTPS redirection

Next configure nginx for Let's Encrypt and for HTTP -> HTTPS redirection. For this purpose we will create new type __sample_nginx_http_letsencrypt_and_ssl_redirect and use it here:

domain="$(cat "$__object/parameter/domain")"
webroot="/var/www/html"
__sample_nginx_http_letsencrypt_and_ssl_redirect "$domain" --webroot "$webroot"

9.4.6. Configuring certificate creation

After HTTP nginx configuration we will create Let's Encrypt certificate using __letsencrypt_cert type. For Let's Encrypt cert configuration ensure that there is a DNS entry for your domain. We assure that cert creation is applied after nginx HTTP is configured for Let's Encrypt to work:

# create SSL cert
require="__package/nginx __sample_nginx_http_letsencrypt_and_ssl_redirect/$domain" \
    __letsencrypt_cert --admin-email admin@test.ungleich.ch \
        --webroot "$webroot" \
        --automatic-renewal \
        --renew-hook "service nginx reload" \
        --domain "$domain" \
        "$domain"

9.4.7. Configuring nginx HTTPS server with uWSGI upstream

Then we can configure nginx HTTPS server that will use created Let's Encrypt certificate:

# configure nginx
require="__package/nginx __letsencrypt_cert/$domain" \
    __file "/etc/nginx/sites-enabled/https-$domain" \
    --source - --mode 0644 << EOF
upstream _bottle {
    server unix:$apphome/uwsgi.sock;
}

server {
    listen 443;
    listen [::]:443;

    server_name $domain;

    access_log  /var/log/nginx/access.log;

    ssl on;
    ssl_certificate      /etc/letsencrypt/live/$domain/fullchain.pem;
    ssl_certificate_key  /etc/letsencrypt/live/$domain/privkey.pem;

    client_max_body_size 256m;

    location / {
        try_files \$uri @uwsgi;
    }

    location @uwsgi {
        include uwsgi_params;
        uwsgi_pass _bottle;
    }
}
EOF

Now our manifest is finished.

9.4.8. Complete __sample_bottle_hosting type manifest listing

Here is complete __sample_bottle_hosting type manifest listing, located in ~/.cdist/type/__sample_bottle_hosting/manifest:

#!/bin/sh

os=$(cat "$__global/explorer/os")

case "$os" in
    devuan)
        :
    ;;
    *)
        echo "OS $os currently not supported" >&2
        exit 1
    ;;
esac

projectname="$(cat "$__object/parameter/projectname")"
user="$(cat "$__object/parameter/user")"
home="/home/$user"
apphome="$home/app"
domain="$(cat "$__object/parameter/domain")"

# create user
__user "$user" --home "$home" --shell /bin/bash
# create user home dir
require="__user/$user" __directory "$home" \
    --owner "$user" --group "$user" --mode 0755
# create app home dir
require="__user/$user __directory/$home" __directory "$apphome" \
    --state present --owner "$user" --group "$user" --mode 0755

# define packages that need to be installed
packages_to_install="nginx uwsgi-plugin-python3 python3-dev python3-pip postgresql postgresql-contrib libpq-dev python3-venv uwsgi python3-psycopg2"

# update package index
__apt_update_index
# install packages
for package in $packages_to_install
    do require="__apt_update_index" __package $package --state=present
done
# install pip3 packages
for package in bottle bottle-pgsql; do
    __package_pip --pip pip3 $package
done

#PostgreSQL db & user
postgres_server=postgresql

# create PostgreSQL db user
require="__package/postgresql" __postgres_role $user --login --createdb
# create PostgreSQL db
require="__postgres_role/$user __package/postgresql" __postgres_database $user \
    --owner $user
# configure uWSGI
require="__package/uwsgi" __file /etc/uwsgi/apps-enabled/$user.ini \
            --owner root --group root --mode 0644 \
            --state present \
            --source - << EOF
[uwsgi]
socket = $apphome/uwsgi.sock
chdir = $apphome
wsgi-file = $projectname/wsgi.py
touch-reload = $projectname/wsgi.py
processes = 4
threads = 2
chmod-socket = 666
daemonize=true
vacuum = true
uid = $user
gid = $user
EOF

# setup nginx HTTP for Let's Encrypt and SSL redirect
domain="$(cat "$__object/parameter/domain")"
webroot="/var/www/html"
__sample_nginx_http_letsencrypt_and_ssl_redirect "$domain" --webroot "$webroot"

# create SSL cert
require="__package/nginx __sample_nginx_http_letsencrypt_and_ssl_redirect/$domain" \
    __letsencrypt_cert --admin-email admin@test.ungleich.ch \
        --webroot "$webroot" \
        --automatic-renewal \
        --renew-hook "service nginx reload" \
        --domain "$domain" \
        "$domain"

# configure nginx
require="__package/nginx __letsencrypt_cert/$domain" \
    __file "/etc/nginx/sites-enabled/https-$domain" \
    --source - --mode 0644 << EOF
upstream _bottle {
    server unix:$apphome/uwsgi.sock;
}

server {
    listen 443;
    listen [::]:443;

    server_name $domain;

    access_log  /var/log/nginx/access.log;

    ssl on;
    ssl_certificate      /etc/letsencrypt/live/$domain/fullchain.pem;
    ssl_certificate_key  /etc/letsencrypt/live/$domain/privkey.pem;

    client_max_body_size 256m;

    location / {
        try_files \$uri @uwsgi;
    }

    location @uwsgi {
        include uwsgi_params;
        uwsgi_pass _bottle;
    }
}
EOF

9.5. Creating __sample_bottle_hosting type gencode-remote

Now define gencode-remote script: ~/.cdist/type/__sample_bottle_hosting/gencode-remote. After manifest is applied it should restart uWSGI and nginx services so that our configuration is active. Our gencode-remote looks like the following:

echo "service uwsgi restart"
echo "service nginx restart"

Our __sample_bottle_hosting type is now finished.

9.6. Creating __sample_nginx_http_letsencrypt_and_ssl_redirect type

Let's now create __sample_nginx_http_letsencrypt_and_ssl_redirect type:

cd ~/.cdist/type
mkdir __sample_nginx_http_letsencrypt_and_ssl_redirect
cd __sample_nginx_http_letsencrypt_and_ssl_redirect
mkdir parameter
echo webroot > parameter/required
touch manifest
touch gencode-remote

Edit manifest:

domain="$__object_id"
webroot="$(cat "$__object/parameter/webroot")"
# make sure we have nginx package
__package nginx
# setup Let's Encrypt HTTP acme challenge, redirect HTTP to HTTPS
require="__package/nginx" __file "/etc/nginx/sites-enabled/http-$domain" \
    --source - --mode 0644 << EOF
server {
    listen *:80;
    listen [::]:80;

    server_name $domain;

    # Let's Encrypt
    location /.well-known/acme-challenge/ {
        root $webroot;
    }

    # Everything else -> SSL
    location / {
        return 301 https://\$host\$request_uri;
    }
}

EOF

Edit gencode-remote:

echo "service nginx reload"

9.7. Creating init manifest

Next create init manifest:

cd ~/.cdist/manifest
printf "__sample_bottle_hosting --projectname sample --user app --domain \$__target_host sample\n" > sample

Using this init manifest our target host will be configured using our __sample_bottle_hosting type with projectname sample, user app and domain equal to __target_host. Here the last positional argument sample is type's object id. For details on __target_host and __object_id see Environment variables (for reading) reference.

9.8. Configuring host

Finally configure test.ungleich.ch:

cdist config -v -i ~/.cdist/manifest/sample test.ungleich.ch

After cdist configuration is successfully finished our host is ready.

9.9. Creating python bottle application

We now need to create Bottle application. As you remember from the beginning of this walkthrough our type does not create the actual python application, its intention is only to configure hosing for specified user and project. It is up to the user to create his/her applications.

Become app user:

su -l app

9.9.1. Preparing database

We need to prepare database for our application. Create table and insert some items:

psql -c "create table items (item varchar(255));"

psql -c "insert into items(item) values('spam');"
psql -c "insert into items(item) values('eggs');"
psql -c "insert into items(item) values('sausage');"

9.9.2. Creating application

Next create sample app:

cd /home/app/app
mkdir sample
cd sample

Create app.py with the following content:

#!/usr/bin/env python3

import bottle
import bottle_pgsql

app = application = bottle.Bottle()
plugin = bottle_pgsql.Plugin('dbname=app user=app password=')
app.install(plugin)

@app.route('/')
def show_index(db):
    db.execute('select * from items')
    items = db.fetchall() or []
    rv = '<html><body><h3>Items:</h3><ul>'
    for item in items:
        rv += '<li>' + str(item['item']) + '</li>'
    rv += '</ul></body></html>'
    return rv

if __name__ == '__main__':
    bottle.run(app=app, host='0.0.0.0', port=8080)

Create wsgi.py with the following content:

import os

os.chdir(os.path.dirname(__file__))

import app
application = app.app

We have configured uWSGI with touch-reload = $projectname/wsgi.py so after we have changed our wsgi.py file uWSGI reloads the application.

Our application selects and lists items from items table.

9.9.3. Openning application

Finally try the application:

http://test.ungleich.ch/

It should redirect to HTTPS and return:

Items:

  • spam
  • eggs
  • sausage

9.10. What's next?

Continue reading next sections ;)