Dive into real world cdist ========================== Introduction ------------ This walkthrough shows real world cdist configuration example. Sample target host is named **test.ungleich.ch**. Just replace **test.ungleich.ch** with your target hostname. Our goal is to configure python application hosting. For writing sample application we will use `Bottle `_ WSGI micro web-framework. It will use PostgreSQL database and it will list items from **items** table. It will be served by uWSGI server. We will also use the Nginx web server as a reverse proxy and we want HTTPS. For HTTPS we will use Let's Encrypt certificate. For setting up hosting we want to use cdist so we will write a new type for that. This type will: - install required packages - create OS user, user home directory and application home directory - create PostgreSQL database - configure uWSGI - configure Let's Encrypt certificate - configure nginx. Our type will not create the actual python application. Its intention is only to configure hosing for specified user and project. It is up to the user to create his/her applications. So let's start. Creating type layout -------------------- We will create a new custom type. Let's call it **__sample_bottle_hosting**. Go to **~/.cdist/type** directory (create it if it does not exist) and create new type layout:: cd ~/.cdist/type mkdir __sample_bottle_hosting cd __sample_bottle_hosting touch manifest gencode-remote mkdir parameter touch parameter/required Creating __sample_bottle_hosting type parameters ------------------------------------------------ Our type will be configurable through the means of parameters. Let's define the following parameters: projectname name for the project, needed for uWSGI ini file user user name domain target host domain, needed for Let's Encrypt certificate. We define parameters to make our type reusable for different projects, user and domain. Define required parameters:: printf "projectname\n" >> parameter/required printf "user\n" >> parameter/required printf "domain\n" >> parameter/required For details on type parameters see `Defining parameters `_. Creating __sample_bottle_hosting type manifest ---------------------------------------------- Next step is to define manifest (~/.cdist/type/__sample_bottle_hosting/manifest). We also want our type to currently support only Devuan. So we will start by checking target host OS. We will use `os `_ global explorer:: os=$(cat "$__global/explorer/os") case "$os" in devuan) : ;; *) echo "OS $os currently not supported" >&2 exit 1 ;; esac If target host OS is not Devuan then we print error message to stderr and exit. For other OS-es support we should check and change package names we should install, because packages differ in different OS-es and in different OS distributions like GNU/Linux distributions. There can also be a different configuration locations (e.g. nginx config directory could be in /usr/local tree). If we detected unsupported OS we should error out. cdist will stop configuration process and output error message. Creating user and user directories ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Then we create user and his/her home directory and application home directory. We will use existing cdist types `__user `_ and `__directory `_:: user="$(cat "$__object/parameter/user")" home="/home/$user" apphome="$home/app" # create user __user "$user" --home "$home" --shell /bin/bash # create user home dir require="__user/$user" __directory "$home" \ --owner "$user" --group "$user" --mode 0755 # create app home dir require="__user/$user __directory/$home" __directory "$apphome" \ --state present --owner "$user" --group "$user" --mode 0755 First we define *user*, *home* and *apphome* variables. User is defined by type's **user** parameter. Here we use **require** which is cdist's way to define dependencies. User home directory should be created **after** user is created. And application home directory is created **after** both user and user home directory are created. For details on **require** see `Dependencies `_. Installing packages ~~~~~~~~~~~~~~~~~~~ Install required packages using existing `__package `_ type. Before installing package we want to update apt package index using `__apt_update_index `_:: # define packages that need to be installed packages_to_install="nginx uwsgi-plugin-python3 python3-dev python3-pip postgresql postgresql-contrib libpq-dev python3-venv uwsgi python3-psycopg2" # update package index __apt_update_index # install packages for package in $packages_to_install do require="__apt_update_index" __package $package --state=present done Here we use shell for loop. It executes **require="__apt_update_index" __package** for each member in a list we define in **packages_to_install** variable. This is much nicer then having as many **require="__apt_update_index" __package** lines as there are packages we want to install. For python packages we use `__package_pip `_:: # install pip3 packages for package in bottle bottle-pgsql; do __package_pip --pip pip3 $package done Creating PostgreSQL database ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Create PostgreSQL database using `__postgres_database `_ and `__postgres_role `_ for creating database user:: #PostgreSQL db & user postgres_server=postgresql # create PostgreSQL db user require="__package/postgresql" __postgres_role $user --login --createdb # create PostgreSQL db require="__postgres_role/$user __package/postgresql" __postgres_database $user \ --owner $user Configuring uWSGI ~~~~~~~~~~~~~~~~~ Configure uWSGI using `__file `_ type:: # configure uWSGI projectname="$(cat "$__object/parameter/projectname")" require="__package/uwsgi" __file /etc/uwsgi/apps-enabled/$user.ini \ --owner root --group root --mode 0644 \ --state present \ --source - << EOF [uwsgi] socket = $apphome/uwsgi.sock chdir = $apphome wsgi-file = $projectname/wsgi.py touch-reload = $projectname/wsgi.py processes = 4 threads = 2 chmod-socket = 666 daemonize=true vacuum = true uid = $user gid = $user EOF We require package uWSGI present in order to create **/etc/uwsgi/apps-enabled/$user.ini** file. Installation of uWSGI also creates configuration layout: **/etc/uwsgi/apps-enabled**. If this directory does not exist then **__file** type would error. We also use stdin as file content source. For details see `Input from stdin `_. For feading stdin we use here-document (**<<** operator). It allows redirection of subsequent lines read by the shell to the input of a command until a line containing only the delimiter and a newline, with no blank characters in between (EOF in our case). Configuring nginx for Let's Encrypt and HTTPS redirection ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Next configure nginx for Let's Encrypt and for HTTP -> HTTPS redirection. For this purpose we will create new type **__sample_nginx_http_letsencrypt_and_ssl_redirect** and use it here:: domain="$(cat "$__object/parameter/domain")" webroot="/var/www/html" __sample_nginx_http_letsencrypt_and_ssl_redirect "$domain" --webroot "$webroot" Configuring certificate creation ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ After HTTP nginx configuration we will create Let's Encrypt certificate using `__letsencrypt_cert `_ type. For Let's Encrypt cert configuration ensure that there is a DNS entry for your domain. We assure that cert creation is applied after nginx HTTP is configured for Let's Encrypt to work:: # create SSL cert require="__package/nginx __sample_nginx_http_letsencrypt_and_ssl_redirect/$domain" \ __letsencrypt_cert --admin-email admin@test.ungleich.ch \ --webroot "$webroot" \ --automatic-renewal \ --renew-hook "service nginx reload" \ --domain "$domain" \ "$domain" Configuring nginx HTTPS server with uWSGI upstream ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Then we can configure nginx HTTPS server that will use created Let's Encrypt certificate:: # configure nginx require="__package/nginx __letsencrypt_cert/$domain" \ __file "/etc/nginx/sites-enabled/https-$domain" \ --source - --mode 0644 << EOF upstream _bottle { server unix:$apphome/uwsgi.sock; } server { listen 443; listen [::]:443; server_name $domain; access_log /var/log/nginx/access.log; ssl on; ssl_certificate /etc/letsencrypt/live/$domain/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/$domain/privkey.pem; client_max_body_size 256m; location / { try_files \$uri @uwsgi; } location @uwsgi { include uwsgi_params; uwsgi_pass _bottle; } } EOF Now our manifest is finished. Complete __sample_bottle_hosting type manifest listing ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Here is complete __sample_bottle_hosting type manifest listing, located in ~/.cdist/type/__sample_bottle_hosting/manifest:: #!/bin/sh os=$(cat "$__global/explorer/os") case "$os" in devuan) : ;; *) echo "OS $os currently not supported" >&2 exit 1 ;; esac projectname="$(cat "$__object/parameter/projectname")" user="$(cat "$__object/parameter/user")" home="/home/$user" apphome="$home/app" domain="$(cat "$__object/parameter/domain")" # create user __user "$user" --home "$home" --shell /bin/bash # create user home dir require="__user/$user" __directory "$home" \ --owner "$user" --group "$user" --mode 0755 # create app home dir require="__user/$user __directory/$home" __directory "$apphome" \ --state present --owner "$user" --group "$user" --mode 0755 # define packages that need to be installed packages_to_install="nginx uwsgi-plugin-python3 python3-dev python3-pip postgresql postgresql-contrib libpq-dev python3-venv uwsgi python3-psycopg2" # update package index __apt_update_index # install packages for package in $packages_to_install do require="__apt_update_index" __package $package --state=present done # install pip3 packages for package in bottle bottle-pgsql; do __package_pip --pip pip3 $package done #PostgreSQL db & user postgres_server=postgresql # create PostgreSQL db user require="__package/postgresql" __postgres_role $user --login --createdb # create PostgreSQL db require="__postgres_role/$user __package/postgresql" __postgres_database $user \ --owner $user # configure uWSGI require="__package/uwsgi" __file /etc/uwsgi/apps-enabled/$user.ini \ --owner root --group root --mode 0644 \ --state present \ --source - << EOF [uwsgi] socket = $apphome/uwsgi.sock chdir = $apphome wsgi-file = $projectname/wsgi.py touch-reload = $projectname/wsgi.py processes = 4 threads = 2 chmod-socket = 666 daemonize=true vacuum = true uid = $user gid = $user EOF # setup nginx HTTP for Let's Encrypt and SSL redirect domain="$(cat "$__object/parameter/domain")" webroot="/var/www/html" __sample_nginx_http_letsencrypt_and_ssl_redirect "$domain" --webroot "$webroot" # create SSL cert require="__package/nginx __sample_nginx_http_letsencrypt_and_ssl_redirect/$domain" \ __letsencrypt_cert --admin-email admin@test.ungleich.ch \ --webroot "$webroot" \ --automatic-renewal \ --renew-hook "service nginx reload" \ --domain "$domain" \ "$domain" # configure nginx require="__package/nginx __letsencrypt_cert/$domain" \ __file "/etc/nginx/sites-enabled/https-$domain" \ --source - --mode 0644 << EOF upstream _bottle { server unix:$apphome/uwsgi.sock; } server { listen 443; listen [::]:443; server_name $domain; access_log /var/log/nginx/access.log; ssl on; ssl_certificate /etc/letsencrypt/live/$domain/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/$domain/privkey.pem; client_max_body_size 256m; location / { try_files \$uri @uwsgi; } location @uwsgi { include uwsgi_params; uwsgi_pass _bottle; } } EOF Creating __sample_bottle_hosting type gencode-remote ---------------------------------------------------- Now define **gencode-remote** script: ~/.cdist/type/__sample_bottle_hosting/gencode-remote. After manifest is applied it should restart uWSGI and nginx services so that our configuration is active. Our gencode-remote looks like the following:: echo "service uwsgi restart" echo "service nginx restart" Our **__sample_bottle_hosting** type is now finished. Creating __sample_nginx_http_letsencrypt_and_ssl_redirect type -------------------------------------------------------------- Let's now create **__sample_nginx_http_letsencrypt_and_ssl_redirect** type:: cd ~/.cdist/type mkdir __sample_nginx_http_letsencrypt_and_ssl_redirect cd __sample_nginx_http_letsencrypt_and_ssl_redirect mkdir parameter echo webroot > parameter/required touch manifest touch gencode-remote Edit manifest:: domain="$__object_id" webroot="$(cat "$__object/parameter/webroot")" # make sure we have nginx package __package nginx # setup Let's Encrypt HTTP acme challenge, redirect HTTP to HTTPS require="__package/nginx" __file "/etc/nginx/sites-enabled/http-$domain" \ --source - --mode 0644 << EOF server { listen *:80; listen [::]:80; server_name $domain; # Let's Encrypt location /.well-known/acme-challenge/ { root $webroot; } # Everything else -> SSL location / { return 301 https://\$host\$request_uri; } } EOF Edit gencode-remote:: echo "service nginx reload" Creating init manifest ---------------------- Next create init manifest:: cd ~/.cdist/manifest printf "__sample_bottle_hosting --projectname sample --user app --domain \$__target_host sample\n" > sample Using this init manifest our target host will be configured using our **__sample_bottle_hosting** type with projectname *sample*, user *app* and domain equal to **__target_host**. Here the last positional argument *sample* is type's object id. For details on **__target_host** and **__object_id** see `Environment variables (for reading) `_ reference. Configuring host ---------------- Finally configure test.ungleich.ch:: cdist config -v -i ~/.cdist/manifest/sample test.ungleich.ch After cdist configuration is successfully finished our host is ready. Creating python bottle application ---------------------------------- We now need to create Bottle application. As you remember from the beginning of this walkthrough our type does not create the actual python application, its intention is only to configure hosing for specified user and project. It is up to the user to create his/her applications. Become app user:: su -l app Preparing database ~~~~~~~~~~~~~~~~~~ We need to prepare database for our application. Create table and insert some items:: psql -c "create table items (item varchar(255));" psql -c "insert into items(item) values('spam');" psql -c "insert into items(item) values('eggs');" psql -c "insert into items(item) values('sausage');" Creating application ~~~~~~~~~~~~~~~~~~~~ Next create sample app:: cd /home/app/app mkdir sample cd sample Create app.py with the following content:: #!/usr/bin/env python3 import bottle import bottle_pgsql app = application = bottle.Bottle() plugin = bottle_pgsql.Plugin('dbname=app user=app password=') app.install(plugin) @app.route('/') def show_index(db): db.execute('select * from items') items = db.fetchall() or [] rv = '

Items:

    ' for item in items: rv += '
  • ' + str(item['item']) + '
  • ' rv += '
' return rv if __name__ == '__main__': bottle.run(app=app, host='0.0.0.0', port=8080) Create wsgi.py with the following content:: import os os.chdir(os.path.dirname(__file__)) import app application = app.app We have configured uWSGI with **touch-reload = $projectname/wsgi.py** so after we have changed our **wsgi.py** file uWSGI reloads the application. Our application selects and lists items from **items** table. Openning application ~~~~~~~~~~~~~~~~~~~~ Finally try the application:: http://test.ungleich.ch/ It should redirect to HTTPS and return: .. container:: highlight .. raw:: html

Items:

  • spam
  • eggs
  • sausage
What's next? ------------ Continue reading next sections ;)