Features

The following features are available in the environment constructed by the included provisioning scripts.

Several features may only apply in a production or development environment. This is differentiated based on the DEBUG setting in the env.sh file.

Well-defined project structure

The provisioning process creates a well-defined directory structure for all project-related files.

The root of this structure is /opt/app/.

The most important subdirectory is /opt/app/src/. This is the project root directory, and the target of the Vagrant synced folder. Subsequently, /opt/app/src/provision/ contains all the provisioning scripts.

Some of the other useful directories in this structure are:

  • /opt/app/conf/: For storage of configuration files such as nginx.conf and gunicorn’s conf.py. Such files are copied here instead of being referenced directly from within provision/conf/ so they may be modified without affecting the committed source files.
  • /opt/app/logs/: For storage of log files output by supervisor, etc.
  • /opt/app/media/: Target for Django’s MEDIA_ROOT.
  • /opt/app/static/: Target for Django’s STATIC_ROOT (in production environments).

The final directory is /opt/app/ln/. This directory is primarily used to simplify the process of configuring the server. It acts as a container for shortcut symlinks to various project-specific files and directories (i.e. those that contain the project name). It is designed to allow using known paths in config files, without forcing customisation in projects that would not otherwise need it. Using the shortcuts in the ln directory, default config files that work out-of-the-box can be provided (such as provision/conf/supervisor/production_programs/gunicorn.conf). Otherwise, such files would require the modification of a series of paths to include the project name.

Locked down user access

SSH access is locked down to the custom webmaster user created during provisioning. SSH is available via public key authentication only - no passwords. In a development environment, only the webmaster and vagrant users are allowed SSH access. In a production environment, only webmaster is granted access. No other users, including root, can SSH into the machine.

The public key to use for the webmaster user must be provided via the PUBLIC_KEY variable in the env.sh file. This will be installed into /home/webmaster/.ssh/authorized_keys.

The webmaster user is given sudo privileges. In development environments, for convenience, it does not require a password. In production environments, it does. A password is not configured as part of the provisioning process, one needs to be manually created afterwards. When logged in as the webmaster user, simply run the passwd command to set a password.

Most provisioned services, such as nginx and gunicorn, are designed to run under the default www-data user.

Warning

Using the provisioning scripts in a production environment with DEBUG set to 1 will leave the webmaster user with open sudo access, unprotected by a password prompt. This is a Bad Idea.

Time zone

The time zone can be set using the TIME_ZONE setting in the env.sh file.

Firewall

In production environments, and if a firewall rules configuration file is provided, a firewall is provisioned using UncomplicatedFirewall.

Git

Git is installed.

Note

A .gitconfig file can be placed in provision/conf/ to enable configuration of the git environment for the webmaster user.

Ag (silver searcher)

The “silver searcher” commandline utility, ag, is installed in the guest machine. ag provides fast code search that is better than ack.

Note

An .agignore file can be placed in provision/conf/ to add some additional automatic “ignores” for the command.

Image libraries

Various system-level image libraries used by Pillow are installed in the guest machine.

To install Pillow itself, it should be included in requirements.txt along with other Python dependencies (see Python dependency installation below). But considering many of its features require external libraries, and the high likelihood that a Django project will require Pillow, those libraries are installed in readiness.

The exact packages installed are taken from the Pillow “depends” script for Ubuntu, though not all are used.

Installed packages:

  • libtiff5-dev
  • libjpeg8-dev
  • zlib1g-dev
  • libfreetype6-dev
  • liblcms2-dev

PostgreSQL

PostgreSQL is installed.

In addition, a database user is created with a username equal to the project name and a password equal to DB_PASS. A database is also created, also with a name equal to the project name, with the aforementioned user as the owner.

The Postgres installation is configured to listen on the default port (5432).

Nginx

In production environments, nginx is installed.

The nginx.conf file used can be modified. Also, the site config can - and must - be modified. See Configuring nginx for details.

Nginx is controlled and monitored by Supervisor. A default supervisor program is provided, but can be modified. See Supervisor programs for details.

Gunicorn

In production environments, gunicorn is installed.

The conf.py file used can be modified. See Configuring gunicorn for details.

Gunicorn is controlled and monitored by Supervisor. A default supervisor program is provided, but can be modified. See Supervisor programs for details.

Supervisor

Supervisor is installed.

The supervisord.conf file used can be modified. See Configuring supervisor for details.

Default programs for Nginx and Gunicorn are provided, but any number of additional programs can be added. See Supervisor programs for details.

Virtualenv

A virtualenv is created using pyenv and its pyenv-virtualenv plugin.

The version of Python used to build the virtualenv can be specified in versions.sh using the BASE_PYTHON_VERSION setting. If not specified, the system version will be used.

The virtualenv is automatically activated when the webmaster user logs in via SSH.

Python dependency installation

If a requirements.txt file is found in the project root directory (/opt/app/src/), the included requirements will be installed into the virtualenv (via pip -r requirements.txt).

In development environments, a dev_requirements.txt file can also be specified to install additional development-specific dependencies, e.g. debugging tools, documentation building packages, etc. This keeps these kinds of packages out of the project’s primary requirements.txt.

Node.js/npm

If a package.json file is found in the project root directory (/opt/app/src/), node.js and npm are installed. The version of node.js installed is dictated by the NODE_VERSION setting in versions.sh.

A node_modules directory is created at /opt/app/node_modules/ and a symlink to this directory is created in the project root directory (/opt/app/src/node_modules). Keeping the node_modules directory out of the synced folder helps avoid potential issues with Windows host machines - path names generated by installing certain npm packages can exceed the maximum Windows allows.

Note

In order to create the node_modules symlink when running a Windows host and using VirtualBox shared folders, vagrant up must be run with Administrator privileges to allow the creation of symlinks in the synced folder. See Windows Hosts for details.

Note

If a package.json file is added to the project at a later date, provisioning can be safely re-run to install node/npm (using the vagrant provision command).

Node.js dependency installation

npm install will be run in the project root directory.

In production environments, npm install --production will be used, limiting the installed dependencies to those listed in the dependencies section of package.json. Otherwise, dependencies listed in dependencies and devDependencies will be installed. See the documentation on npm install.

Multiple Python versions and tox support

The base Python version (used to create the virtualenv under which all relevant Python processes for the project will be run) and additional versions of Python can be specified in versions.sh, via the BASE_PYTHON_VERSION and PYTHON_VERSIONS, respectively.

All specified Python versions are installed with pyenv. The pyenv global command is used to provide system-wide access to all installed versions, with the following priority:

  • PYTHON_VERSIONS, in the order they are defined
  • The specified BASE_PYTHON_VERSION, if there is one and if it doesn’t already appear in PYTHON_VERSIONS
  • The system Python

For example:

# The following settings...
BASE_PYTHON_VERSION='3.6.4'
PYTHON_VERSIONS=('2.7.14' '3.5.4')

# ... yield the command:
pyenv global 2.7.14 3.5.4 3.6.4 system

If you want the specified base version to appear somewhere specific among the list of versions, include it explicitly in PYTHON_VERSIONS:

# The following settings...
BASE_PYTHON_VERSION='3.6.4'
PYTHON_VERSIONS=('3.6.4' '2.7.14' '3.5.4')

# ... yield the command:
pyenv global 3.6.4 2.7.14 3.5.4 system

This support is most useful when using tox to test your code under multiple versions of Python.

env.py

Several of the env.sh settings are designed to eliminate hardcoding environment-specific and/or sensitive settings in Django’s settings.py file. Things like the database password, the SECRET_KEY and the DEBUG flag should be configured per environment and not be committed to source control.

12factor recommends these types of settings be loaded into environment variables, with these variables subsequently used in settings.py. But environment variables can be a kind of invisible magic, and it is not easy to simply view the entire set of environment variables that exist for a given project’s use. To make this possible, an env.py file is written by the provisioning scripts.

This ordinary Python file simply defines a dictionary called environ, containing settings defined as key/value pairs. It can then be imported by settings.py and used in a manner very similar to using environment variables.

# Using env.py
from . import env
env.environ.get('DEBUG')

# Using environment variables
import os
os.environ.get('DEBUG')

The environ dictionary is used rather than simply providing a set of module-level constants primarily to allow simple definition of default values:

env.environ.get('DEBUG', False)

The default environ dictionary will contain the following key/values:

  • DEBUG: Will be True if DEBUG is set to 1, False if it is set to 0.
  • DB_USER: Set to the value of the project name.
  • DB_PASSWORD: Set to the value of DB_PASS. Automatically generated by default.
  • TIME_ZONE: Set to the value of TIME_ZONE.
  • SECRET_KEY: Set to the value of SECRET_KEY. Automatically generated by default.

If a specific project has additional sensitive or environment-specific settings that are better not committed to source control, it is possible to modify the way env.py is written such that it can contain those settings as well, or at least placeholders for them. See Customising env.py for more details.

Note

The env.py file should not be committed to source control. Doing so would defeat the purpose!

Project-specific provisioning

In addition to the above generic provisioning, any special steps required by individual projects can be included using the provision/project.sh file. If found, this shell script file will be executed during the provisioning process. This file can be used to install additional system libraries, create/edit configuration files, etc.

For more information, see the Project-specific Provisioning documentation.

Shortcut commands

The following shell commands are made available on the system path for convenience:

  • pull+: For git users. A helper script for pulling in the latest changes from origin/master and performing several post-pull updates. It must be run from the project root directory (/opt/app/src/). Specifically, and in order of operation, the script:

    • Runs git pull origin master as the www-data user
    • Runs python manage.py collectstatic (production environments only), also as the www-data user
    • Checks for differences in requirements.txt#
    • Asks to install from requirements, if any differences were found
    • Runs pip install -r requirements.txt if installing was requested
    • Checks for unapplied migrations (using Django’s showmigrations management command)
    • Asks to apply the migrations, if any were found
    • Runs python manage.py migrate if applying was requested
    • Runs python manage.py remove_stale_contenttypes if using Django 1.11+
    • Restarts gunicorn (production environments only)

#: When first run, pull+ detects differences between the requirements.txt file as it existed before the pull vs after the pull. Even if no differences are found, the installed packages may still be out of date if an updated requirements.txt was pulled in prior to running the command. After the first run, it stores a temporary copy of requirements.txt any time updates are chosen to be installed. It can then compare the newly-pulled file to this temporary copy, enabling it to detect changes from any pulls that took place in the meantime as well. However, if the requirements are updated manually (outside of using this command), it will detect differences in the files even if the installed packages are up to date.