Django Full Stack Set Up and Deployment

28th February, 2017

After installing all the basic set up for Django web deveopment ( http://www.shubhamdipt.com/blog/mac-set-up-for-django-web-development )

Step by Step process: (for Django, PostgreSQL)
[ replace <project> by your project's name]

Create virtual environment
mkvirtualenv <project>
Activate the environment
workon <project>
Install Django
pip install django
Create a Django project
django-admin startproject <project>
Enter the project diretory
cd <project>
Initiate Git (version control)
git init
Add remote repository to git
git add remote <repo name>
Create .gitgnore file
vim .gitignore
Create directories: static, templates, locale, requirements
Inside requirements folder, create base.txt, local.txt, production.txt files.
Create a database in postgresql <project_name> using pgAdmin.
Install psycopg2 for postgresql.
pip install psycopg2
Install whitenoise to serve static files in production.
pip install whitenoise
Install raven for Sentry logging.
pip instal raven
Install cloudinary
pip install cloudinary
Save the requirements in requirements.txt file.
pip freeze > requirements.txt

Copy the follwing to the settings.py

"""
Django settings for <project> project.

Generated by 'django-admin startproject' using Django 1.10.5.

For more information on this file, see
https://docs.djangoproject.com/en/1.10/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/1.10/ref/settings/
"""

import os
import json
import cloudinary
import raven


PROJECT_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
DEP_NAME = os.getenv('DEP_NAME', 'x/local').split("/")[1]
VERSION = 1

# for Redirection
PROJECT_DOMAIN = os.environ.get('PROJECT_DOMAIN')  # only Production environment
SUBDOMAIN_REDIRECT = True
SSL_REDIRECT = False


'''########################################################################
#######                     Django Basic                              #####
########################################################################'''

# Application definition

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django.contrib.sites',
    'django.contrib.sitemaps',
    'django.contrib.admin',
    'django.contrib.humanize',
)

MIDDLEWARE_CLASSES = (
    'django.middleware.gzip.GZipMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.locale.LocaleMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'django.middleware.security.SecurityMiddleware',
    'whitenoise.middleware.WhiteNoiseMiddleware',
)

ROOT_URLCONF = '<projet>.urls'

WSGI_APPLICATION = '<project>.wsgi.application'

LANGUAGES = (
    # ('de', 'Deutsch'),
    ('en', 'English'),
)

LOCALE_PATHS = (
    os.path.join(PROJECT_ROOT, 'locale'),
)

LANGUAGE_CODE = 'en'

TIME_ZONE = 'Europe/Berlin'

USE_I18N = True

USE_L10N = True

USE_TZ = True

SITE_ID = 1


'''########################################################################
#######                     HEROKU DEPLOYMENT                        #####
########################################################################'''

config = {
   key: value for key, value in os.environ.items()
}
aws_database_url = config.get('RDS_DB_NAME', None)
heroku_database_url = config.get('DATABASE_URL', None)
docker = config.get('DOCKER', None)

if aws_database_url:
    # aws production settings
    DEBUG = False
    SECRET_KEY = config.get('SECRET_KEY')
    ALLOWED_HOSTS = ['*']
    # For sentry settings
    INSTALLED_APPS += (
       'raven.contrib.django.raven_compat',
    )
    RAVEN_CONFIG = {
       'dsn': config.get('SENTRY_DSN', ''),
    }
    DATABASES = {
        'default': {
           'ENGINE': 'django.db.backends.postgresql',
           'NAME': os.environ['RDS_DB_NAME'],
           'USER': os.environ['RDS_USERNAME'],
           'PASSWORD': os.environ['RDS_PASSWORD'],
           'HOST': os.environ['RDS_HOSTNAME'],
           'PORT': os.environ['RDS_PORT'],
        }
    }
elif heroku_database_url:
    # heroku production settings
    DEBUG = False
    SECRET_KEY = config.get('SECRET_KEY')
    ALLOWED_HOSTS = ['*']
    import dj_database_url
    DATABASES = {
       'default': dj_database_url.parse(heroku_database_url),
    }
    # For sentry settings
    INSTALLED_APPS += (
       'raven.contrib.django.raven_compat',
    )
    RAVEN_CONFIG = {
       'dsn': config.get('SENTRY_DSN', ''),
    }
elif docker:
    # Docker settings
    DEBUG = True
    cred_file = open(os.path.join(PROJECT_ROOT, 'local_config.json'))
    creds = json.load(cred_file)
    config = creds.get('CONFIG')
    SECRET_KEY = config.get("SECRET_KEY")
    ALLOWED_HOSTS = ['0.0.0.0']
    DATABASES = {
       'default': {
           'ENGINE': 'django.db.backends.postgresql',
           'NAME': 'postgres',
           'USER': 'postgres',
           'HOST': 'db',
           'PORT': 5432,
       }
    }
else:
    # development/test settings
    DEBUG = True
    cred_file = open(os.path.join(PROJECT_ROOT, 'local_config.json'))
    creds = json.load(cred_file)
    config = creds.get('CONFIG')
    SECRET_KEY = config.get("SECRET_KEY")
    ALLOWED_HOSTS = []
    DATABASES = {
       'default': {
           'ENGINE': 'django.db.backends.postgresql',
           'NAME': '<project>',
       }
    }

'''########################################################################
#######                    Templating                                 #####
########################################################################'''

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [
            '{0}/templates/'.format(PROJECT_ROOT),
        ],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.contrib.auth.context_processors.auth',
                'django.template.context_processors.debug',
                'django.template.context_processors.i18n',
                'django.template.context_processors.media',
                'django.template.context_processors.static',
                'django.template.context_processors.tz',
                'django.contrib.messages.context_processors.messages',
                'django.template.context_processors.request',
            ],
        },
    },
]


'''########################################################################
#######             File Handling + Storage                           #####
########################################################################'''

STATICFILES_DIRS = (
    os.path.join(PROJECT_ROOT, 'static'),
)

STATIC_URL = '/static/'

STATIC_ROOT = os.path.join(PROJECT_ROOT, 'staticfiles')


'''########################################################################
#######                    Caching                                    #####
########################################################################'''

# ToDo: Add Redis settings
open_redis_url = config.get('REDIS_URL')
if open_redis_url:
    CACHES = {
        'default': {
            'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
        }
    }
else:
    CACHES = {
        "default": {
            "BACKEND": "django_redis.cache.RedisCache",
            "LOCATION": open_redis_url,
            "OPTIONS": {
                "CLIENT_CLASS": "django_redis.client.DefaultClient",
                "IGNORE_EXCEPTIONS": True,
            },
            'KEY_PREFIX': DEP_NAME,
            'VERSION': VERSION,
            'TIMEOUT': 86400,  # 1 day
        },
    }

CACHING_MAX_AGE_BASIC = 60 * 60 * 24
SESSION_ENGINE = "django.contrib.sessions.backends.db"
# Cacheable files and compression support using whitenoise
STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'


'''########################################################################
#######                     APIs + Other Tools                        #####
########################################################################'''

SUIT_CONFIG = {
    'ADMIN_NAME': 'Dr. Shubham Dipt',
    'MENU_EXCLUDE': ('auth.group', 'auth', 'site'),
    'MENU_OPEN_FIRST_CHILD': True
}

CRISPY_TEMPLATE_PACK = 'bootstrap3'


'''########################################################################
#######             Debug Toolbar settings                          #####
########################################################################'''

# Debug toolbar setting
if DEBUG:
    INTERNAL_IPS = ('127.0.0.1',)
    INSTALLED_APPS += (
        'debug_toolbar',
    )
    MIDDLEWARE_CLASSES += (
        'debug_toolbar.middleware.DebugToolbarMiddleware',
    )


'''########################################################################
#######                     Keys and Other Information               #####
########################################################################'''

# Sendwithus settings
EMAIL_TEMPLATE_LANGUAGES = {
    'de': 'de-DE',
    'en': 'en-US',
}
SENDWITHUS_API_KEY = config.get("SENDWITHUS_API_KEY", '')
# Cloudinary settings.
os.environ.setdefault("CLOUDINARY_URL", config.get('CLOUDINARY_URL', ''))
cloudinary.config(secure=True)  # use cloudinary always with SSL

and replace <project> with your project's name.

(optional for Production) Add middleware.py to the same folder as settings.py for specific redirects like HTTP to HTTPS and subdomain redirects.

Add PROJECT_DOMAIN="<your domain>" as environment in production server.
Add '<project>.middleware.SubdomainRedirect' to the top of the MIDDLEWARE_CLASSES in settings.py.

# -*- coding: utf-8 -*-
from django.http import HttpResponsePermanentRedirect
from django.conf import settings


class SSLRedirect:
    """Redirect the URL to SSL if option enabled."""

    def process_request(self, request):
        """Process request."""
        if not request.is_secure() and settings.SSL_REDIRECT:

            if settings.DEBUG and request.method == 'POST':
                raise RuntimeError(
                    ("Django can't perform a SSL redirect while maintaining "
                     "POST data. Please structure your views so that "
                     "redirects only occur during GETs.""")
                )

            return HttpResponsePermanentRedirect(
                u"https://{}{}".format(
                    request.get_host(),
                    request.get_full_path(),
                )
            )


class SubdomainRedirect:
    """Redirect the URL if the HOST is invalid."""

    def process_request(self, request):
        """Process request."""

        if settings.SUBDOMAIN_REDIRECT and settings.PROJECT_DOMAIN:

            if request.get_host() != settings.PROJECT_DOMAIN:
                if settings.SSL_REDIRECT:
                    redirect_url = u"https://{}{}"
                else:
                    redirect_url = u"http://{}{}"
                return HttpResponsePermanentRedirect(
                    redirect_url.format(
                        settings.PROJECT_DOMAIN,
                        request.get_full_path(),
                    )
                )

(optional) Add context_processors.py in the same folder as settings.py to determine the environment of the project. It can be useful in certain cases, for e.g., to prevent google analytics from execution or in development servers. Moreover it also serves to return the full url of the webpage.

Add '<project>.context_processors.global_settings' to the context processors in the TEMPLATES section in settings.py file.

from django.conf import settings

def global_settings(request):
    # return any necessary values

    if settings.SSL_REDIRECT:
        complete_url = u"https://{}{}"
    else:
        complete_url = u"http://{}{}"
    complete_url = complete_url.format(settings.PROJECT_DOMAIN,
                                       request.get_full_path())

    return {
        'DEP_NAME': settings.DEP_NAME,
        'WHOLE_URL': complete_url,
    }

(Good practice to keep settings as config file) Add a file called local_config.json in project's folder.
```
{
  "CONFIG": {
      "SENDWITHUS_API_KEY": "XXX",
      "SECRET_KEY": "XXX",
      "CLOUDINARY_URL": "XXX"
    }
}
```
Note: Keep private keys in this file for running the application locally and as environment variables in production environment.
FRONTEND set up using Node.js, Grunt, Pug
- Add the static files in static folder.
- Add the templates in templates folder.
- Add package.json, bower.json and Gruntfile.js

PRODUCTION Set up

Requirements settings: create a folder called requirements and inside that create three files
base.txt - which contains the basic libraries to be installed
local.txt - which extends base.txt and contains the libraries specific for testing in local environment as follows:
```
-r base.txt
model_mommy==1.2.5
ecdsa==0.13
Fabric3==1.12.post1
paramiko==1.17.2
pycrypto==2.6.1
```
production.txt - which extends base.txt as follows:
```
-r base.txt
dj-database-url==0.4.1
gunicorn==19.6.0
```

Add gunicorn_config.py for gunicorn configuration.

import os
import multiprocessing

size = int(os.getenv("SIZE", 1))
# Maximum 5 worker per 128MB
workers = min(multiprocessing.cpu_count() * 2 + 1, size * 1)
timeout = int(os.getenv("GUNICORN_TIMEOUT", 120))

Deployment

Using HEROKU

Add Procfile

web: gunicorn <project>.wsgi:application --config gunicorn_config.py

Add runtime.txt
```
python-3.5.2
```

Add fabfile.py for easy deployment using Fabric (note: fabric should be included in local.txt in requirements folder) as follows:

# -*- coding: utf-8 -*-
import os
import json
import random
import requests
from fabric.colors import green, yellow, red, cyan, white, blue
from fabric.contrib.console import confirm
from fabric.api import runs_once, lcd, local, task


APP_NAME = '<project>'
DJANGO_SETTINGS_MODULE = '%s.settings' % APP_NAME

SLACK_URL = None

installations = {
    'master': 'master',            # PRODUCTION
    'staging': 'staging',          # STAGING (things REALLY should work here)
    'development': 'development',  # DEV (things get merged here)
}

heroku_app_base_url = 'https://%s-%s.herokuapp.com'

urls = {
    'master': heroku_app_base_url % (APP_NAME, 'master'),
    'staging': heroku_app_base_url % (APP_NAME, 'staging'),
    'development': heroku_app_base_url % (APP_NAME, 'development'),
}


'''########################################################################
#######                     General FABRIC tasks                      #####
########################################################################'''


@task
def heroku_login():
    """ For Heroku logging in."""
    local('heroku login')
    

@task
def push_and_deploy(deployment=None, force_push=False):
    """Push code to Git and then push to Heroku for deployment."""

    # Getting the current git branch
    all_git_branch = local("git branch", capture=True).split('\n')
    current_branch = None
    for br in all_git_branch:
        if '*' in br:
            current_branch = br[2:]
            print(yellow("Current branch: {}".format(current_branch)))
    if not current_branch:
        print(red("No git branch initiated"))
        return

    # Verifying current branch and deployment.
    if deployment:
        if current_branch != deployment:
            print(red("Branch mismatch: deploying wrong branch to {}-{}".format(
                APP_NAME, deployment)))
            return
    else:
        deployment = current_branch

    # Messaging
    bar_length = 20 + len(deployment)
    print(green(" "))
    print(green("=" * bar_length))
    print(white('Pushing to Heroku: {}'.format(deployment)))
    print(green("=" * bar_length))

    app_name_branch = '%s-%s' % (APP_NAME, deployment)
    heroku_login()

    local_push_cmd = 'git push origin {}'.format(deployment)
    add_remote = 'heroku git:remote -a {}'.format(app_name_branch)
    push_cmd = 'git push heroku {}:master'.format(deployment)

    if force_push:
        local_push_cmd = "%s -f" % local_push_cmd
        push_cmd = "%s -f" % push_cmd

    print(green('Pushing to Git .....'))
    local(local_push_cmd)

    print(green('Adding remote Heroku app .....'))
    local(add_remote)

    print(green('Pushing to Heroku .. '))
    print(yellow('Initiating a deployment to APP: {}'.format(app_name_branch)))
    local(push_cmd)

Good practice: create heroku apps as <app_name>-master or <app_name>-development.
APP_NAME should be defined in fabfile.py.
Run in terminal: fab push_and_deploy
(note: this command deploys the <current-branch> to app_name-<current-branch> only.)
TIPS:
heroku run 'bash' --app <project>-master (to run bash in heroku)
- python manage.py createsuperuser (to create superuser in heroku app)

Using AWS Elastic Beanstalk

Install awsebcli
pip install awsebcli
eb init (and enter details of the aws environment)

Create a folder in the project folder called .ebextensions and add the following the files:

01_packages.config

packages:
  yum:
    git: []
    postgresql95-devel: []
    libjpeg-turbo-devel: []

02_python.config

option_settings:
 "aws:elasticbeanstalk:application:environment":
   DJANGO_SETTINGS_MODULE: "<project>.settings"
   "PYTHONPATH": "/opt/python/current/app:$PYTHONPATH"
 "aws:elasticbeanstalk:container:python":
   WSGIPath: <project>/wsgi.py
   NumProcesses: 3
   NumThreads: 20
 "aws:elasticbeanstalk:container:python:staticfiles":
   "/static/": "staticfiles/"
container_commands:
 01_migrate:
   command: "source /opt/python/run/venv/bin/activate && python manage.py migrate --noinput"
   leader_only: true
 02_collectstatic:
   command: "source /opt/python/run/venv/bin/activate && python manage.py collectstatic --noinput"

eb create (to create the environment in aws)
eb deploy (to deploy in the created environment)
TIPS:
sudo eb ssh <aws app name> (for ssh in the aws environment, note: ssh keys should be in the .ssh folder)
- cd /opt/python/current/app/ (to change to django app folder)
- source ../env activate (activates the virtual environment)
- python manage.py createsuperuser (to create superuser in aws application)

Create Environment variables in production environment
- SECRET_KEY=<secret-key for django>
- PROJECT_DOMAIN=<root domain>
- DEP_NAME=<project>/master (e.g. for master)
- REDIS_URL=<redis url for cache>
- SENDWITHUS_API_KEY=<sendwithus-key>
- SENTRY_DSN=<sentry-dsn-url>
- CLOUDINARY_URL=<cloudinary-url>
- Database settings:
  - Heroku: DATABASE_URL=<database-url>
  - AWS:
    - RDS_DB_NAME=<RDS_DB_NAME>
    - RDS_USERNAME=<RDB_USERNAME>
    - RDS_PASSWORD=<RDS_PASSWORD>
    - RDS_HOSTNAME=<RDS_HOSTNAME>
    - RDS_PORT=<RDS_PORT>

That's it ENJOY !