Django Full Stack Set Up and Deployment

14th February, 2019

After installing all the basic set up for Django web deveopment ( http://www.shubhamdipt.com/blog/mac-set-up-for-django-web-development )

Step by Step process: (for Django, PostgreSQL)
[ replace <project> by your project's name]

  1. Create virtual environment
    mkvirtualenv <project>
  2. Activate the environment
    workon <project>
  3. Install Django
    pip install django
  4. Create a Django project
    django-admin startproject <project>
  5. Enter the project diretory
    cd <project>
  6. Initiate Git (version control)
    git init
  7. Add remote repository to git
    git add remote <repo name>
  8. Create .gitgnore file
    vim .gitignore
  9. Create directories: static, templates, locale, requirements
  10. Inside requirements folder, create base.txt, local.txt, production.txt files.
  11. Create a database in postgresql <project_name> using pgAdmin.
  12. Install psycopg2 for postgresql.
    pip install psycopg2
  13. Install whitenoise to serve static files in production.
    pip install whitenoise
  14. Install raven for Sentry logging.
    pip instal raven
  15. Install cloudinary
    pip install cloudinary
  16. Save the requirements in requirements.txt file.
    pip freeze > requirements.txt
  17. Copy the follwing to the settings.py
    """
    Django settings for <project> project.
    
    Generated by 'django-admin startproject' using Django 1.10.5.
    
    For more information on this file, see
    https://docs.djangoproject.com/en/1.10/topics/settings/
    
    For the full list of settings and their values, see
    https://docs.djangoproject.com/en/1.10/ref/settings/
    """
    
    import os
    import json
    import cloudinary
    import raven
    
    
    PROJECT_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
    DEP_NAME = os.getenv('DEP_NAME', 'x/local').split("/")[1]
    VERSION = 1
    
    # for Redirection
    PROJECT_DOMAIN = os.environ.get('PROJECT_DOMAIN')  # only Production environment
    SUBDOMAIN_REDIRECT = True
    SSL_REDIRECT = False
    
    
    '''########################################################################
    #######                     Django Basic                              #####
    ########################################################################'''
    
    # Application definition
    
    INSTALLED_APPS = (
        'django.contrib.auth',
        'django.contrib.contenttypes',
        'django.contrib.sessions',
        'django.contrib.messages',
        'django.contrib.staticfiles',
        'django.contrib.sites',
        'django.contrib.sitemaps',
        'django.contrib.admin',
        'django.contrib.humanize',
    )
    
    MIDDLEWARE_CLASSES = (
        'django.middleware.gzip.GZipMiddleware',
        'django.contrib.sessions.middleware.SessionMiddleware',
        'django.middleware.locale.LocaleMiddleware',
        'django.middleware.common.CommonMiddleware',
        'django.middleware.csrf.CsrfViewMiddleware',
        'django.contrib.auth.middleware.AuthenticationMiddleware',
        'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
        'django.contrib.messages.middleware.MessageMiddleware',
        'django.middleware.clickjacking.XFrameOptionsMiddleware',
        'django.middleware.security.SecurityMiddleware',
        'whitenoise.middleware.WhiteNoiseMiddleware',
    )
    
    ROOT_URLCONF = '<projet>.urls'
    
    WSGI_APPLICATION = '<project>.wsgi.application'
    
    LANGUAGES = (
        # ('de', 'Deutsch'),
        ('en', 'English'),
    )
    
    LOCALE_PATHS = (
        os.path.join(PROJECT_ROOT, 'locale'),
    )
    
    LANGUAGE_CODE = 'en'
    
    TIME_ZONE = 'Europe/Berlin'
    
    USE_I18N = True
    
    USE_L10N = True
    
    USE_TZ = True
    
    SITE_ID = 1
    
    
    '''########################################################################
    #######                     HEROKU DEPLOYMENT                        #####
    ########################################################################'''
    
    config = {
       key: value for key, value in os.environ.items()
    }
    aws_database_url = config.get('RDS_DB_NAME', None)
    heroku_database_url = config.get('DATABASE_URL', None)
    docker = config.get('DOCKER', None)
    
    if aws_database_url:
        # aws production settings
        DEBUG = False
        SECRET_KEY = config.get('SECRET_KEY')
        ALLOWED_HOSTS = ['*']
        # For sentry settings
        INSTALLED_APPS += (
           'raven.contrib.django.raven_compat',
        )
        RAVEN_CONFIG = {
           'dsn': config.get('SENTRY_DSN', ''),
        }
        DATABASES = {
            'default': {
               'ENGINE': 'django.db.backends.postgresql',
               'NAME': os.environ['RDS_DB_NAME'],
               'USER': os.environ['RDS_USERNAME'],
               'PASSWORD': os.environ['RDS_PASSWORD'],
               'HOST': os.environ['RDS_HOSTNAME'],
               'PORT': os.environ['RDS_PORT'],
            }
        }
    elif heroku_database_url:
        # heroku production settings
        DEBUG = False
        SECRET_KEY = config.get('SECRET_KEY')
        ALLOWED_HOSTS = ['*']
        import dj_database_url
        DATABASES = {
           'default': dj_database_url.parse(heroku_database_url),
        }
        # For sentry settings
        INSTALLED_APPS += (
           'raven.contrib.django.raven_compat',
        )
        RAVEN_CONFIG = {
           'dsn': config.get('SENTRY_DSN', ''),
        }
    elif docker:
        # Docker settings
        DEBUG = True
        cred_file = open(os.path.join(PROJECT_ROOT, 'local_config.json'))
        creds = json.load(cred_file)
        config = creds.get('CONFIG')
        SECRET_KEY = config.get("SECRET_KEY")
        ALLOWED_HOSTS = ['0.0.0.0']
        DATABASES = {
           'default': {
               'ENGINE': 'django.db.backends.postgresql',
               'NAME': 'postgres',
               'USER': 'postgres',
               'HOST': 'db',
               'PORT': 5432,
           }
        }
    else:
        # development/test settings
        DEBUG = True
        cred_file = open(os.path.join(PROJECT_ROOT, 'local_config.json'))
        creds = json.load(cred_file)
        config = creds.get('CONFIG')
        SECRET_KEY = config.get("SECRET_KEY")
        ALLOWED_HOSTS = []
        DATABASES = {
           'default': {
               'ENGINE': 'django.db.backends.postgresql',
               'NAME': '<project>',
           }
        }
    
    '''########################################################################
    #######                    Templating                                 #####
    ########################################################################'''
    
    TEMPLATES = [
        {
            'BACKEND': 'django.template.backends.django.DjangoTemplates',
            'DIRS': [
                '{0}/templates/'.format(PROJECT_ROOT),
            ],
            'APP_DIRS': True,
            'OPTIONS': {
                'context_processors': [
                    'django.contrib.auth.context_processors.auth',
                    'django.template.context_processors.debug',
                    'django.template.context_processors.i18n',
                    'django.template.context_processors.media',
                    'django.template.context_processors.static',
                    'django.template.context_processors.tz',
                    'django.contrib.messages.context_processors.messages',
                    'django.template.context_processors.request',
                ],
            },
        },
    ]
    
    
    '''########################################################################
    #######             File Handling + Storage                           #####
    ########################################################################'''
    
    STATICFILES_DIRS = (
        os.path.join(PROJECT_ROOT, 'static'),
    )
    
    STATIC_URL = '/static/'
    
    STATIC_ROOT = os.path.join(PROJECT_ROOT, 'staticfiles')
    
    
    '''########################################################################
    #######                    Caching                                    #####
    ########################################################################'''
    
    # ToDo: Add Redis settings
    open_redis_url = config.get('REDIS_URL')
    if open_redis_url:
        CACHES = {
            'default': {
                'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
            }
        }
    else:
        CACHES = {
            "default": {
                "BACKEND": "django_redis.cache.RedisCache",
                "LOCATION": open_redis_url,
                "OPTIONS": {
                    "CLIENT_CLASS": "django_redis.client.DefaultClient",
                    "IGNORE_EXCEPTIONS": True,
                },
                'KEY_PREFIX': DEP_NAME,
                'VERSION': VERSION,
                'TIMEOUT': 86400,  # 1 day
            },
        }
    
    CACHING_MAX_AGE_BASIC = 60 * 60 * 24
    SESSION_ENGINE = "django.contrib.sessions.backends.db"
    # Cacheable files and compression support using whitenoise
    STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
    
    
    '''########################################################################
    #######                     APIs + Other Tools                        #####
    ########################################################################'''
    
    SUIT_CONFIG = {
        'ADMIN_NAME': 'Dr. Shubham Dipt',
        'MENU_EXCLUDE': ('auth.group', 'auth', 'site'),
        'MENU_OPEN_FIRST_CHILD': True
    }
    
    CRISPY_TEMPLATE_PACK = 'bootstrap3'
    
    
    '''########################################################################
    #######             Debug Toolbar settings                          #####
    ########################################################################'''
    
    # Debug toolbar setting
    if DEBUG:
        INTERNAL_IPS = ('127.0.0.1',)
        INSTALLED_APPS += (
            'debug_toolbar',
        )
        MIDDLEWARE_CLASSES += (
            'debug_toolbar.middleware.DebugToolbarMiddleware',
        )
    
    
    '''########################################################################
    #######                     Keys and Other Information               #####
    ########################################################################'''
    
    # Sendwithus settings
    EMAIL_TEMPLATE_LANGUAGES = {
        'de': 'de-DE',
        'en': 'en-US',
    }
    SENDWITHUS_API_KEY = config.get("SENDWITHUS_API_KEY", '')
    # Cloudinary settings.
    os.environ.setdefault("CLOUDINARY_URL", config.get('CLOUDINARY_URL', ''))
    cloudinary.config(secure=True)  # use cloudinary always with SSL
    
    and replace <project> with your project's name.


  18. (optional for Production) Add middleware.py to the same folder as settings.py for specific redirects like HTTP to HTTPS and subdomain redirects.

    Add PROJECT_DOMAIN="<your domain>" as environment in production server.
    Add  '<project>.middleware.SubdomainRedirect' to the top of the MIDDLEWARE_CLASSES in settings.py.

    # -*- coding: utf-8 -*-
    from django.http import HttpResponsePermanentRedirect
    from django.conf import settings
    
    
    class SSLRedirect:
        """Redirect the URL to SSL if option enabled."""
    
        def process_request(self, request):
            """Process request."""
            if not request.is_secure() and settings.SSL_REDIRECT:
    
                if settings.DEBUG and request.method == 'POST':
                    raise RuntimeError(
                        ("Django can't perform a SSL redirect while maintaining "
                         "POST data. Please structure your views so that "
                         "redirects only occur during GETs.""")
                    )
    
                return HttpResponsePermanentRedirect(
                    u"https://{}{}".format(
                        request.get_host(),
                        request.get_full_path(),
                    )
                )
    
    
    class SubdomainRedirect:
        """Redirect the URL if the HOST is invalid."""
    
        def process_request(self, request):
            """Process request."""
    
            if settings.SUBDOMAIN_REDIRECT and settings.PROJECT_DOMAIN:
    
                if request.get_host() != settings.PROJECT_DOMAIN:
                    if settings.SSL_REDIRECT:
                        redirect_url = u"https://{}{}"
                    else:
                        redirect_url = u"http://{}{}"
                    return HttpResponsePermanentRedirect(
                        redirect_url.format(
                            settings.PROJECT_DOMAIN,
                            request.get_full_path(),
                        )
                    )
    ​
  19. (optional) Add context_processors.py in the same folder as settings.py to determine the environment of the project. It can be useful in certain cases, for e.g., to prevent google analytics from execution or in development servers. Moreover it also serves to return the full url of the webpage.

    Add '<project>.context_processors.global_settings' to the context processors in the TEMPLATES section in settings.py file.
    from django.conf import settings
    
    def global_settings(request):
        # return any necessary values
    
        if settings.SSL_REDIRECT:
            complete_url = u"https://{}{}"
        else:
            complete_url = u"http://{}{}"
        complete_url = complete_url.format(settings.PROJECT_DOMAIN,
                                           request.get_full_path())
    
        return {
            'DEP_NAME': settings.DEP_NAME,
            'WHOLE_URL': complete_url,
        }​


  20. (Good practice to keep settings as config file) Add a file called local_config.json in project's folder.
    {
      "CONFIG": {
          "SENDWITHUS_API_KEY": "XXX",
          "SECRET_KEY": "XXX",
          "CLOUDINARY_URL": "XXX"
        }
    }​

    Note: Keep private keys in this file for running the application locally and as environment variables in production environment.

  21. FRONTEND set up using Node.js, Grunt, Pug
    • Add the static files in static folder.
    • Add the templates in templates folder.
    • Add package.json, bower.json and Gruntfile.js
  22. PRODUCTION Set up
    • Requirements settings: create a folder called requirements and inside that create three files 
      base.txt - which contains the basic libraries to be installed
      local.txt - which extends base.txt and contains the libraries specific for testing in local environment as follows:
      -r base.txt
      model_mommy==1.2.5
      ecdsa==0.13
      Fabric3==1.12.post1
      paramiko==1.17.2
      pycrypto==2.6.1

      production.txt - which extends base.txt as follows:
      -r base.txt
      dj-database-url==0.4.1
      gunicorn==19.6.0
    • Add gunicorn_config.py for gunicorn configuration.
      import os
      import multiprocessing
      
      size = int(os.getenv("SIZE", 1))
      # Maximum 5 worker per 128MB
      workers = min(multiprocessing.cpu_count() * 2 + 1, size * 1)
      timeout = int(os.getenv("GUNICORN_TIMEOUT", 120))​
    • Deployment
      • Using HEROKU
        • Add Procfile
          web: gunicorn <project>.wsgi:application --config gunicorn_config.py
        • Add runtime.txt
          python-3.5.2
        • Add fabfile.py for easy deployment  using Fabric (note: fabric should be included in local.txt in requirements folder) as follows:
          # -*- coding: utf-8 -*-
          import os
          import json
          import random
          import requests
          from fabric.colors import green, yellow, red, cyan, white, blue
          from fabric.contrib.console import confirm
          from fabric.api import runs_once, lcd, local, task
          
          
          APP_NAME = '<project>'
          DJANGO_SETTINGS_MODULE = '%s.settings' % APP_NAME
          
          SLACK_URL = None
          
          installations = {
              'master': 'master',            # PRODUCTION
              'staging': 'staging',          # STAGING (things REALLY should work here)
              'development': 'development',  # DEV (things get merged here)
          }
          
          heroku_app_base_url = 'https://%s-%s.herokuapp.com'
          
          urls = {
              'master': heroku_app_base_url % (APP_NAME, 'master'),
              'staging': heroku_app_base_url % (APP_NAME, 'staging'),
              'development': heroku_app_base_url % (APP_NAME, 'development'),
          }
          
          
          '''########################################################################
          #######                     General FABRIC tasks                      #####
          ########################################################################'''
          
          
          @task
          def heroku_login():
              """ For Heroku logging in."""
              local('heroku login')
              
          
          @task
          def push_and_deploy(deployment=None, force_push=False):
              """Push code to Git and then push to Heroku for deployment."""
          
              # Getting the current git branch
              all_git_branch = local("git branch", capture=True).split('\n')
              current_branch = None
              for br in all_git_branch:
                  if '*' in br:
                      current_branch = br[2:]
                      print(yellow("Current branch: {}".format(current_branch)))
              if not current_branch:
                  print(red("No git branch initiated"))
                  return
          
              # Verifying current branch and deployment.
              if deployment:
                  if current_branch != deployment:
                      print(red("Branch mismatch: deploying wrong branch to {}-{}".format(
                          APP_NAME, deployment)))
                      return
              else:
                  deployment = current_branch
          
              # Messaging
              bar_length = 20 + len(deployment)
              print(green(" "))
              print(green("=" * bar_length))
              print(white('Pushing to Heroku: {}'.format(deployment)))
              print(green("=" * bar_length))
          
              app_name_branch = '%s-%s' % (APP_NAME, deployment)
              heroku_login()
          
              local_push_cmd = 'git push origin {}'.format(deployment)
              add_remote = 'heroku git:remote -a {}'.format(app_name_branch)
              push_cmd = 'git push heroku {}:master'.format(deployment)
          
              if force_push:
                  local_push_cmd = "%s -f" % local_push_cmd
                  push_cmd = "%s -f" % push_cmd
          
              print(green('Pushing to Git .....'))
              local(local_push_cmd)
          
              print(green('Adding remote Heroku app .....'))
              local(add_remote)
          
              print(green('Pushing to Heroku .. '))
              print(yellow('Initiating a deployment to APP: {}'.format(app_name_branch)))
              local(push_cmd)​
        • Good practice: create heroku apps as <app_name>-master or <app_name>-development.
          APP_NAME should be defined in fabfile.py.
          Run in terminal: fab push_and_deploy
          (note: this command deploys the <current-branch> to app_name-<current-branch> only.)
        • TIPS:
          heroku run 'bash' --app <project>-master (to run bash in heroku)
          • python manage.py createsuperuser (to create superuser in heroku app)
      • Using AWS Elastic Beanstalk
        • Install awsebcli
          pip install awsebcli
        • eb init (and enter details of the aws environment)
        • Create a folder in the project folder called .ebextensions and add the following the files:

          • 01_packages.config
            packages:
              yum:
                git: []
                postgresql95-devel: []
                libjpeg-turbo-devel: []
          • 02_python.config
            option_settings:
             "aws:elasticbeanstalk:application:environment":
               DJANGO_SETTINGS_MODULE: "<project>.settings"
               "PYTHONPATH": "/opt/python/current/app:$PYTHONPATH"
             "aws:elasticbeanstalk:container:python":
               WSGIPath: <project>/wsgi.py
               NumProcesses: 3
               NumThreads: 20
             "aws:elasticbeanstalk:container:python:staticfiles":
               "/static/": "staticfiles/"
            container_commands:
             01_migrate:
               command: "source /opt/python/run/venv/bin/activate && python manage.py migrate --noinput"
               leader_only: true
             02_collectstatic:
               command: "source /opt/python/run/venv/bin/activate && python manage.py collectstatic --noinput"​
        • eb create (to create the environment in aws)
        • eb deploy (to deploy in the created environment)
        • TIPS:
          sudo eb ssh <aws app name> (for ssh in the aws environment, note: ssh keys should be in the .ssh folder)
          • cd /opt/python/current/app/ (to change to django app folder)
          • source ../env activate (activates the virtual environment)
          • python manage.py createsuperuser (to create superuser in aws application)
    • Create Environment variables in production environment
      • SECRET_KEY=<secret-key for django>
      • PROJECT_DOMAIN=<root domain>
      • DEP_NAME=<project>/master (e.g. for master)
      • REDIS_URL=<redis url for cache>
      • SENDWITHUS_API_KEY=<sendwithus-key>
      • SENTRY_DSN=<sentry-dsn-url>
      • CLOUDINARY_URL=<cloudinary-url>
      • Database settings:
        • Heroku: DATABASE_URL=<database-url>
        • AWS:
          • RDS_DB_NAME=<RDS_DB_NAME>
          • RDS_USERNAME=<RDB_USERNAME>
          • RDS_PASSWORD=<RDS_PASSWORD>
          • RDS_HOSTNAME=<RDS_HOSTNAME>
          • RDS_PORT=<RDS_PORT>

That's it ENJOY !