Revert back to previous Django Migration

This procedure is only for the development environment while actively working on changes. Say you have several migrations already created but since after making changes realized that some of them are unnecessary and you want to come up with just one migration before checking in code for production deploy. This procedure will revert back migrations to previous level and then will give opportunity to create just one migration file that can then be deployed to production. This can potentially damage databases if not applied properly so use your discernment.

1. Show migrations for an app
$ python3 manage.py showmigrations orchestrator
...
 [X] 0051_auto_20191217_2328
 [X] 0052_auto_20200212_2250
 [X] 0053_job_miscellaneous

2. If you want to revert back the last 2 migrations
$ python3 manage.py migrate orchestrator 0051_auto_20191217_2328
Operations to perform:
  Target specific migration: 0051_auto_20191217_2328, from orchestrator
Running migrations:
  Rendering model states... DONE
  Unapplying orchestrator.0053_job_miscellaneous... OK
  Unapplying orchestrator.0052_auto_20200212_2250... OK

3. Show migrations again
$ python3 manage.py showmigrations orchestrator
...
 [X] 0051_auto_20191217_2328
 [ ] 0052_auto_20200212_2250
 [ ] 0053_job_miscellaneous

4. Delete the undesired migration files
$ rm 0052_auto_20200212_2250.py 0053_job_miscellaneous.py

5. Show migrations again
$ python3 manage.py showmigrations orchestrator
...
 [X] 0051_auto_20191217_2328

6. Now make a new migration. This will have all latest changes in model
$ python3 manage.py makemigrations
Migrations for 'orchestrator':
  orchestrator/migrations/0052_job_miscellaneous.py
    - Add field miscellaneous to job

7. Now apply new migration
$ python3 manage.py migrate
Running migrations:
  Applying orchestrator.0052_job_miscellaneous... OK

Get Custom Settings in Django

1. Insert additional settings this way...

In settings.py, or settings/development.py, can insert whatever data structure...

RESOURCE_CREDENTIALS = {
    'client': {
        'USERNAME': 'root',
        'PASSWORD': 'abc123',
    },
    'server': {
        'USERNAME': 'root',
        'PASSWORD': 'ABC123!',
    }
}


2. Then from the app access it this way...

e.g., in views.py...

from django.conf import settings

RESOURCE_CREDENTIALS = getattr(settings, 'RESOURCE_CREDENTIALS')
USERNAME = RESOURCE_CREDENTIALS["client"]["USERNAME"]
PASSWORD = RESOURCE_CREDENTIALS["client"]["PASSWORD"]

Celery Tasks Parallel and Chained Execution Workflow

In celery, it is very easy to chain and parallelize execution of tasks, e.g. to satisfy this example workflow...

      _____
     |task1|      1. exec task1
    /      \
 __/__    __\__
|task2|  |task3|  2. parallel task1 & task2
   \        /
    \ _____/
     |task4|      3. exec task4 when both task1 & task2 are done

from celery import chain, group, chord
from orchestrator.tasks.host_tasks import acquire_hosts, prepare_hosts, return_hosts
from orchestrator.tasks.job_tasks import start_job, stop_job
from orchestrator.tasks.vcenter_tasks import acquire_vcenter, return_vcenter


def start_performance_run_workflow(job_id):
    print('Starting performance run workflows for job %s' % job_id)
    # The effect of this chain, is that only the first one is routed
    # to the "regular.priority" queue
    # subsequent tasks are routed to "celery" default queue
    workflow = chain(
        acquire_hosts.si(job_id),
        chord(
            (prepare_hosts.si(job_id), acquire_vcenter.si(job_id)),
            start_job.si(job_id)
        )
    ).apply_async(queue="regular.priority")

Django LDAP Logging

Got from here

Add in settings.py

import logging, logging.handlers
logfile = "/tmp/django-ldap-debug.log"
my_logger = logging.getLogger('django_auth_ldap')
my_logger.setLevel(logging.DEBUG)
handler = logging.handlers.RotatingFileHandler(
   logfile, maxBytes=1024 * 500, backupCount=5)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
my_logger.addHandler(handler)

Django Development Production Settings

I choose the 2nd solution from this post

Dir structure:
[myapp]$ tree
.
├── __init__.py
├── settings
│   ├── __init__.py
│   ├── defaults.py
│   ├── development.py
│   ├── production.py
│   └── staging.py

### __init.py__
from myapp.settings.development import *

### defaults.py
### sensible choices for default settings

### dev.py
from myapp.settings.defaults import *
DEBUG = True
### other development-specific stuff

### production.py
from myapp.settings.defaults import *
DEBUG = False
### other production-specific stuff

### staging.py
from myapp.settings.defaults import *
DEBUG = True
### other staging-specific stuff

Django Data Seeding

This is a more flexible way of seeding data

1. Create required directories and files
$ mkdir testapp/management
$ mkdir testapp/management/commands
$ touch testapp/management/__init__.py
$ touch testapp/management/commands/__init__.py

2. Write code that populates the db
$ vi testapp/management/commands/populate_db.py
from django.core.management.base import BaseCommand
from testapp.models import Region
from django.db import connection

class Command(BaseCommand):
    args = ''
    help = 'Management script that populates the database with initial seed'

    def _truncate_region(self):
        cursor = connection.cursor()
        cursor.execute("TRUNCATE TABLE `testapp_region`")

    def _create_regions(self):
        regions = (
        ('REG001', [
                {"name": "vmk-4027","vlanId":4027,"usefor":["vsan","nfs"]},
                {"name": "vmk-4028","vlanId":4028,"usefor":["traffic"]},
                {"name": "vmk-102","vlanId":102,"usefor":["pub"]},
            ]),
        ('REG002', [
                {"name": "vmk-4029","vlanId":4029,"usefor":["vsan","nfs"]},
                {"name": "vmk-4030","vlanId":4030,"usefor":["traffic"]},
                {"name": "vmk-102","vlanId":103,"usefor":["pub"]},
            ]),
        )
        for item in regions:
            region = Region(name=item[0], vlans=item[1])
            region.save()

    def handle(self, *args, **options):
        self._truncate_region()
        self._create_regions()

3. Execute the program
$ python manage.py populate_db

Async Background Tasks with Celery on Django

From this post and this post and this post

1. Install Redis - got from here
$ brew install redis
$ ln -sfv /usr/local/opt/redis/*.plist ~/Library/LaunchAgents
$ launchctl load ~/Library/LaunchAgents/homebrew.mxcl.redis.plist
$ redis-cli PING
PONG

2. Install celery with redis support bundle
$ pip3 install -U "celery[redis]"

3. Create celery.py in django project
$ vi sample_project/sample_project/celery.py
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'sample_project.settings')

app = Celery('sample_project')

# Using a string here means the worker don't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()


@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

4. Import celery app in __init__.py
from __future__ import absolute_import, unicode_literals

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app

__all__ = ['celery_app']

5. Add config in settings.py
$ vi sample_project/sample_project/settings.py
# celery
CELERY_BROKER_URL = 'redis://localhost:6379/0'

6. Create app
$ python3 manage.py startapp testapp
$ vi sample_project/sample_project/settings.py
INSTALLED_APPS = (
    # (...)
    'testapp',
)

7. Create task
$ vi sample_project/sample_project/testapp/tasks.py
from __future__ import absolute_import

from celery import shared_task

@shared_task
def test(param):
    return 'The test task executed with argument "%s" ' % param

8. Testing

Run server
$ python3 manage.py runserver

Run worker
$ celery -A sample_project worker --loglevel=info

From another terminal...
$ python3 manage.py shell
Python 3.6.2 (default, Jul 17 2017, 16:44:45)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from testapp.tasks import test
>>> test.delay('This is a test')


From the worker terminal
2017-07-26 00:50:54,529: INFO/MainProcess] Received task: testapp.tasks.test[4f1ab1d4-645d-480b-a1b5-6e2887e34b52]
[2017-07-26 00:50:54,533: INFO/ForkPoolWorker-2] Task testapp.tasks.test[4f1ab1d4-645d-480b-a1b5-6e2887e34b52] succeeded in 0.0010309600038453937s: 'The test task executed with argument "This is a test" '

9. For results backend using Django ORM

a. $ pip3 install django-celery-results

b. Install app in settings.py
INSTALLED_APPS = (
    ...,
    'django_celery_results',
)

c. Create celery database tables
$ python3 manage.py migrate django_celery_results

d. Configure Celery to use the django-celery-results backend.
In settings.py...
CELERY_RESULT_BACKEND = 'django-db'

Django with Postgres on Mac

1. Create virtual env following this post
2. Install django
(my_env) $ pip3 install psycopg2
(my_env) $ pip3 install django
3. Create django project
(my_env) $ django-admin startproject sample_project
(my_env) $ python3 manage.py runserver
4. Install postgres following this post
5. Create database
$ createdb -U postgres testdb
6. Modify settings in django
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'testdb',
        'USER': 'postgres',
        'PASSWORD': '',
        'HOST': 'localhost',
        'PORT': '5432',
    }
}
7. Migrate
$ python3 manage.py migrate
8. Create superuser
$ python3 manage.py createsuperuser

DRF JWT Authentication

i.e., Django REST Framework with JSON Web Token Authentication. Got the solution from here: http://getblimp.github.io/django-rest-framework-jwt/ http://zqpythonic.qiniucdn.com/data/20141006233346/index.html

$ pip install djangorestframework-jwt

In settings.py:
REST_FRAMEWORK = {
    'DEFAULT_PERMISSION_CLASSES': (
        'rest_framework.permissions.IsAuthenticated',
    ),
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.BasicAuthentication',
        'rest_framework.authentication.SessionAuthentication',
        'rest_framework_jwt.authentication.JSONWebTokenAuthentication',
    ),
}

In urls.py:
from rest_framework_jwt.views import obtain_jwt_token, refresh_jwt_token

urlpatterns = patterns(
    # ...
    url(r'^api-token-auth/', obtain_jwt_token),
    url(r'^api-token-refresh/', refresh_jwt_token)
)

To test:
$ curl -X POST -d "username=admin&password=password123" http://localhost:8000/api-token-auth/
$ curl -X POST -H "Content-Type: application/json" -d '{"username":"admin","password":"password123"}' http://localhost:8000/api-token-auth/
$ curl -H "Authorization: JWT " http://localhost:8000/protected-url/

Django LDAP Integration

These posts helped me: http://kacperdziubek.pl/python/django-ldap-open-directory-integration/ https://pythonhosted.org/django-auth-ldap/

Install:
$ pip install django-auth-ldap

Then in settings.py, add:
import ldap
from django_auth_ldap.config import LDAPSearch

AUTHENTICATION_BACKENDS = (
    'django.contrib.auth.backends.ModelBackend',
    'django_auth_ldap.backend.LDAPBackend',
)

AUTH_LDAP_SERVER_URI = "ldap://my.appauth.com" # ip or host name of Open Directory server
AUTH_LDAP_BIND_DN = "CN=Accounts,OU=US Security,DC=corp,DC=com"
AUTH_LDAP_BIND_PASSWORD = "MySecurePassword"
AUTH_LDAP_USER_SEARCH = LDAPSearch("OU=US Users,dc=corp,dc=com",
    ldap.SCOPE_SUBTREE, "(sAMAccountName=%(user)s)")
AUTH_LDAP_CONNECTION_OPTIONS = {
    # make search fast
    ldap.OPT_REFERRALS: 0
}

AUTH_LDAP_USER_ATTR_MAP = {
    "username": "sAMAccountName",
    "first_name": "givenName",
    "last_name": "sn",
    "email": "mail"
}

The mapping above would result into automatically saving the information to the users table.

Django Notes

1. Backup and seed data using fixtures
# Save data
$ python manage.py dumpdata --format=json myapp > myapp/fixtures/initial_data.json
# Load data
$ python manage.py loaddata vlanapi/fixtures/initial_data.json

2. Open python shell - interactive console
$ python manage.py shell
Python 2.7.12 (default, Jul  1 2016, 15:12:24) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>

3. Start a project
$ django-admin startproject mysite

4. Start initial migration
$ python manage.py migrate

5. Start server
$ python manage.py runserver

6. Create an app
$ python manage.py startapp newapp

7. Create and run a migration
$ python manage.py makemigrations
$ python manage.py migrate