Celery Commands Etc

This summary is not available. Please click here to view the post.

Getting Started with Kubernetes

1. Install Minikube
$ brew update
$ brew install kubectl
$ brew cask install minikube

2. Install xhyve
$ brew install docker-machine-driver-xhyve

3. Set superuser privileges for xhyve (this appears during installation)
$ sudo chown root:wheel /usr/local/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
$ sudo chmod u+s /usr/local/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve

4. Start minikube
$ minikube start --vm-driver=xhyve

5. Test
$ kubectl config current-context
minikube

6. Get cluster nodes info
$ kubectl get nodes
NAME       STATUS    ROLES     AGE       VERSION
minikube   Ready         22m       v1.8.0

7. Stop cluster
$ minikube stop
Stopping local Kubernetes cluster...
Machine stopped.

8. Delete minikube
$ minikube delete
Deleting local Kubernetes cluster...
Machine deleted.

9. Show available kubernetes versions
$ minikube get-k8s-versions
The following Kubernetes versions are available when using the localkube bootstrapper:
 - v1.8.0
 - v1.7.5
 - v1.7.4
...

10. Get v1.8.0
$ minikube start --vm-driver=xhyve --kubernetes-version="v1.8.0"

11. Start dashboard
$ minikube dashboard

Get Custom Settings in Django

1. Insert additional settings this way...

In settings.py, or settings/development.py, can insert whatever data structure...

RESOURCE_CREDENTIALS = {
    'client': {
        'USERNAME': 'root',
        'PASSWORD': 'abc123',
    },
    'server': {
        'USERNAME': 'root',
        'PASSWORD': 'ABC123!',
    }
}


2. Then from the app access it this way...

e.g., in views.py...

from django.conf import settings

RESOURCE_CREDENTIALS = getattr(settings, 'RESOURCE_CREDENTIALS')
USERNAME = RESOURCE_CREDENTIALS["client"]["USERNAME"]
PASSWORD = RESOURCE_CREDENTIALS["client"]["PASSWORD"]

How to Integrate Clarity UI on your Angular App

I heart Clarity UI.

Install
$ yarn add clarity-icons
$ yarn add @webcomponents/custom-elements@1.0.0
$ yarn add clarity-ui
$ yarn add clarity-angular

Use clarity on Angular
$ vi .angular-cli.json
"styles": [
  ...
  "../node_modules/clarity-icons/clarity-icons.min.css",
  "../node_modules/clarity-ui/clarity-ui.min.css"
],
"scripts": [
  ...
  "../node_modules/@webcomponents/custom-elements/custom-elements.min.js",
  "../node_modules/clarity-icons/clarity-icons.min.js"
],

$ vi app.module.ts
import { ClarityModule } from "clarity-angular";
imports: [
  ...
  ClarityModule.forRoot(),
  ...
],

Angular Development Production Settings

This came from this post

Angular apps created using angular-cli will have src/environments directory.

1. Add another file in there called environment.staging.ts with the following entry...
export const environment = {
  production: false,
  envName: 'staging'
};

2. Modify .angular.cli.json
"environments": {
        "dev": "environments/environment.ts",
        "prod": "environments/environment.prod.ts",
        "staging": "environments/environment.staging.ts"
      }

3. Now when creating a build or serving the app..
$ ng build -prod
$ ng build -staging
$ ng serve -prod
$ ng serve -staging

4. To use...
import { environment } from './environment';
export class MyappAppComponent {  
  title = 'myapp works!';
  environmentName = environment.envName;
}

VS Code Cheat Sheet

Full list of shortcuts

1. Detach file to another window
⌘K, O

2. Format file
⇧ + Opt + F

3. Preview side-by-side
⌘K, V 

4. Screen shot
⇧⌘3 - whole screen
⇧⌘4 - selection

Celery Tasks Parallel and Chained Execution Workflow

In celery, it is very easy to chain and parallelize execution of tasks, e.g. to satisfy this example workflow...

      _____
     |task1|      1. exec task1
    /      \
 __/__    __\__
|task2|  |task3|  2. parallel task1 & task2
   \        /
    \ _____/
     |task4|      3. exec task4 when both task1 & task2 are done

from celery import chain, group, chord
from orchestrator.tasks.host_tasks import acquire_hosts, prepare_hosts, return_hosts
from orchestrator.tasks.job_tasks import start_job, stop_job
from orchestrator.tasks.vcenter_tasks import acquire_vcenter, return_vcenter


def start_performance_run_workflow(job_id):
    print('Starting performance run workflows for job %s' % job_id)
    # The effect of this chain, is that only the first one is routed
    # to the "regular.priority" queue
    # subsequent tasks are routed to "celery" default queue
    workflow = chain(
        acquire_hosts.si(job_id),
        chord(
            (prepare_hosts.si(job_id), acquire_vcenter.si(job_id)),
            start_job.si(job_id)
        )
    ).apply_async(queue="regular.priority")

Angular Material Table Dynamic Height

For md-table, the height can be dynamically adjusted this way...

1. sample-table.component.html
<div class="example-container mat-elevation-z8" [style.max-height.px]="tableHeight">
  ...
  <md-table #table [dataSource]="dataSource" mdSort>

2. sample-table.component.ts
import { HostListener } from '@angular/core';

export class SampleTableComponent {
  tableHeight: number;

  ngOnInit() {
    this.tableHeight = window.innerHeight - 90;
  }

  @HostListener('window:resize', ['$event'])
  onWindowResize(event){
    this.tableHeight = event.target.innerHeight - 90;
    console.log("height: " + this.tableHeight );
  }

Use httpie to play with your APIs

Got tired of using curl. I switched to httpie.

1. Install
$ pip3 install httpie

2. I use jwt
$ pip3 install -U httpie-jwt-auth

3. First get token
$ http POST :8001/api-token-auth/ 'username=admin' 'password=abc123'

4. Now set env variables
$ export JWT_AUTH_TOKEN=your_token
$ export JWT_AUTH_PREFIX=JWT

5. Now do whatever
$ http --auth-type=jwt :8001/users/

6. If this error is encountered...
SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

Use --verify=no as in...

$ http --verify=no POST ...

Django LDAP Logging

Got from here

Add in settings.py

import logging, logging.handlers
logfile = "/tmp/django-ldap-debug.log"
my_logger = logging.getLogger('django_auth_ldap')
my_logger.setLevel(logging.DEBUG)
handler = logging.handlers.RotatingFileHandler(
   logfile, maxBytes=1024 * 500, backupCount=5)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
my_logger.addHandler(handler)

Angular Starter Project with Material

Follow angular setup from this post.
Reference posts from here and here. And from a starter project.

1. Create angular project
$ ng new management-gui
$ cd management-gui

2. Get Angular Material
$ yarn add @angular/material @angular/animations @angular/cdk

3. Get HammerJs and add to .angular-cli.json
$ yarn add hammerjs
$ vi .angular-cli.json
"scripts": [
  "../node_modules/hammerjs/hammer.min.js"
],

4. Add Angular Material on to app.module.ts
$ vi src/app/app.module.ts
import { BrowserModule } from '@angular/platform-browser';
import { NgModule } from '@angular/core';
import { FormsModule } from '@angular/forms';
import { HttpModule } from '@angular/http';
import { MaterialModule } from '@angular/material';
import { BrowserAnimationsModule } from '@angular/platform-browser/animations';
import 'hammerjs';

import { AppComponent } from './app.component';

@NgModule({
  declarations: [
    AppComponent
  ],
  imports: [
    BrowserModule,
    FormsModule,
    HttpModule,
    MaterialModule,
    BrowserAnimationsModule
  ],
  providers: [],
  bootstrap: [AppComponent]
})
export class AppModule { }

5. Set global styles.css with an Angular Material theme
$ vi src/styles.css
@import '~@angular/material/prebuilt-themes/indigo-pink.css';

6. Add Material icons in index.html
$ vi src/index.html

Django Development Production Settings

I choose the 2nd solution from this post

Dir structure:
[myapp]$ tree
.
├── __init__.py
├── settings
│   ├── __init__.py
│   ├── defaults.py
│   ├── development.py
│   ├── production.py
│   └── staging.py

### __init.py__
from myapp.settings.development import *

### defaults.py
### sensible choices for default settings

### dev.py
from myapp.settings.defaults import *
DEBUG = True
### other development-specific stuff

### production.py
from myapp.settings.defaults import *
DEBUG = False
### other production-specific stuff

### staging.py
from myapp.settings.defaults import *
DEBUG = True
### other staging-specific stuff

Django Data Seeding

This is a more flexible way of seeding data

1. Create required directories and files
$ mkdir testapp/management
$ mkdir testapp/management/commands
$ touch testapp/management/__init__.py
$ touch testapp/management/commands/__init__.py

2. Write code that populates the db
$ vi testapp/management/commands/populate_db.py
from django.core.management.base import BaseCommand
from testapp.models import Region
from django.db import connection

class Command(BaseCommand):
    args = ''
    help = 'Management script that populates the database with initial seed'

    def _truncate_region(self):
        cursor = connection.cursor()
        cursor.execute("TRUNCATE TABLE `testapp_region`")

    def _create_regions(self):
        regions = (
        ('REG001', [
                {"name": "vmk-4027","vlanId":4027,"usefor":["vsan","nfs"]},
                {"name": "vmk-4028","vlanId":4028,"usefor":["traffic"]},
                {"name": "vmk-102","vlanId":102,"usefor":["pub"]},
            ]),
        ('REG002', [
                {"name": "vmk-4029","vlanId":4029,"usefor":["vsan","nfs"]},
                {"name": "vmk-4030","vlanId":4030,"usefor":["traffic"]},
                {"name": "vmk-102","vlanId":103,"usefor":["pub"]},
            ]),
        )
        for item in regions:
            region = Region(name=item[0], vlans=item[1])
            region.save()

    def handle(self, *args, **options):
        self._truncate_region()
        self._create_regions()

3. Execute the program
$ python manage.py populate_db

Async Background Tasks with Celery on Django

From this post and this post and this post

1. Install Redis - got from here
$ brew install redis
$ ln -sfv /usr/local/opt/redis/*.plist ~/Library/LaunchAgents
$ launchctl load ~/Library/LaunchAgents/homebrew.mxcl.redis.plist
$ redis-cli PING
PONG

2. Install celery with redis support bundle
$ pip3 install -U "celery[redis]"

3. Create celery.py in django project
$ vi sample_project/sample_project/celery.py
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'sample_project.settings')

app = Celery('sample_project')

# Using a string here means the worker don't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()


@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

4. Import celery app in __init__.py
from __future__ import absolute_import, unicode_literals

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app

__all__ = ['celery_app']

5. Add config in settings.py
$ vi sample_project/sample_project/settings.py
# celery
CELERY_BROKER_URL = 'redis://localhost:6379/0'

6. Create app
$ python3 manage.py startapp testapp
$ vi sample_project/sample_project/settings.py
INSTALLED_APPS = (
    # (...)
    'testapp',
)

7. Create task
$ vi sample_project/sample_project/testapp/tasks.py
from __future__ import absolute_import

from celery import shared_task

@shared_task
def test(param):
    return 'The test task executed with argument "%s" ' % param

8. Testing

Run server
$ python3 manage.py runserver

Run worker
$ celery -A sample_project worker --loglevel=info

From another terminal...
$ python3 manage.py shell
Python 3.6.2 (default, Jul 17 2017, 16:44:45)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from testapp.tasks import test
>>> test.delay('This is a test')


From the worker terminal
2017-07-26 00:50:54,529: INFO/MainProcess] Received task: testapp.tasks.test[4f1ab1d4-645d-480b-a1b5-6e2887e34b52]
[2017-07-26 00:50:54,533: INFO/ForkPoolWorker-2] Task testapp.tasks.test[4f1ab1d4-645d-480b-a1b5-6e2887e34b52] succeeded in 0.0010309600038453937s: 'The test task executed with argument "This is a test" '

9. For results backend using Django ORM

a. $ pip3 install django-celery-results

b. Install app in settings.py
INSTALLED_APPS = (
    ...,
    'django_celery_results',
)

c. Create celery database tables
$ python3 manage.py migrate django_celery_results

d. Configure Celery to use the django-celery-results backend.
In settings.py...
CELERY_RESULT_BACKEND = 'django-db'

Django with Postgres on Mac

1. Create virtual env following this post
2. Install django
(my_env) $ pip3 install psycopg2
(my_env) $ pip3 install django
3. Create django project
(my_env) $ django-admin startproject sample_project
(my_env) $ python3 manage.py runserver
4. Install postgres following this post
5. Create database
$ createdb -U postgres testdb
6. Modify settings in django
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'testdb',
        'USER': 'postgres',
        'PASSWORD': '',
        'HOST': 'localhost',
        'PORT': '5432',
    }
}
7. Migrate
$ python3 manage.py migrate
8. Create superuser
$ python3 manage.py createsuperuser

Mac Cheat Sheet

1. Install sshpass
$ brew install http://git.io/sshpass.rb

2. Using iTerm2, configure profiles and use sshpass so there's no need to enter passwords for ssh, follow this post for configuring iTerm2. What worked for me is in Command, select "Login shell", then send text at start:
sshpass -p 'password' ssh -o StrictHostKeyChecking=no user@server

Postgres Cheat Sheet

1. Postgres version
$ psql --version

2. Connect to server locally using default postgres acct
$ psql -U postgres

3. Connect with a password
$ psql -U postgres -W

4. Common commands
\h - help
\q - quit
\l - list databases
\d - list tables
\d table - describe table

5. Create db
postgres=# CREATE DATABASE testdb;

6. Use the db
postgres=# \c testdb;
testdb=#

7. List of users
postgres=# \du

8. List postgres versions running
$ pg_lsclusters
Ver Cluster Port Status Owner    Data directory               Log file
9.1 main    5433 down   postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.4 main    5432 down   postgres /var/lib/postgresql/9.4/main /var/log/postgresql/postgresql-9.4-main.log
9.6 main    5434 online postgres /var/lib/postgresql/9.6/main /var/log/postgresql/postgresql-9.6-main.log

9. Start/Stop/Restart/Status of postgresql
$ sudo systemctl start postgresql
$ sudo systemctl stop postgresql
$ sudo systemctl restart postgresql
$ sudo systemctl status postgresql

10. Backup/Restore
$ pg_dump -Fc -h hostname -p 5432 -d dbname -f /path/backupfilename
$ pg_restore -d dbname -v -1 /path/backupfilename

11. Cron daily backup
$ crontab -e
0 5 * * * export PGPASSWORD=MyPass && pg_dump -Fc -h hostname -p 5432 -d dbname -f /path/backupfilename_`date +20\%y\%m\%d` >> /path/pg_dump_dbname.log 2>&1

Install Postgres on Mac

Followed this post

1. Go to http://postgresapp.com/ then download and install
2. Add Postgres to PATH
$ vi ~/.bash_profile
export PATH=$PATH:/Applications/Postgres.app/Contents/Versions/latest/bin
$ source ~/.bash_profile
3. Start Postgres by double clicking the app, then on the top icon, start "Initialize"
4. Try connecting using default postgres role
$ psql -U postgres
psql (9.6.3)
Type "help" for help.

postgres=#

Install Python 3.6 on MacOS

Generally followed this post

I. Install brew
1. Install Xcode from the App Store
2. Install Xcode’s separate Command Line Tools app
$ xcode-select --install
3. Install and setup homebrew
$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
4. Make sure brew is in PATH
$ vi ~/.bash_profile
export PATH=/usr/local/bin:$PATH
5. Source it
$ source ~/.bash_profile
6. Try brew
$ brew doctor
Your system is ready to brew.

II. Install Python
$ brew search python
$ brew install python3
$ python3 --version
Python 3.6.2
To install a python package
$ pip3 install package_name
To upgrade python
$ brew update
$ brew upgrade python3

III. Creating Virtual environments
$ mkdir sample_project
$ cd sample_project
$ python3.6 -m venv my_env
$ source my_env/bin/activate

Change bash command prompt in Debian

Followed this post

$ vi /etc/bash.bashrc
Add this at the end:

# If id command returns zero, you have root access.
if [ $(id -u) -eq 0 ];
then # you are root, set red colour prompt
   PS1="\\[$(tput setaf 1)\\][\\u@\\h:\\w] (apps)# \\[$(tput sgr0)\\]"
else # normal
   PS1="[\\u@\\h:\\w] (apps)$ "
fi

Angular Cheat Sheet

Creating Components

import { Component } from '@angular/core'
@Component({
 selector: 'courses',
 template: '<h2>Courses</h2>'
})
export class Component { }

Then, register them in AppModule (or the module the belong to):

@NgModule({
 declarations: [CoursesComponent]
})
export class AppModule { … }

Using Components

import { CoursesComponent } from './courses.component'
@Component({
 template: '<courses></courses>'
})

Templates
Interpolation syntax:
{{ title }}

Displaying lists:
<ul>
 <li *ngFor=“let course of courses”>
 {{ course }}
 </li>
</ul>

Services

import { Injectable } from '@angular/core';
@Injectable()
export class CourseService {
}

Dependency Injection

Register a service as a provider in AppModule (or the module it belongs to):

@NgModule({
 providers: [CourseService]
})
export class AppModule { … }

Then, inject it into the constructor of the components that need it:

export class CourseComponent {
 constructor(courseService: CourseService) {}
}

Directives
Basic structure
import { Directive } from '@angular/core'

@Directive({
 selector: '[autoGrow]',
 host: {
 '(focus)': 'onFocus()',
 '(blur)': 'onBlur()'
 }
})
export class AutoGrowDirective {
 onFocus() { … }
 onBlur() { … }
}

To access and modify DOM elements

import { ElementRef, Renderer } from '@angular/core'

export class AutoGrowDirective {
 constructor(
 private el: ElementRef,
 private renderer: Renderer) {
 }
 onFocus(){
 this.renderer.setElementStyle(this.el.nativeElement,
 'width', '200');
 }
}

Registration

Once you implement a directive, you should register it in AppModule (or the module it belong to):

@NgModule({
 declarations: [AutoGrowDirective]
})
export class AppModule { }

Angular Bindings

Interpolation
<h1>{{ title }}</h1>

Property binding
<img [src]="imageUrl" />
<img bind-src="imageUrl" />

Class binding
<li [class.active]="isActive" />

Style binding
<button [style.backgroundColor]="isActive ? 'blue' : 'gray'">

Event binding
<button (click)="onClick($event)">
<button on-click="onClick($event)">

Two-way binding
<input type="text" [(ngModel)]="firstName">
<input type="text" bindon-ngModel="firstName">

Input Properties
Using @Input annotation
import { Input } from ‘@angular/core’;
@Component(…)
export class FavoriteComponent {
 @Input(‘is-favorite’) isFavorite = false;
}

Using component metadata
@Component({
 inputs: [‘isFavorite:is-favorite’]
})
export class FavoriteComponent {
 isFavorite = false;
}

In the host component
<favorite [is-favorite]=“post.isFavorite”></favorite>


Output Properties
Using @Output annotation
import { Output } from ‘@angular/core’;
@Component(…)
export class FavoriteComponent {
 @Output(‘favorite-change’) change = new EventEmitter();

 onClick() {
 this.change.emit({ newValue: this.isFavorite });
 }
}


Using component metadata
@Component({
 outputs: [‘change:favoriteChange’]
})
export class FavoriteComponent {
 change = new EventEmitter();

 onClick() {
 this.change.emit({ newValue: this.isFavorite });
 }
}


In the host component
<favorite (favoriteChange)=“onChange()”></favorite>


Templates
@Component({
 template: ‘…’, // or
 templateUrl: ‘app/template.template.html’
})


Styles
@Component({
 styles: [‘…’],
 styleUrls: [‘…’, ‘…’];
})

Getting Ready for Angular Development

On Ubuntu:
Install nodejs
1. $ sudo apt-get install nodejs
2. $ sudo ln -s /usr/bin/nodejs /usr/bin/node
3. $ sudo apt-get install npm

Install typescript
1. $ sudo npm install -g typescript
2. $ sudo npm install -g typings

Install angular cli
1. $ sudo npm install -g @angular/cli

On Debian:
$ curl -sL https://raw.githubusercontent.com/creationix/nvm/v0.32.0/install.sh -o install_nvm.sh
$ bash install_nvm.sh
$ source ~/.profile
$ nvm ls-remote
$ nvm install 8.5.0
$ nvm use 8.5.0
$ node -v
v8.5.0
$ npm -v
5.3.0
$ npm install -g yarn
$ yarn global add @angular/cli
$ vi ~/.profile
# yarn
export PATH="$PATH:`yarn global bin`"
$ ng set --global packageManager=yarn
$ ng --version
@angular/cli: 1.4.3
node: 8.5.0
os: linux x64

On Mac:
I followed this and this

Install nodejs
1. Install xcode
$ xcode-select --install
2. Install nvm
$ curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.2/install.sh | bash
3. Open another bash session to get settings from nvm install
4. List installed node versions
$ nvm list
5. Or list from the cloud (last 9 versions)
$ nvm ls-remote | tail -n9
6. Install latest
$ nvm install 8.2.1
7. Setup this version as default
$ nvm use 8.2.1
$ nvm alias default 8.2.1
8. Check node version
$ node -v
v8.2.1
9. Update npm
$ npm install -g npm
10. Check npm version
$ npm -v
5.3.0

Install yarn package manager
1. $ npm install -g yarn

Install angular cli
1. $ yarn global add @angular/cli@1.2.7
2. Check version
$ ng --version
@angular/cli: 1.2.7
node: 8.2.1
os: darwin x64
3. Globally config angular-cli to use yarn
$ ng set --global packageManager=yarn

Test scaffold first angular app
1. $ ng new hello-world-app
2. Start server
$ cd hello-world-app
$ ng serve
3. Browse http://localhost:4200

Setup IDE
1. Install Visual Studio Code