9.24.2013

Bash Keyboard Shortcuts

Ctrl + A        Go to the beginning of the line you are currently typing on
Ctrl + E        Go to the end of the line you are currently typing on
Ctrl + L        Clears the Screen, similar to the clear command
Ctrl + U        Clears the line before the cursor position. If you are at the end of the line, clears the entire line.
Ctrl + H        Same as backspace
Ctrl + R        Let’s you search through previously used commands
Ctrl + C        Kill whatever you are running
Ctrl + D        Exit the current shell
Ctrl + Z        Puts whatever you are running into a suspended background process. fg restores it.
Ctrl + W        Delete the word before the cursor
Ctrl + K        Clear the line after the cursor
Ctrl + T        Swap the last two characters before the cursor
Esc + T         Swap the last two words before the cursor
Alt + F         Move cursor forward one word on the current line
Alt + B         Move cursor backward one word on the current line
Tab             Auto-complete files and folder names

9.19.2013

Vi Text Editor My Frequently Used Commands

h              move cursor one character to left
j              move cursor one line down
k              move cursor one line up
l              move cursor one character to right
w              move cursor one word to right
b              move cursor one word to left
0              move cursor to beginning of line
$              move cursor to end of line
nG             move cursor to line n
1G             move cursor to 1st line
G              move cursor to last line
control-f      scroll forward one screen
control-b      scroll backward one screen

gx             jump to method definition

zi             switch folding on or off
za             toggle current fold open/closed
zc             close current fold
zR             open all folds
zM             close all folds
zv             expand folds to reveal cursor

i              insert to left of current cursor position (end with ESC)
a              append to right of current cursor position (end with ESC)
dw             delete current word (end with ESC)
cw             change current word (end with ESC)
r              change current character
~              change case (upper-, lower-) of current character

dd             delete current line
D              delete portion of current line to right of the cursor
x              delete current character
dgg            delete current line to the top of the file
jdG            delete all lines below current line
ma             mark currrent position
d`a            delete everything from the marked position to here
`a             go back to the marked position
p              dump out at current place your last deletion (``paste'')

u              undo the last command 
.              repeat the last command 

J              combine (``join'') next line with this one

:w             write file to disk, stay in vi
:q!            quit VI, do not write file to disk,
ZZ             write file to disk, quit vi

:r filename    read in a copy of the specified file to the current
               buffer

/string        search forward for string (end with Enter)
?string        search backward for string (end with Enter)
n              repeat the last search (``next search'')
N              repeat the last search backward
/\cstring      case insensitive prepend with \c

:s/s1/s2       replace (``substitute'') (the first) s1 in this line by s2
:lr/s/s1/s2/g  replace all instances of s1 in the line range lr by s2
               (lr is of form 'a,b', where a and b are either explicit
               line numbers, or . (current line) or $ (last line)
:map k s       map the key k to a string of vi commands s (see below)
:abb s1 s2     expand the string s1 in append/insert mode to a string 
               s2 (see below)
%              go to the "mate," if one exists, of this parenthesis
               or brace or bracket (very useful for programmers!)

:%s/foo/bar/g  replace every 'foo' with a 'bar'
:!command      execute a command from vi
:%!perltidy    format perl code using perltidy from within vi
:%!perltidy -l=0    ignore line length 
:noh           remove search highlighting
:sp            to split window horizontally
:vsp           to split window vertically
<ctrl-w>h|j|k|l to navigate between split windows
<ctrl-w>q      to quit split window 

Comment/Uncomment:
Press Ctrl+V and go down until the last commented line and press x, that will delete all the # characters vertically.

For commenting a block of text is almost the same: First, go to the first line you want to comment, press Ctrl+V, and select until the last line. Second, press Shift+I, then #, then Esc.

Indent several lines at once:
Press Ctrl+V and go down until the last line to be indented, press Shift+>

Cut/Copy and Paste:
Position the cursor where you want to begin cutting.
Press v (or upper case V if you want to cut whole lines).
Move the cursor to the end of what you want to cut.
Press d to cut or y to copy.
Move to where you would like to paste.
Press P to paste before the cursor, or p to paste after.
Copy and paste can be performed with the same steps, only pressing y instead of d in step 4.
The name of the mark used is related to the operation (d:delete or y:yank).

9.17.2013

Linux Cheat Sheet

1. Find Out Linux Distribution Name and Version:
$ cat /etc/∗-release
Red Hat Enterprise Linux Server release 5 (Tikanga)
2. Use scp (Cygwin) to transfer files:
>scp -r [user_name]@[host_name or host_ip]:[Path to your directory] [target_path_dir]
>scp -r root@server.domain.com:/tmp/example.zip .
>scp -r user@your.server.example.com:/path/to/foo /home/user/Desktop/
from local to remote server:
>scp file* root@server.domain.com:/tmp
3. Copy files from a linux server to another linux server
rsync -auv -e ssh --progress source_folder/ useraccount@machine.whatever.com:/destination_folder/
4. Find files using 'find'
#find files from current directory and subdirectories that matches mon*
$ find . -name mon\*

#find files from current directory and subdirectories displaying the full path
$ find `pwd` -name koala\*
#or
$ find `pwd` | grep koala

#find files older than 90 days and move to another directory
$ find /home/perf-builder/deploy -maxdepth 1 -mtime +90 -type f -exec mv "{}" /perflogs/deploy_package_archive/ \;

#find files that are writable by anyone
$ find -maxdepth 1 -type f -perm /222

#find files that are writable by their owners
$ find -maxdepth 1 -type f -perm /200
5. Delete environment variable
$ env | grep rvm
OLDPWD=/home/samdc/.rvm
$ unset OLDPWD
6. Set environment variable
$ export PATH=/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
7. Monitor log file as the file grows using tail
$ tail -f /home/sammydc/logs/request.log
8. Search a directory for files containing certain string
$ grep -r "192.168.1.5" /etc/
$ grep -E "whatever" -r .   - recursive on current dir and subdirs
# multi-line grep
$ grep -Ezl "FC_count\s+2.*autoselect\s+yes.*client_os_version_linux\s+CentOS 6.5.*fc_switch\s+ls-dur3-mds2.*motherboard_product_name\s+.*C240.*use_for_group\s+Perf" *.cfg
# grep from different directories from July 21 – July 28
grep 'caught exception: ERROR' /mnt/perfresults/*/logs/2016072[1-8]_*/controller_log
9. This will show process id, user, command and arguments:
$ ps -e -o "pid,user,comm,args"
10. Cat a bunch of files at once
$ for f in /var/log/*.log; do cat $f; done
11. Iterate thru a list and apply operation
Copy a file to multiple destination
$ for FILE in {544..579}; do cp perf-c440 perf-c$FILE; done

Ping several clients
$ for CLIENT in client1 client2 client3; do ping $CLIENT -c2; done
12. Filter using grep with regex
$ ps -efw | grep -E '5\.5.*dd670'
13. List directories up to 3 levels down, then filter
$ ls -d ./*/*/* | grep 416041
14. Get number of cpus
$ cat /proc/cpuinfo | grep processor | wc -l
15. Get memory info
$ cat /proc/meminfo
16. Get disks info
$ cat /proc/partitions
$ df -H
17. How to determine biggest directories/files occupying disk
$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda5             26011288  24703036         0 100% /
/dev/sda1               202219     63562    128217  34% /boot
$ df -BG  (in GigaBytes)
Filesystem           1G-blocks      Used Available Use% Mounted on
/dev/dd_dg0_0p15            2G        2G        1G  68% /
/dev/dd_dg0_0p14            5G        4G        2G  72% /ddr
/dev/dd_dg0_0p13           30G       30G        0G 100% /ddr/var

$ du -a /home | sort -n -r | head -n 10
14259824        /home
14259052        /home/delacs
14259048        /home/delacs/scheduler
14256188        /home/delacs/scheduler/ctl
9382796 /home/delacs/scheduler/ctl/run_request.log
3238172 /home/delacs/scheduler/ctl/resourcedb.log
1036148 /home/delacs/scheduler/ctl/requestdb.log
569516  /home/delacs/scheduler/ctl/update_resource_status.log
29492   /home/delacs/scheduler/ctl/www_index.log
2056    /home/delacs/scheduler/resources

$ du -h /home | sort -h -r | head -n 10 (human readable form)
$ du -ahx / | sort -h -r | head -n 50 (exclude mounted directories)
18. Which processes take so much... Memory
$ ps -e -o pid,vsz,command | sort -n -r -k 2 | head -n 20
22796 345320 /usr/sbin/httpd
22799 344772 /usr/sbin/httpd
22907 344392 /usr/sbin/httpd
27749 242936 run_request_id_20140614_0800_mweb_dur
26515 217668 run_request_id_20140725_1738_mweb_dur
20061 217476 run_request_id_20140724_1600_mweb_dur
27748 215428 run_request_id_20140614_0800_mweb_dur
...
CPU
$ ps -eo pcpu,pid,user,args | sort -k1 -r -n | head -20
 8.8  9341 ioperf   /usr/bin/python /home/ioperf/maestro-production/cloudapi/pcrun.py --noTty --logLevel DEBUG -s /tmp/tmpsap5Fn/cfg.json -w /tmp/tmpsap5Fn/cfg.json -r /tmp/tmpsap5Fn/cfg.json --scriptDir /home/ioperf/maestro-production --resultDirBase /data/maestro_prod/vsan-fio
 7.0 29457 ioperf   /usr/bin/python /home/ioperf/maestro-production/cloudapi/pcrun.py --noTty --logLevel DEBUG -s /tmp/tmpUvSJTh/cfg.json -w /tmp/tmpUvSJTh/cfg.json -r /tmp/tmpUvSJTh/cfg.json --scriptDir /home/ioperf/maestro-production --resultDirBase /data/maestro_prod/vsan-fio
 0.4  8817 root     /usr/bin/etserver --daemon --cfgfile=/etc/et.cfg
...
19. Find files and delete them
The basic find command syntax is:
find dir-name criteria action
dir-name : - Defines the working directory such as look into /tmp/
criteria : Use to select files such as "*.sh"
action : The find action (what-to-do on file) such as delete the file.

$ find . -name ".nfs000000000006a8b200000062" -exec rm -rf {} \;

Options:
-name "FILE-TO-FIND" : File pattern.
-exec rm -rf {} \; : Delete all files matched by file pattern.
20. Echo string with a tab and append to end of a file
$ echo -e "autoselect\tyes" >> perf-c576
21. Count number of directories
$ echo 201404*/ | wc
22. Prepend contents of a file to another file
echo -e "`cat list.idx.forcopy`\n$(cat list.idx)" > list.idx
23. Script execution time
$ time ./monitor_regression.pl
...
Output
...
real    0m57.766s
user    0m0.444s
sys     0m0.805s
24. NFS mount
# mount -t nfs -o rw,nosuid,nodev,tcp,intr,nolock perfresultsSC:/backup /mnt/perfresultsSC/
25. Time sync
# ntpdate time-server-name-or-ip
26. Start process in background
$ ./my_process >std.txt 2>err.txt &
$ ./my_process >std.txt 2>&1 & (both stdout and stderr goes to the same file)
$ ./my_process >/dev/null 2>err.txt & (don't care about output, but stderr is recorded)
27. Monitor CPU & Memory utilization
Get CPU utilization every 2 minutes and run forever
$ mpstat -P ALL 120 > mpstat.log &
Get memory utilization every 2 minutes in 2 weeks
$ sar -r 120 10080 > sar.log &
27. Put currently running process in the background
$ Ctrl+Z
$ bg
# be able to exit out of ssh connection without affecting the running process
$ disown -h
27. Memory usage of a process, units in MB
$ smem -P server.rb
  PID User     Command         Swap      USS      PSS      RSS 
 2176 samdc    ruby server.rb     0   240052   241281   246296 
28. Create a tarball
$ tar -cvf openssh.deploy.tar file1 file2
$ tar -zcvf archive-name.tar.gz directory-name
$ tar -zcvf archive-name.tar.gz directory-name --remove-files (to remove original directory and files)
29. View tarball contents
$ tar -tvf openssh.deploy.tar
30. Extract tarball
$ tar -xvf openssh.deploy.tar
31. Kill processes at once
$ ps -efw | grep rvc | grep '1  ' | awk '{print $2}' | xargs -I{} kill -9 {}
Safer...
$ ps -ewf | grep rvc | grep 10.92.81.163 | awk '{print $2}' | xargs -I{} kill -9 {}
30. Rename files in directory at once
$ for file in *.log; do mv "$file" "$file.old"; done
31. How to remove all empty directories in a subtree
$ find ROOTDIR -type d -empty -delete
e.g.
$ find /mnt/maestro/maestro_prod/vsan-fio/15278 -type d -empty -delete
32. Find out how much space files occupy in a directory
$ du -bch

9.13.2013

Ruby Gems Issues

Just a list of issues that I want to take note of while using ruby gems.

Encountered this problem when I tried to install Sinatra:
>gem install sinatra
ERROR:Could not find a valid gem 'sinatra' (>= 0), here is why: Unable to download data from https://rubygems.org/ - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (https://s3.amazonaws.com/production.s3.rubygems.org/latest_specs.4.8.gz)
Solution is to specify the source explicitly and use http instead of https:
>gem install sinatra --source http://rubygems.org/

9.12.2013

MySQL Cheat Sheet

1. Connect to MySQL server in localhost
>mysql -u username -p password [databasename]
2. Executing SQL Statements from a Text File
mysql> source c:\filename.sql
3. Migrate database
# everything
$ mysqldump -u [user] -p --opt mydb > mydb.sql

# To export to file (data only)
mysqldump -u [user] -p --no-create-info mydb > mydb.sql

# To export to file (structure only)
mysqldump -u [user] -p --no-data mydb > mydb.sql

# To import to database
mysql -u [user] -p mydb < mydb.sql