Vi Text Editor My Frequently Used Commands

h              move cursor one character to left
j              move cursor one line down
k              move cursor one line up
l              move cursor one character to right
w              move cursor one word to right
b              move cursor one word to left
0              move cursor to beginning of line
$              move cursor to end of line
nG             move cursor to line n
1G             move cursor to 1st line
G              move cursor to last line
control-f      scroll forward one screen
control-b      scroll backward one screen

gx             jump to method definition

zi             switch folding on or off
za             toggle current fold open/closed
zc             close current fold
zR             open all folds
zM             close all folds
zv             expand folds to reveal cursor

i              insert to left of current cursor position (end with ESC)
a              append to right of current cursor position (end with ESC)
dw             delete current word (end with ESC)
cw             change current word (end with ESC)
r              change current character
~              change case (upper-, lower-) of current character

dd             delete current line
D              delete portion of current line to right of the cursor
x              delete current character
dgg            delete current line to the top of the file
jdG            delete all lines below current line
ma             mark currrent position
d`a            delete everything from the marked position to here
`a             go back to the marked position
p              dump out at current place your last deletion (``paste'')

u              undo the last command 
.              repeat the last command 

J              combine (``join'') next line with this one

:w             write file to disk, stay in vi
:q!            quit VI, do not write file to disk,
ZZ             write file to disk, quit vi

:r filename    read in a copy of the specified file to the current
               buffer

/string        search forward for string (end with Enter)
?string        search backward for string (end with Enter)
n              repeat the last search (``next search'')
N              repeat the last search backward
/\cstring      case insensitive prepend with \c

:s/s1/s2       replace (``substitute'') (the first) s1 in this line by s2
:lr/s/s1/s2/g  replace all instances of s1 in the line range lr by s2
               (lr is of form 'a,b', where a and b are either explicit
               line numbers, or . (current line) or $ (last line)
:map k s       map the key k to a string of vi commands s (see below)
:abb s1 s2     expand the string s1 in append/insert mode to a string 
               s2 (see below)
%              go to the "mate," if one exists, of this parenthesis
               or brace or bracket (very useful for programmers!)

:%s/foo/bar/g  replace every 'foo' with a 'bar'
:!command      execute a command from vi
:%!perltidy    format perl code using perltidy from within vi
:%!perltidy -l=0    ignore line length 
:noh           remove search highlighting
:sp            to split window horizontally
:vsp           to split window vertically
<ctrl-w>h|j|k|l to navigate between split windows
<ctrl-w>q      to quit split window 

Comment/Uncomment:
Press Ctrl+V and go down until the last commented line and press x, that will delete all the # characters vertically.

For commenting a block of text is almost the same: First, go to the first line you want to comment, press Ctrl+V, and select until the last line. Second, press Shift+I, then #, then Esc.

Indent several lines at once:
Press Ctrl+V and go down until the last line to be indented, press Shift+>

Cut/Copy and Paste:
Position the cursor where you want to begin cutting.
Press v (or upper case V if you want to cut whole lines).
Move the cursor to the end of what you want to cut.
Press d to cut or y to copy.
Move to where you would like to paste.
Press P to paste before the cursor, or p to paste after.
Copy and paste can be performed with the same steps, only pressing y instead of d in step 4.
The name of the mark used is related to the operation (d:delete or y:yank).

Linux Cheat Sheet

1. Find Out Linux Distribution Name and Version:

$ cat /etc/∗-release
Red Hat Enterprise Linux Server release 5 (Tikanga)

2. Use scp (Cygwin) to transfer files:

>scp -r [user_name]@[host_name or host_ip]:[Path to your directory] [target_path_dir]
>scp -r root@server.domain.com:/tmp/example.zip .
>scp -r user@your.server.example.com:/path/to/foo /home/user/Desktop/

from local to remote server:

>scp file* root@server.domain.com:/tmp

3. Copy files from a linux server to another linux server

rsync -auv -e ssh --progress source_folder/ useraccount@machine.whatever.com:/destination_folder/

4. Find files using 'find'

#find files from current directory and subdirectories that matches mon*
$ find . -name mon\*

#find files from current directory and subdirectories displaying the full path
$ find `pwd` -name koala\*
#or
$ find `pwd` | grep koala

#find files older than 90 days and move to another directory
$ find /home/perf-builder/deploy -maxdepth 1 -mtime +90 -type f -exec mv "{}" /perflogs/deploy_package_archive/ \;

#find files that are writable by anyone
$ find -maxdepth 1 -type f -perm /222

#find files that are writable by their owners
$ find -maxdepth 1 -type f -perm /200

5. Delete environment variable

$ env | grep rvm
OLDPWD=/home/samdc/.rvm
$ unset OLDPWD

6. Set environment variable

$ export PATH=/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

7. Monitor log file as the file grows using tail

$ tail -f /home/sammydc/logs/request.log

8. Search a directory for files containing certain string

$ grep -r "192.168.1.5" /etc/
$ grep -E "whatever" -r .   - recursive on current dir and subdirs
# multi-line grep
$ grep -Ezl "FC_count\s+2.*autoselect\s+yes.*client_os_version_linux\s+CentOS 6.5.*fc_switch\s+ls-dur3-mds2.*motherboard_product_name\s+.*C240.*use_for_group\s+Perf" *.cfg
# grep from different directories from July 21 – July 28
grep 'caught exception: ERROR' /mnt/perfresults/*/logs/2016072[1-8]_*/controller_log

9. This will show process id, user, command and arguments:

$ ps -e -o "pid,user,comm,args"

10. Cat a bunch of files at once

$ for f in /var/log/*.log; do cat $f; done

11. Iterate thru a list and apply operation

Copy a file to multiple destination
$ for FILE in {544..579}; do cp perf-c440 perf-c$FILE; done

Ping several clients
$ for CLIENT in client1 client2 client3; do ping $CLIENT -c2; done

12. Filter using grep with regex

$ ps -efw | grep -E '5\.5.*dd670'

13. List directories up to 3 levels down, then filter

$ ls -d ./*/*/* | grep 416041

14. Get number of cpus

$ cat /proc/cpuinfo | grep processor | wc -l

15. Get memory info

$ cat /proc/meminfo

16. Get disks info

$ cat /proc/partitions
$ df -H

17. How to determine biggest directories/files occupying disk

$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda5             26011288  24703036         0 100% /
/dev/sda1               202219     63562    128217  34% /boot
$ df -BG  (in GigaBytes)
Filesystem           1G-blocks      Used Available Use% Mounted on
/dev/dd_dg0_0p15            2G        2G        1G  68% /
/dev/dd_dg0_0p14            5G        4G        2G  72% /ddr
/dev/dd_dg0_0p13           30G       30G        0G 100% /ddr/var

$ du -a /home | sort -n -r | head -n 10
14259824        /home
14259052        /home/delacs
14259048        /home/delacs/scheduler
14256188        /home/delacs/scheduler/ctl
9382796 /home/delacs/scheduler/ctl/run_request.log
3238172 /home/delacs/scheduler/ctl/resourcedb.log
1036148 /home/delacs/scheduler/ctl/requestdb.log
569516  /home/delacs/scheduler/ctl/update_resource_status.log
29492   /home/delacs/scheduler/ctl/www_index.log
2056    /home/delacs/scheduler/resources

$ du -h /home | sort -h -r | head -n 10 (human readable form)
$ du -ahx / | sort -h -r | head -n 50 (exclude mounted directories)

18. Which processes take so much... Memory

$ ps -e -o pid,vsz,command | sort -n -r -k 2 | head -n 20
22796 345320 /usr/sbin/httpd
22799 344772 /usr/sbin/httpd
22907 344392 /usr/sbin/httpd
27749 242936 run_request_id_20140614_0800_mweb_dur
26515 217668 run_request_id_20140725_1738_mweb_dur
20061 217476 run_request_id_20140724_1600_mweb_dur
27748 215428 run_request_id_20140614_0800_mweb_dur
...

CPU

$ ps -eo pcpu,pid,user,args | sort -k1 -r -n | head -20
 8.8  9341 ioperf   /usr/bin/python /home/ioperf/maestro-production/cloudapi/pcrun.py --noTty --logLevel DEBUG -s /tmp/tmpsap5Fn/cfg.json -w /tmp/tmpsap5Fn/cfg.json -r /tmp/tmpsap5Fn/cfg.json --scriptDir /home/ioperf/maestro-production --resultDirBase /data/maestro_prod/vsan-fio
 7.0 29457 ioperf   /usr/bin/python /home/ioperf/maestro-production/cloudapi/pcrun.py --noTty --logLevel DEBUG -s /tmp/tmpUvSJTh/cfg.json -w /tmp/tmpUvSJTh/cfg.json -r /tmp/tmpUvSJTh/cfg.json --scriptDir /home/ioperf/maestro-production --resultDirBase /data/maestro_prod/vsan-fio
 0.4  8817 root     /usr/bin/etserver --daemon --cfgfile=/etc/et.cfg
...

19. Find files and delete them

The basic find command syntax is:
find dir-name criteria action
dir-name : - Defines the working directory such as look into /tmp/
criteria : Use to select files such as "*.sh"
action : The find action (what-to-do on file) such as delete the file.

$ find . -name ".nfs000000000006a8b200000062" -exec rm -rf {} \;

Options:
-name "FILE-TO-FIND" : File pattern.
-exec rm -rf {} \; : Delete all files matched by file pattern.

20. Echo string with a tab and append to end of a file

$ echo -e "autoselect\tyes" >> perf-c576

21. Count number of directories

$ echo 201404*/ | wc

22. Prepend contents of a file to another file

echo -e "`cat list.idx.forcopy`\n$(cat list.idx)" > list.idx

23. Script execution time

$ time ./monitor_regression.pl
...
Output
...
real    0m57.766s
user    0m0.444s
sys     0m0.805s

24. NFS mount

# mount -t nfs -o rw,nosuid,nodev,tcp,intr,nolock perfresultsSC:/backup /mnt/perfresultsSC/

25. Time sync

# ntpdate time-server-name-or-ip

26. Start process in background

$ ./my_process >std.txt 2>err.txt &
$ ./my_process >std.txt 2>&1 & (both stdout and stderr goes to the same file)
$ ./my_process >/dev/null 2>err.txt & (don't care about output, but stderr is recorded)

27. Monitor CPU & Memory utilization

Get CPU utilization every 2 minutes and run forever
$ mpstat -P ALL 120 > mpstat.log &
Get memory utilization every 2 minutes in 2 weeks
$ sar -r 120 10080 > sar.log &

27. Put currently running process in the background

$ Ctrl+Z
$ bg
# be able to exit out of ssh connection without affecting the running process
$ disown -h

27. Memory usage of a process, units in MB

$ smem -P server.rb
  PID User     Command         Swap      USS      PSS      RSS 
 2176 samdc    ruby server.rb     0   240052   241281   246296 

28. Create a tarball

$ tar -cvf openssh.deploy.tar file1 file2
$ tar -zcvf archive-name.tar.gz directory-name
$ tar -zcvf archive-name.tar.gz directory-name --remove-files (to remove original directory and files)

29. View tarball contents

$ tar -tvf openssh.deploy.tar

30. Extract tarball

$ tar -xvf openssh.deploy.tar

31. Kill processes at once

$ ps -efw | grep rvc | grep '1  ' | awk '{print $2}' | xargs -I{} kill -9 {}
Safer...
$ ps -ewf | grep rvc | grep 10.92.81.163 | awk '{print $2}' | xargs -I{} kill -9 {}

30. Rename files in directory at once

$ for file in *.log; do mv "$file" "$file.old"; done

31. How to remove all empty directories in a subtree

$ find ROOTDIR -type d -empty -delete
e.g.
$ find /mnt/maestro/maestro_prod/vsan-fio/15278 -type d -empty -delete

32. Find out how much space files occupy in a directory

$ du -bch

Ruby Gems Issues

Just a list of issues that I want to take note of while using ruby gems.

Encountered this problem when I tried to install Sinatra:

>gem install sinatra
ERROR:Could not find a valid gem 'sinatra' (>= 0), here is why: Unable to download data from https://rubygems.org/ - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (https://s3.amazonaws.com/production.s3.rubygems.org/latest_specs.4.8.gz)

Solution is to specify the source explicitly and use http instead of https:

>gem install sinatra --source http://rubygems.org/

MySQL Cheat Sheet

1. Connect to MySQL server in localhost

>mysql -u username -p password [databasename]

2. Executing SQL Statements from a Text File

mysql> source c:\filename.sql

3. Migrate database

# everything
$ mysqldump -u [user] -p --opt mydb > mydb.sql

# To export to file (data only)
mysqldump -u [user] -p --no-create-info mydb > mydb.sql

# To export to file (structure only)
mysqldump -u [user] -p --no-data mydb > mydb.sql

# To import to database
mysql -u [user] -p mydb < mydb.sql

PowerShell Script that removes VSS bindings and files from a Visual Studio Project

Below is a powershell script I found from the web that I have been using for a while to remove the bindings from SourceSafe.

First, unbind the project from VSS within Visual Studio, then run this script.

# This script removes VSS bindings and files from a VS Project
# Taken from http://blog.magenic.com/blogs/daniels/archive/2008/11/18/Removing-VSS-Bindings-to-Migrate-SSRS-2005-Solutions-to-TFS-Using-Powershell.aspx

# First thing to do before running this script is to specify where the project is located
$Folder = "C:\dev\vs\2008\SalesSync"
#first clear the read-only flag from all files
get-childitem "$folder" -Recurse | % {         
        # Test for ReadOnly flag and remove if present 
     if ($_.attributes -band [system.IO.FileAttributes]::ReadOnly) {  
            $_.attributes = $_.attributes -bxor [system.IO.FileAttributes]::ReadOnly 
   }
}

#next delete all files that are *.suo, *.user, and *.*scc - we don't want them in TFS
Get-ChildItem  $folder *.suo -Recurse -Force | Remove-Item -Force
Get-ChildItem  $folder *.*scc -Recurse -Force | Remove-Item -Force
Get-ChildItem  $folder *.user -Recurse -Force | Remove-Item -Force

#next get all the .sln file - and remove the VSS binding information
$files = Get-ChildItem $folder *.sln -Recurse 
foreach ($file in $files) {
 $fileout = $file.FullName + ".new"
 Set-Content $fileout $null
 $switch=0
 Get-Content $file.FullName | % {
  if ($switch -eq 0) {
   if ($_  -eq " GlobalSection(SourceCodeControl) = preSolution") {
    #we found the section to skip - so set the flag and don't copy the content
    $switch=1}
   else {
    #we haven't found it yet - so copy the content
    Add-Content $fileout $_
   }        
   }
  elseif ($switch -eq 1) {
   if ($_ -eq " EndGlobalSection") {
    #last line to skip - after it we start writing the content again
    $switch=2}
   }
  else 
   { #write remaining lines
    Add-Content $fileout $_}
 }
 #remove the original .sln and rename the new one
 $newname = $file.Name
 Remove-Item $file.FullName
 Rename-Item $fileout -NewName $newname
}

Client-Side Validation for a Custom DataAnnonation in MVC 4

This is a continuation of my previous post. This is based from this excellent tutorial.

First step is to implement IClientValidatable interface in the FileTypesAttribute class:

public class FileTypesAttribute : ValidationAttribute, IClientValidatable {
        private readonly List _types;
        private readonly string specifiedTypes;

        public FileTypesAttribute(string types) {
            _types = types.Split(',').ToList();
            specifiedTypes = types;
        }

        public override bool IsValid(object value) {
            if (value == null) return true;
            var postedFile = value as HttpPostedFileBase;
            var fileExt = System.IO.Path.GetExtension(postedFile.FileName).Substring(1);
            return _types.Contains(fileExt, StringComparer.OrdinalIgnoreCase);
        }

        public override string FormatErrorMessage(string name) {
            return string.Format("Invalid file type. Only {0} are supported.", String.Join(", ", _types));
        }

        public IEnumerable GetClientValidationRules(ModelMetadata metadata, ControllerContext context) {
            var rule = new ModelClientValidationRule();
            rule.ErrorMessage = FormatErrorMessage(metadata.GetDisplayName());
            rule.ValidationType = "filetypes";
            rule.ValidationParameters.Add("types", specifiedTypes);
            yield return rule;
        }
    }

Things to note here are the validationtype="filetypes" and the added parameter="types", which can be observed in the emitted html below:

<input data-val="true" data-val-filetypes="Invalid file type. Only xls, xlsx are supported."
data-val-filetypes-types="xls,xlsx" data-val-required="The AttachmentTrace field is required."
id="AttachmentTrace" name="AttachmentTrace" type="file" value="" />

The added parameter called types has a value of "xls,xlsx". This parameter and validationtype are significant because it will be used for the jquery validator and adapter as follows:

$(function () {
    // custom validation method
    $.validator.addMethod("filetypes",
        function (value, element, params) {
            if (value == null) return true;
            var fileExtension = value.split('.').pop().toLowerCase();
            var types = params.toLowerCase();
            return types.indexOf(fileExtension) !== -1;
        }
    );
    // add to unobtrusive adapters
    $.validator.unobtrusive.adapters.addSingleVal("filetypes", "types");
} (jQuery));

The value has the contents of the input element, while params has the parameter value of "xls,xlsx" from the data annotation.

ModelState.IsValid always returning False for RegularExpression ValidationAttribute for a File Upload in MVC 4

I posted a question in stackoverflow and ended up answering it on my own. Some people gave me clues along the way so I'm thankful. I just want to do a recap of the solution that way I have it in my notes...

I was having issues with model validation using RegularExpression as indicated in my class below, the property AttachmentTrace is a file attachment that is uploaded during form post.

public class Certificate {
    [Required]
    // TODO:  Wow looks like there's a problem with using regex in MVC 4, this does not work!
    [RegularExpression(@"^.*\.(xlsx|xls|XLSX|XLS)$", ErrorMessage = "Only Excel files (*.xls, *.xlsx) files are accepted")]
    public string AttachmentTrace { get; set; }
}

All the while I thought there's something wrong with my Regex. But after looking closely on this issue, I found out that what is being validated on the server side is the string "System.Web.HttpPostedFileWrapper", and not the actual filename. That's the reason why ModelState.IsValid returns a false everytime, no matter how right the regex is. But I cannot simply switch the property type of string to HttpPostedFileBase in my model, because I'm using EF code-first migration, which will result into an unpleasant error message when adding a migration.

So the solution to this is to employ a ViewModel, instead of the Entity directly:

public class CertificateViewModel {
    // .. other properties
    [Required]
    [FileTypes("xls,xlsx")]
    public HttpPostedFileBase AttachmentTrace { get; set; }
}

The next step is to create a custom ValidationAttribute for the FileTypes:

public class FileTypesAttribute : ValidationAttribute {
    private readonly List _types;

    public FileTypesAttribute(string types) {
        _types = types.Split(',').ToList();
    }

    public override bool IsValid(object value) {
        if (value == null) return true;
        var postedFile = value as HttpPostedFileBase;
        var fileExt = System.IO.Path.GetExtension(postedFile.FileName).Substring(1);
        return _types.Contains(fileExt, StringComparer.OrdinalIgnoreCase);
    }

    public override string FormatErrorMessage(string name) {
        return string.Format("Invalid file type. Only {0} are supported.", String.Join(", ", _types));
    }
}

In the controller Action, use the ViewModel instead of the Entity, then map the ViewModel back to the Entity via AutoMapper:

public ActionResult Create(CertificateViewModel certificate, HttpPostedFileBase attachmentTrace, HttpPostedFileBase attachmentEmail) {
        if (ModelState.IsValid) {
            // Let's use AutoMapper to map the ViewModel back to our Certificate Entity
            // We also need to create a converter for type HttpPostedFileBase -> string
            Mapper.CreateMap().ConvertUsing(new HttpPostedFileBaseTypeConverter());
            Mapper.CreateMap();
            Certificate myCert = Mapper.Map(certificate);
            // other code ...
        }
        return View(myCert);
    }

For the AutoMapper, create a TypeConverter for HttpPostedFileBase:

public class HttpPostedFileBaseTypeConverter : ITypeConverter {

    public string Convert(ResolutionContext context) {
        var fileBase = context.SourceValue as HttpPostedFileBase;
        if (fileBase != null) {
            return fileBase.FileName;
        }
        return null;
    }
}

I know, it's a lot of work, but this will ensure the file extension validation will be done correctly. But wait, this is not yet complete, I need to also make sure that the client side validation will work for this and that will be the topic of my next post.