vickistan.com

The du command is very useful when you are looking at disk space usage. Here are some of the most commonly used parameters:

-h human-readable Displays sizes in human-readable format, using units such as KB, MB, GB, etc
-a, –all Lists all files with their sizes (combine with h to get the best result)
-d, –max-depth=N
–time Lists all files with their sizes and the timestamp of its last modification
-X, –exclude=<pattern> Lists all files except those that match <pattern>

To find the disk usage of all files in the directory except the tarred files:

du -ah --exclude=".tgz" --exclude=".tar.gz"

To find the 10 largest files in the current directory ordered by file size:
(This gets both directories and files by size)
for i in G M K; do du -ah | grep [0-9]$i | sort -nr -k 1; done | head -n 11
or
du -ah . | sort -n -r | head -n 10

(This gets only files ordered by size)
du -hsx * | sort -rh | head -10
or
du -ah --max-depth=1 | sort -h

#Run iostat to get the average cpu usage every 10 seconds for 10 times:
iostat -x 10 10 >> /home/<user>/iostat.out.$(date +"%Y-%m-%d")

#Run dstat to get cpu stats for cpus 1, 3 and the total:
dstat -C 0,3,total

top and htop are free CLI process viewers. They list the top resource using processes being run.The information includes:

• PID — Process Id The task’s unique process ID
• PPID — Parent Process PID (Process ID) of a task’s parent
• RUSER — Real User NameThe real user name of the task’s owner 
• UID — User IdThe effective user ID of the task’s owner
• USER — User NameThe effective user name of the task’s owner
• GROUP — Group NameThe effective group name of the task’s owner.
• NI — Nice value The nice value of the task. A negative nice value means higher priority, whereas a positive nice value means lower priority. Zero in this field simply means priority will not be adjusted.

top options:
–a : Sort by memory usage This switch makes top to sort the processes by allocated memory
–d : Delay time update interval as: -d ss.tt (seconds.tenths)
–p : Monitor PIDs as: -pN1 -pN2 … or -pN1, N2 [,…]
–u : <somebody> Monitor only processes with an effective UID or user name matching <somebody>.
htop options:
-u –user=USERNAME Show only the processes of a given user
-d –delay=DELAY Delay between updates, in tenths of seconds

Examples:

To run top in batch mode (updating every 3 seconds) and write it to a file in my directory
top -b -d 10 -n 3 >> /home/<user>/top-file

To run htop to get only processes for my user:
htop -u <user>

Watch disk usage updating every 3 seconds:
watch -d -n 3 'df -h'

Sed strings make VIM very powerful. You can use them to do many things. The changes will not be written to the file until you save the file, so you are safe to experiment a bit.

To delete lines 4-12:

:4,12 d

To delete the current line:
:d

To replace the word foo with the word bar starting at the line where the cursor is through the end of the file:

:.,$s/foo/bar/

To add the word foo at the end of lines 24 – 34:

:24,34 s/$/foo/

To remove the spaces in the middle of a pip list output line and replace it with == so it can be used to install (if you have to recreate your pyenv):

1.) remove the header lines:
:1,2 d
2.) replace the spaces with double equals:
:%s/\s+/==/g

To remove == and everything after from a pip list so that pip will install the newest version:

:%s/==.*//

To remove all lines that contain the strings error, warn, or fail (remove the /d to show the lines that the command will delete):

:g/error\|warn\|fail/d

To remove all lines that don’t contain the strings error, warn, or fail (remove the /d to show the lines that the command will delete):

:g!/error\|warn\|fail/d

v can replace the g! if you prefer:

:v/error\|warn\|fail/d

To reformat a paragraph in vim:

1. Use <CTRL-J> to join all lines in the paragraph
2. :gq

To remove all commented and blank lines from a file (remove the /d to show the lines that the command will delete):

:g/\v^(#|$)/d

To change the last character in a line to a semi-colon:

:%s/.{1}$/\;/

To replace all single quotes in a file with double quotes:

:%s/'/\"/g

To remove all extra spaces in all lines of a file:

:%s/ +//g

Use g to execute a cmd on regex to delete blank lines:

:g/^$/d

Substitute starting string to eol with newstring followed by comma:

:% s/<string>.*/<newstring>,/

Substitute invalid non-printable characters (U+00A0) with spaces:

%s/\%u00a0/ /g

Remove everything on line from the comma to eol:
:%s/,.*//

Remove all lines not beginning with IPv4 IP:
:%v/^\(\d\+\.\)\{3}\d\+/d

Tar and encrypt:
tar cz <dir>/ | openssl enc -aes-256-cbc -pbkdf2 -iter 10000 -e > out.tar.gz.enc

Decrypt:
openssl enc -aes-256-cbc -d -in out.tar.gz.enc | tar xz

Since RedHat/Ubuntu/Debian’s /proc/cpuinfo has a separate entry for each CPU core, you can use this command to count them:
cat /proc/cpuinfo | grep processor | wc -l

Or do it with the following Python script:
—-
!/usr/bin/env python3

import os
import psutil

l1, l2, l3 = psutil.getloadavg()
CPU_use = (l3/os.cpu_count()) * 100

print(“CPU_use as computed by psutil.getloadavg: ” + str(CPU_use))

print(“cpu percent reported by psutil: ” + str(psutil.cpu_percent()))
print(“cpu count reported by psutil: ” + str(psutil.cpu_count()))
print(“cpu count reported by psutil with logical set to False: ” + str(psutil.cpu_count(logical=False)))

current_process = psutil.Process()
print(“cpu percent reported by psutil.Process ” + str(current_process.cpu_percent()))
—-

To convert a UTF-8 encoded file to UTF-16, you can use iconv on the command line:

iconv -f utf-8 -t utf-16 oldfile > newfile

To connect to MySQL through a tunnel

Open a tunnel on your local machine listening on localhost:3307 and forwarding everything to the mysqlserver server on port 3306, and doing it all via the ssh service on the gateway machine.

ssh -L 3307:domain.name.of.mysqlserver:3306 username@domain.name.of.gatewayserver

Now start mysql connecting to localhost on the port that you are tunneling mysql from the mysql server.

mysql -u username -p -h 127.0.0.1 -P 3307 databasename

mysql assumes it’s connecting to localhost, but on a non-standard port. In fact, the connection is being made securely to the remote mysql server, via the gateway machine and the local “mouth” of the ssh tunnel on your own machine.

—————————————————

tunnel all outbound E-mail traffic back to my personal server to avoid having to change SMTP servers when I am behind firewalls.

ssh -f user@personal-server.com -L 2000:personal-server.com:25 -N

—————————————————-

to send my Google Talk traffic encrypted through the firewall back to my server at home and then out to Google.

ssh -f -L 3000:talk.google.com:5222 home -N

—————————————————