Linux Shell 脚本菜谱 | Linux Shell Scripting Cookbook

Notice: 那时我第一次用 Markdown。(The Old Golden Days)

Chap 1: Shell Something Out

Bash (Bourne Again Shell)

Chap 2: Have a Good Command

Concatenating with Cat
# cat somefile.list or `someoutput | cat`
cat a.txt b.txt
echo "hello" | cat
# Getting rid of blank lines
cat -s file.txt
# more options:
#     -T(Display Tab as ^T);
#     -n(line numbers);
#     -b(line numbers, exculding blank lines)
Recording and Playing back of terminal session
# start recording
script -t 2> timing.log -a output.session
# do whatever you want here
pwd; echo "hello, scripting recording"; dir -al;
# exit recording
# yesterday once more!
scriptreplay timing.log output.session
Finding files and file listing
Playing with Xargs
Translating with tr

tr [option] set1 set2

mapping from set1 to set2 (with specified options), notice the len(set1) && len(set2) issue

echo "WHO IS THIS" | tr 'A-Z' 'a-z' # other valid sets: 'a-fg-z', 'a-n0-9', 'aA'...

tr '0-9' '987654321' # encription
tr '987654321' '0-9' # decryption

# the famous ROT13 Encryption.
# symmetric, use it the second time to decrypt
tr 'a-zA-Z' 'n-za-mN-ZA-M'

# Deleting chars
tr -d '[set1]'

# more options:
#      Complementing character set: -c
#      Squeezing Characters with tr: tr -s ' '
# Character Classes, e.g. '[:alnum:]':
#      1. alnum, alpha, digit
#      2. cntrl: control (nonprinting) chars
#      3. graph (graphic), print (printable)
#      4. lower, upper, space (whitespace chars)
#      5. punct, xdigit (hexadecimal chars)
Just for fun from dvorak4tzx
for x in {30..37};
    for y in {40..47};
        echo -e -n "\e[1;${x}m \e[1;${y}m bash ";
    echo -e "\e[1;30m";
colorful bash

colorful bash

Checksum and Verification

Crptographic tools and Hashes

Sorting Unique and Duplicates
# or `sort files > output.txt`
sort files -o output.txt

# more options:
#    -n: numeric sort, -M: sort Months, -r: Reverse
#    -m: merge two sorted file: `sort -m sorted1.txt sorted2.txt`
#    -k NUM: sort by colomun NUM(1..INF)
sort file1.txt file2.txt | unique
# `unique` options:
#   -u(dups will not print out),
#   -c(count),
#   -d(only dups),
#   -s, -w(#pass, #compare)
Temporary file naming and random numbers
dirname=`mktemp -d`

# generate a name but not create it now
mktemp -u

# with file name prefixed with "test" and
# suffixed with randomly generated 3 chars
mktemp test.XXX
Spliting files and data
# split [args] [prefix], e.g.
# -d: numeric suffix, -a 4: 4 suffix)
split -b 10k data.file -d -a 4

# ==> part01, part02, ...
split -b 10k data.file -d -a 2 part

# split to several text files each with 10 lines of text
split -l 10 Python.txt

# csplit - split a file into sections determined by context lines, e.g.
csplit server.log /SERVER/ -n 2 -s {*} -f server -b "%2d.log"

# Slicing filenames based on extension
file="sample.jpg"; name=${file%.*}; echo $name
# ==> "sample" (% is nongreedy, %% is greedy)

file="sample.jpg"; name=${file#*.}; echo $name
# ==> "jpg" (from left, ##: greedy)
Renaming and Moving files in Bulk
mv $before $after

# rename all jpg files to JPG files
rename *.jpg *.JPG

# <space> to "_"
rename 's/ /_/g'

# Upper to Lower:
rename 'y/A-Z/a-z/' *
# Lower to Upper:
rename 'y/a-z/A-Z/' *
Spell Checking and dictionary manipulation
ls /usr/share/dict
dvorak4tzx found something

Automating interactive Input

Making commands quicker by running parallel processes

Chap 3: File in, File out

Generating files of any size
The Intersection and set difference (A-B) on text files
Finding and deleting duplicate files
# create some duplicated files
echo "hello" > test ;
cp test test_copy1 ;
cp test test_copy2;
echo "next" > other;
# Filename:
# Description: Find and remove duplicate files and
#              keep one sample of each file.
ls -lS --time-style=long-iso | \
awk 'BEGIN {
    getline; getline;
    name1=$8; size=$5
        if (size==$5) {
            "md5sum "name1 | getline; csum1=$1;
            "md5sum "name2 | getline; csum2=$1;
            if ( csum1==csum2 ) {
                print name1; print name2
        size=$5; name1=name2;
    }' | \
sort -u > duplicate_files

cat duplicate_files | \
xargs -I {} md5sum {} | \
sort | uniq -w 32 | \
awk '{ print "^"$2"$" }' | \
sort -u > duplicate_sample

echo Removing..
comm duplicate_files duplicate_sample -2 -3 | \
tee /dev/stderr | xargs rm

echo Removed duplicates files successfully.
Working with file permissions, ownership, and sticky bit

File permissions and ownership are one of the distinguishing features of the Unix/Linux filesystems such as extfs (extended FS).

# permission set
-r, -w, -x
# special permissions
# file, x -> S, Permission S (S)
user:  setuid
# The setuid permission enables an executable file to be executed
# effectively as its owner, even when the executable is run by another
# user.

group: setgid
# This enables the item to run an executable file with an effective
# group as the owner group.

others:  NULL
# Others have the same read, write, and execute permissions as the user
# and group. But it does not have permission S (such as setuid or
# setgid).

# directories, Sticky Bit (t, T)
# only the user who created the directory can delete the files in the
# directory, even if the group and others have write permissions.

# ------rwt , ------rwT
cd /tmp; dir -al # to have a look
chmod, chown
# chmod
chmod u=rwx g=rw o=r filename
chmod 764 filename
chmod a+x filename
chmod a-x filename
# Setting sticky bit
chmod a+t directory_name
# Applying permissions recursively to files
chmod 777 . -R
# chown: `chown filename`, e.g.
chown slynux.slynux
# set setuid S
chmod +s executable_file

setuid is restricted such that setuid would not work for scripts, but only for Linux ELF binaries

Making files Immutable

Check mounted partion: cat /etc/mtab

rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
/dev/sda2 /media/Share fuseblk rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,blksize=4096 0 0
# /etc/resolv.conf, stores a list of DNS servers, by default is set to your
# ISP's DNS Server
chattr +i file # make immutable
chattr -i file # make not immutable

touch -d "Fri Jun 25 20:50:14 IST 1999" filename, specify timestamp

other options: -a modifies only the access time, -m for modification time

Finding symbolic links and their target


ln -s target symbolic_link_name # create symbolic link
ls -l symbolic_link_name # print out symlink
ls -l | grep "^l" # print out all symbolic links
find . -type l -print # use `find`, can also print out subdir files
readlink symbolic_link_name
Enumeration file types statics


file /etc/passwd # ==> /etc/passwd: ASCII text
file -b /etc/passwd # excluding file name, ==> ASCII text

make it a script to automate


if [ $# -ne 1 ];
    echo "Usage is $0 basepath";
declare -A statarray;
while read line;
    ftype=`file -b "$line" | cut -d, -f1`
    let statarray["$ftype"]++;
done < <(find $path -type f -print) # notice the <space> between "<", WHY?
echo ============ File types and counts =============
for ftype in "${!statarray[@]}";
    echo $ftype : ${statarray["$ftype"]}

run bash `pwd`, and on my computer I get:

============ File types and counts =============
assembler source : 2
PDF document : 1
HTML document : 1
Emacs v18 byte-compiled Lisp data : 1
GIF image data : 1
PNG image data : 1
Bourne-Again shell script : 1
ASCII text : 1


<(find $path -type f -print) is equivalent to a filename, the first < is for input redirection and the second < is for converting the subprocess output to a filename. In Bash 3.x and higher we have a new operator <<< that lets us use a string output as an input file. So we can do this:

done <<< "`find $path -type f -print`"
Using loopback files
Creating partions inside it
losetup /dev/loop1 loopbackfile.img
fdisk /dev/loop1
losetup -o 32256 /dev/loop2 loopbackfile.img

# Here,
#    /dev/loop2 will represent the first partition,
#    -o is the offset flag, 32256 bytes are for a DOS partition scheme.
#    The first partition starts after an offset of 32256 bytes from the
#    start of the hard disk.
Quicker way to mount loopback disk images with partitions
# this part have not tested myself
kpartx -v -a diskimage.img

# output:
# "add map loop0p1 (252:0): 0 114688 linear /dev/loop0 8192"
# "add map loop0p2 (252:1): 0 15628288 linear /dev/loop0 122880"

# This creates mappings from the partitions in the disk image to devices in
# /dev/mapper which you can then mount.
mount /dev/mapper/loop0p1 /mnt/disk1 # mount it

# do something and then unmount it
unmount /mnt/disk1

# remove the mappings by
kpartx -d diskimage.img
Finding the difference between files, patching

diff, patch

diff version1.txt version2.txt
diff -u version1.txt version2.txt
diff -u version1.txt version2.txt > version.patch

# version1.txt is no difference with version2.txt now
patch -p1 version1.txt < version.patch
Using head and tail for printing the last or first 10 lines
head file # print out 10 lines
cat text | head # read from stdin
head -n 4 file # 4 lines
head -n -4 flie # print all lines except the last 4 lines
seq 11 | head -n -4

# tail is  like head, except
tail -n +4 file # syntax: tail -n +M, print all except first (M-1) lines

# -f --follow option
tail -f /var/log/message

# this can read  last lines of log file, but can't keep tracking on
dmesg | tail -f

# Suppose we are reading a growing file, and a process Foo is appending
# data to the file, the -f tail should be executed until the process Foo
# dies.
tail -f file --pid $(pidof geany) # $(pidof geany) or $(pgrep geany)
Several ways to list dirs
ls -d */
ls -F | grep "/$"
ls -l | grep "^d"
print . -type d -maxdepth 1 -print
# I prefer `dir`, instead of `ls`, though there is one more typing

Fast command-line navigation

pushd and popd

pushd /var/www
pushd /usr/src
pushd +3 # 0, 1, 2, 3
dirs | tr ' ' '\n' | cat -n # I prefer this
# more useful tricks
cd - # switching to last path
cd # switching to home dir
Counting the number of lines, words, and characters in a file

wc: print newline, word, and byte counts for each file

wc -l file # count lines
wc -w file # count words
wc -c file # count characters
echo -n 1234 | wc -c # 4
echo    1234 | wc -c # 5, "1234\n"
wc file # lines, words, chars
wc file -L # the lenth of the longest line (metric: char)
Printing the directory tree


# tree path -P PATTERN, e.g.
tree -P "*.txt"
tree /root/Desktop -P "*.txt" -h # -h: print filesize in human way
tree PATH -H http://localhost -o output.html # output to a html file
tree -H `pwd` -o refs.html # create reference html pages

Chap 4: Texting and Driving

Using regular expressions

refs and see also

  1. date fomats

    date format
    Weekday %a(Sun) or %A(Sunday)
    Month %b(Feb) or %B(February)
    Day %d(02)
    Date in Format(mm/dd/yy) %D(02/02/14)
    Year %y(14) or %Y(2014)
    Hour %I(06) %H(18)
    Minute %M
    Second %S
    Nano Second %N
    Epoch Unix Time in seconds %s