UNIX/Linux Shell Scripting

title

UNIX/Linux Shell Scripting

author

Lukasz Sokolowski (NobleProg Ltd)

UNIX/Linux Shell Scripting

UNIX/Linux Shell Scripting Training Materials

Copyright Notice

Copyright © 2004-2025 by NobleProg Limited All rights reserved.

This publication is protected by copyright, and permission must be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or likewise.

THIS IS A DRAFT

This text may not be complete.

Review of Basics - optional

Files

touch filo.txt
echo first line > filo1.txt

Directories and Processes

mkdir diro

ps aux
ps -C chrome
top

Displaying Directories - ls

ls -la
ls -s
ls -d
ls -h
ls -S

Getting Help - man, info

--help

help ls

man ls
man man
man 8 useradd
man 5 passwd

info ls
info coreutils 'tac invocation'

Displaying Files - cat, more

 cat filo.txt
 more filo1.txt
 less filo.txt
 tac filo

less internal options h, /, ?

Simple Commands

• Sequence of optional variable assignments
• Fol‐lowed by blank-separated words and redirections
• Terminated by a control operator
• First word is command to be executed (arg 0)
• Remaining words are arguments for invoked command
• Returns its exit status or 128+n if terminated by signal n

Piping - using | symbol

Output of one command becomes the input of a second

ls | grep Do
gunzip file.tar.gz | tar xvf -

Redirection - using > symbol

• Input and output of a command may be redirected before it is executed
• May be used to open and close files for the current shell execution environment
• File descriptors: stdin(0), stdout(1) and stderr(2)

ls -l /dev/std*
ls -l /proc/self/fd/[0-2]

date
at 1020
ls -l /proc/self/fd > /var/tmp/fdtest.at
<CTRL+D>

Redirection - using >> symbol

Append to file

date >> backup.log
echo backup succeeded >> backup.log

Redirecting Input - using < symbol

command < file

read p1 < header.txt
read p2 < body.txt
read p3 < footer.txt
echo $p1 $p2 $p3 >> hbf.txt

Redirecting Errors

cd /var/spool/
ls -l * 2> /var/tmp/unac-in-spool
ls -l * > /var/tmp/spool 2>&1
ls -l * 2>&1 > /var/tmp/spool1
ls -l * &> /var/tmp/spool2

Using /dev/null

We just want a command to execute, no matter what output or errors it gives

ls -l * &> /dev/null

Foreground Jobs

vi body.txt
<CTRL+Z> # stop process
fg

Background Jobs - &

g /var/tmp/spooll1 &
jobs

Stopping Jobs - kill

ps a
kill 8209
jobs

Special Files - . and ..

ls -la .
ls -la ..

The Command Prompt - PS1

less ~/.bashrc

Creating Aliases

cd
alias
alias mnia='ls -laR > /var/tmp/testsearch'
unalias mnia
echo "# My aliases" >> .bash_aliases
echo "alias mnia='ls -laR > /var/tmp/testsearch'" >> .bash_aliases

Using Aliases

Useful

• for specifying the default version of a command that exists in several versions on your system
• to specify default options to a command
• for correcting incorrect spelling

mnia
ll

Exercises

• Exercise nr 1 (from point 13)
• Reverse one by one character, use 'filo1.txt'
• Hack 'ls' command, it should result with 'sl' program instead (after playing go back to the usual 'ls')

Command history

history
!!
!4
history 10
less .bash_history
<CTRL+R>

Command Line Edit - using Arrow Keys

UP and DOWN arrow keys allow to search through history of typed commands

Command Line Edit - using vi

vim mniam.txt
:
<CTRL+R>
'%', '/', '.', ':'
history

www.openvim.com

vim-adventures.com

Customizing the Shell

• Startup files:
  • /etc/profile
  • /etc/bashrc (bash.bashrc)
  • $HOME/.bash_profile (.bashrc)
  • $HOME/.bash_logout, $HOME/.bash_history, $HOME/.bash_aliases
  • /etc/passwd
• Environmental variables
• Commands:

shopt -s cdspell
shopt -u
export
set
help
env

The Home Directory - HOME

echo $HOME
cd
adduser
cat /etc/passwd

The Working Directory - pwd

pwd
env pwd
whereis pwd
/bin/pwd
help pwd
echo $PWD

Changing Directories - cd

cd
cd ..
cd ../../
cd -LPe@

Make Directory - mkdir

mkdir -v 1 2 3
mkdir 2
mkdir -p 3/3a/3aa
ls -la -R 3/
mkdir -m a=rwx jimmy

Remove Directory - rmdir

rmdir -v 2
rmdir -p 3/3a/3aa

Interactive Copy - cp -i

cp mniam 1
cp -i mniam 1

Interactive Move - mv -i

mv mniam 1/mniam
cp 1/mniam mniam
mv -i mniam 1/mniam

Interactive Remove - rm -i

 
rm -i 1/mniam

Removing Directories - rm -r

rm -r 3

Wild Cards - File Name Generation

?, *, ^, [], [-], {}

echo rm -R *
ls -l h??.txt
ls -l [hf]*
ls -l [b-h]*
ls -l [^hf]*
ls -l {b*,p*,*tuf*}

Word count - wc

Print the number of bytes, words and lines in files

wc -m header.txt
wc -l footer.txt
wc -L hbf.txt
wc -w header.txt footer.txt
wc -w -
umpa lumpa dumpa
<CTRL+D>

Finding Files - find

find / -name passwd
sudo find / -name passwd
find . -empty
find . -executable
find . -group drupalpro
find . -type d

Finding Text in Files - grep, sort, cut, uniq, tr

cat /etc/passwd | grep -v -c root
ls */* | sort -t / -k 1,1R -k 2,2
cut -f 2 -d " " header.txt
uniq -d header.txt
tr a-z A-Z

Exercise

• Exercise nr 10b (from point 13)

Quiz

What will be the result? First try to guess without testing commands

df -h | grep -v File | awk '{ print $5 }' | cut -d "%" -f1 -

ls */* | sort -t / -k 1,1R -k 2,2

awk '{print length, $0}' /etc/passwd | sort -n | cut -f2- -d' '

Changing Permissions - chmod

chmod a+rwx header.txt
chmod 755 footer.txt
chown drupalpro:drupalpro hbf.txt

Shell Programming

Creating Shell Scripts

cd
mkdir myAwesomeScrs
cd myAwesomeScrs
touch myScr.sh
nano myScr.sh
# adding shabang -> #!/bin/bash
# printing to screen -> echo 'my first script'

Testing the Command

cd
cd myAwesomeScrs
myScr.sh
# shouldn't work yet, why?

Creating the Script File

cd ~/myAwesomeScrs
> scrFile.sh
echo '# This is a comment' > scrFile1.sh

Adding Execute Permissions

cd ~/myAwesomeScrs
echo '# this one will work' > scrFile2.sh
echo 'echo "works"' >> scrFile2.sh
chmod u+x scrFile2.sh

Testing the Script

cd ~/myAwesomeScrs
bash scrFile2.sh

Running the Script - time

time ./scrFile2.sh
time -p ./scrFile2.sh
help time

Exercise

• Exercise nr 10c (from point 13)

Running the Script - dot

./scrFile2.sh
. scrFile2.sh
help .

Running the Script - exec

exec ./scrFile2.sh
# what the heck?! (-;
# open CLI again
bash
exec ./scrFile2.sh
help exec

Running the Script - ksh

ksh
apt install ksh
ksh scrFile2.sh
ksh --help

Debugging the Script

cd ~/myAwesomeScrs
echo '#!/bin/bash -x' > scrFile3.sh
echo '' >> scrFile3.sh
echo 'echo "this will be doubled"' >> scrFile3.sh
echo '' >> scrFile3.sh
echo 'date' >> scrFile3.sh
echo '' >> scrFile3.sh
echo 'echo "End of story"' >> scrFile3.sh
chmod u+x scrFile3.sh
./scrFile3.sh

Trap

#!/bin/bash
# traptest.sh

trap "echo Booh!" SIGINT SIGTERM # man 7 signal , kill -l
echo "pid is $$"

while :			# This is the same as "while true".
do
        sleep 60	# This script is not really doing anything.
done

Exercise

• Exercise nr 2 (from Point 13)

Passing Parameters

./generator
echo 'echo "First argument is: $1"' >> scr.sh
./scr.sh meow

The read Command

read var
echo $var
read var1 var2 var3
echo $var{1..3}

Using read in Scripts

echo -e "Give me fever: \n"
read fevervar
[ $fevervar ] && echo "You're so hot "$fevervar
[ ! $fevervar ] && echo "Don't cheat!" && exit 1
exit 0

Choosing the Shell

Comparison of shells

#!/bin/sh
### Bourne shell, simple and restricted, good for security reasons
#!/bin/csh
### C shell, supports internally a lot of C language
#!/bin/ksh
### Korn shell, backward-compatible with the Bourne shell and includes many features of the C shell
#!/bin/bash
### man bash, info bash
#!/bin/zsh
### https://www.zsh.org/

Tidying The Output

# scrFile4.sh

### With echo
echo -e "I have some tidy content in me.\nEach sentence is separated into the new line.\nAlso there is one tabulator here:\there I'am."
echo "Hey Mate. What is the biggest kangaroo in the world? Give me the proper answer: "
read kangaroo
echo -e "You gave the wrong one, sorry.. It's not the $kangaroo. It was Roger!"

### Readability
printf 'n\np\n3\n\n\n\n\nt\n3\n8e\nw' | fdisk /dev/sda
# with HERE docs
fdisk /dev/sda <<EOF
n
n
3


t
8e
w
EOF

### Exit codes
cd ~/myAwesomeScrs
touch scrFile999.sh ; [ -f "scrFile999.sh" ] && echo "You are focused and following the examples properly! Good job, nice one, Mate!" && exit 0
#rm scrFile999.sh ; [ ! -f "scrFile999.sh" ] && echo "You naughty boy! Come on, be a good one and follow the trainer's steps, right?" && exit 1
#  Run this script and check the value of global var: '$?'
#  Comment out the line 25. Uncomment the line 26, then run this script again and check '$?'

Tidying The Output Con't

• Colors

env | grep color
cat ~/.bashrc | grep color

• Parsing

# scrFile5.sh
echo '<div class="">Tidy.<div>' > mytags.html
apt install tidy -y
tidy mytags.html

Suppressing Input - stty

• Manages baud rate, line discipline, deviations

stty sane
stty -echo # echo
stty -a

• Examples

grep stty /etc/*/*

Special Characters and Variables

• Quoted Strings - "', ''
• Escape Character - \
• Quoted Strings - `
• System Variables
• User Variables
• Variables in Scripts
• Other Special Variables

Quotation Marks

# scrFile6.sh

a=375
hello=$a

echo "$hello"    # 375
echo "${hello}"  # 375
echo
hello="A B  C   D"
echo $hello   # A B C D
echo "$hello" # A B  C   D
echo
echo '$hello'  # $hello

Escape Character - \

echo "This will print
as two lines."

echo "This will print \
as one line."

p=20
echo "\$p = $p"

Quoted Strings - `

#!/bin/bash

a=23              # Simple case
echo $a
b=$a
echo $b

# Now, getting a little bit fancier (command substitution).

a=`echo Hello!`   # Assigns result of 'echo' command to 'a' ...
echo $a
#  Note that including an exclamation mark (!) within a
#+ command substitution construct will not work from the command-line,
#+ since this triggers the Bash "history mechanism."
#  Inside a script, however, the history functions are disabled by default.

a=`ls -l`         # Assigns result of 'ls -l' command to 'a'
echo $a           # Unquoted, however, it removes tabs and newlines.
echo
echo "$a"         # The quoted variable preserves whitespace.

exit 0

Using $(...)

• Save the result to a variable
  • Command substitution

script_name=$(basename $0)
echo "The name of this script is $script_name."

# The $(...) form of command substitution treats a double backslash in a different way than `...`.
echo `echo "\\"`
echo $(echo "\\")

System Variables - set

• The set command enables options within a script.

#!/bin/bash
set -o verbose
# Echoes all commands before executing.

set -v
# Exact same effect as above.

set +v
# Command echoing off.
command
# Not echoed.

Variable with a null value

#!/bin/bash

echo "$uninitialized"                                # (blank line)
let "uninitialized += 5"                             # Add 5 to it.
echo "$uninitialized"                                # 5

User Variables - unset

• Destruction of variables unset p1 p2 p2
• Example

#!/bin/bash

variable=hello                     
echo "variable = $variable"
unset variable

Variables in Scripts

• Positional parameters ($0, $#, $1, $2, $3 ...)
  • Arguments passed to the script from the command line [1] : $0, $1, $2, $3 . . .
  • $0 is the name of the script itself, $1 is the first argument, $2 the second, $3 the third, and so forth.
  • After $9, the arguments must be enclosed in brackets, for example, ${10}, ${11}, ${12}

#!/bin/bash
# scriptname 1 2 3

MINPARAMS=3
echo "The number of parameters is \"$#\"."
echo "The name of this script is \"$0\"."
echo "The name of this script is \"`basename $0`\"."
echo
if [ -n "$1" ]
then
 echo "Parameter #1 is $1"
fi 

if [ -n "$2" ]
then
 echo "Parameter #2 is $2"
fi 

if [ -n "$3" ]
then
 echo "Parameter #3 is $3"
fi

Exercise

• Exercise nr 3

Other Special Variables

• $@ and $*
  • Denote all the positional parameters.
• $#
  • Number of params passed to the script
• $?
  • Use the status of completion of the program in a script

# return last command status code
echo $?

Other Special Variables Example

#!/bin/bash
# Try multiple parameters with white spaces as well

for vars in $*
do
  echo "Looping \$*, \$vars is:"
  echo $vars
done

for vars in $@
do
  echo "Looping \$*, \$vars is:"
  echo $vars
done

exit 0

Selection

Conditional statements

• The if / then / else
  • One liner if,then
  • Nesting if statements
  • The if / elif
• What is truth
• Quoting variables being tested
• and vs or lists
• case statement

The if / then

• Weird syntax
  • fi statement
  • no curlies {}
• Fragile with empty spaces ([condition-true ])

if [ condition-true ]
then
   command 1
   command 2
   ...
fi

The if / then / else

if [ condition-true ]
then
   command 1
   command 2
   ...
else   
   command 3
   command 4
   ...
fi

One liner if,then

• Keywords (or commands) begin statements
• Before a new statement on the same line begins, the old one must terminate

if [ -x "$filename" ]; then ...

What is truth

• One of the biggest easter eggs in the shell
• Zero means ok, everything else is not

if [ 0 ]      # zero
then
  echo "0 is true."
else          # Or else ...
  echo "0 is false."
fi            # 0 is true.

Nesting if statements

a=3

if [ "$a" -gt 0 ]
then
  if [ "$a" -lt 5 ]
  then
    echo "The value of \"a\" lies somewhere between 0 and 5."
  fi
fi

# Same result as:

if [ "$a" -gt 0 ] && [ "$a" -lt 5 ]
then
  echo "The value of \"a\" lies somewhere between 0 and 5."
fi

Quoting variables being tested

• A Good Practice

#!/bin/bash
# if-3.sh

if [[ -z "$1" ]]
then
  echo "Need one argument!"
fi

Exercise

• Exercise nr 4

and list (1)

• A group of and operators
• Sort of net, with piped logic

#!/bin/bash
# if-4.sh
# command-1 && command-2 && command-3 && ... command-n

if [ ! -z "$1" ] && echo "Argument #1 = $1" && [ ! -z "$2" ] && \
	echo "Argument #2 = "$2""
then
	echo "2 arguments passed to script."
else
	echo "Fewer than 2 arguments passed to script."
fi

exit $?

and list (2)

#!/bin/bash
# if-5.sh

ARGS=1        
E_BADARGS=85

[[ $# -ne "$ARGS" ]] && echo "Usage: $(basename $0) "$ARGS" argument(s)" && exit "$E_BADARGS"

echo "Correct number of arguments passed to this script."
exit 0

or list (1)

• A group of or operators

#!/bin/bash
# if-7.sh
# command-1 || command-2 || command-3 || ... command-n

E_BADARGS=85

[[ -z "$1" ]] && echo "Usage: `basename $0` filename" && exit $E_BADARGS  

file=$1
  
[ ! -f "$file" ] && echo "File \"$file\" not found. Refusing to delete a nonexistent file."
[ ! -f "$file" ] || (rm -f $file; echo "File \"$file\" deleted.")

exit $?

or list (2)

# /etc/rc.d/init.d/single
[ -x /usr/bin/clear ] && /usr/bin/clear

for i in /etc/rc1.d/S[0-9][0-9]* ; do
        [ -x "$i" ] || continue

        # Reject backup files and files generated by rpm.
        case "$1" in
                *.rpmsave|*.rpmorig|*.rpmnew|*~|*.orig)
                        continue;;
        esac
        [ "$i" = "/etc/rc1.d/S00single" ] && continue
  # ==> Set script name, but don't execute it yet.
        $i start
done

The if / elif

• Multiple conditions
• Good for different facts

if [ condition1 ]
then
   command1
   command2
   command3
elif [ condition2 ]
# Same as else if
then
   command4
   command5
else
   default-command
fi

case statement

• Multiple conditions
• Good for same fact with different values
• Really weird syntax - *), esac

case "$variable" in 

 "$condition1" ) 
 command... 
 ;; 

 "$condition2" ) 
 command... 
 ;; 

esac

case

#!/bin/bash

case $( arch ) in   # $( arch ) returns machine architecture.
                    # Equivalent to 'uname -m' ...
  i386 ) echo "80386-based machine";;
  i486 ) echo "80486-based machine";;
  i586 ) echo "Pentium-based machine";;
  i686 ) echo "Pentium2+-based machine";;
  *    ) echo "Other type of machine";;
esac

exit 0

Exercise

• Exercise nr 5

Operators

• No floating point arithmetic
• Operators and, or (&&, ||)
• test command

No floating point arithmetic

a=1.5

let "b = $a + 1.3"  # Error.

Operators and, or (&&, | |)

if [ $condition1 ] && [ $condition2 ]
#  Same as:  if [ $condition1 -a $condition2 ]

if [[ $condition1 && $condition2 ]] 

if [ $condition1 ] || [ $condition2 ]
# Same as:  if [ $condition1 -o $condition2 ]

if [[ $condition1 || $condition2 ]]

test command

• An alternative for square brackets

if test -x $file
then
...

if [[ -x $file ]]
then
...

bash$ whereis [
: /usr/bin/[ /usr/share/man/man1/[.1.gz
bash$ type test
test is a shell builtin <---- no need to call /usr/bin/test

bash$ info test

Iteration, Looping

• Iteration - while, until
• Infinite Loop
• Iteration - for

The loop for / in

for arg in [list]
do 
 command(s)... 
done

for loop - 1

for arg in "$var1" "$var2" "$var3" ... "$varN"  
# In pass 1 of the loop, arg = $var1	    
# In pass 2 of the loop, arg = $var2	    
# In pass 3 of the loop, arg = $var3	    
# ...
# In pass N of the loop, arg = $varN

# Arguments in [list] quoted to prevent possible word splitting.

for - 2

for arg in [list] ; do

for - 3

#!/bin/bash

for planet in Mercury Venus Earth Mars Jupiter Saturn Uranus Neptune Pluto
do
  echo $planet  # Each planet on a separate line.
done

for - 4

#!/bin/bash

for planet in "Mercury Venus Earth Mars Jupiter Saturn Uranus Neptune Pluto"
do
  echo $planet
done

exit 0

for - 5

#!/bin/bash

for planet in "Mercury 36" "Venus 67" "Earth 93"  "Mars 142" "Jupiter 483"
do
  set -- $planet  
  echo "$1		$2,000,000 miles from the sun"
done

exit 0

for - 6

#!/bin/bash

FILES="/usr/sbin/accept
/usr/bin/fakefile
/sbin/ypbind"    

for file in $FILES
do
  if [ ! -e "$file" ]      
  then
    echo "$file does not exist."; echo
    continue                
  fi
  ls -l $file | awk '{ print $8 "         file size: " $5 }'  # Print 2 fields.
  whatis `basename $file`   # File info.
  echo
done  

exit 0

for - 7

#!/bin/bash

filename="*txt"

for file in $filename
do
 echo "Contents of $file"
 echo "---"
 cat "$file"
 echo
done

Exercise

• Exercise nr 11

The command seq

• Gives a sequenced data

seq 5
seq 5 10
seq 5 2 9
seq -s ,5

The while loop

• Be careful
  • It might not execute your code at all if the condition is false

while [ condition ]
do 
 command(s)... 
done

Example 1 - Simple while loop

#!/bin/bash

var0=0
LIMIT=10

while [ "$var0" -lt "$LIMIT" ].
do
  echo -n "$var0 "
  let "var0 += 1"
done

Example 2 - Another while loop

#!/bin/bash

while [ "$var1" != "end" ]
do
  echo "Input variable #1 (end to exit) "
  read var1
  echo "variable #1 = $var1"
  echo
done  

exit 0

Example 4 - C-style syntax in a while loop

#!/bin/bash
# while-4.sh

LIMIT=10
((a = 1))

while (( a <= LIMIT ))   
do                       
  echo -n "$a "
  ((a += 1))             

done

Example 5 - call function in test condition

t=0

condition ()
{
  ((t++))
  if [ $t -lt 5 ]
  then
    return 0  # true
  else
    return 1  # false
  fi
}

while condition
do
  echo "Still going: t = $t"
done

Example 6 - reading file

#!/bin/bash

[ $# -lt 1 ] && echo "Usage: `basename $0` filename" && exit || filename=$1

cat $filename |   # Supply input from a file.
while read line   # As long as there is another line to read ...
do
  echo $line
done

Loop until

until [ condition-is-true ]
do 
 command(s)... 
done

Example 1 - until loop

#!/bin/bash

END_CONDITION=end

until [ "$var1" = "$END_CONDITION" ]
do
  echo "Input variable #1 "
  echo "($END_CONDITION to exit)"
  read var1
  echo "variable #1 = $var1"
  echo
done

Example 2 - C-style

#!/bin/bash

LIMIT=10
var=0

until (( var > LIMIT ))
do
  echo -n "$var "
  (( var++ ))
done

exit 0

Controlling the loop

• Break
• Continue

Break

• Exits the step in the loop
• Finishes the rest of the looping

#!/bin/bash

LIMIT=20
a=0

while [ "$a" -le "$LIMIT" ]
do
 a=$(($a+1))

 if [ "$a" -gt 2 ]
 then
   break  # Skip entire rest of loop.
 fi

 echo -n "$a "
done

Continue

• Exits the step in the loop
• Doesn't finish the rest of the looping

#!/bin/bash

LIMIT=19
echo "Printing Numbers 1 through 20 (but not 3 and 11)."

a=0
while [ $a -le "$LIMIT" ]
do
 a=$(($a+1))
 if [ "$a" -eq 3 ] || [ "$a" -eq 11 ]
 then
   continue
 fi
 echo -n "$a "
done 

# Exercise:
# Why does the loop print up to 20?

More than 9 Parameters - shift

 shift N

• Shifts positional parameters to the left by N
  • from 'N+1' to '$#' are renamed to variable names from '$1' to '$# - N+1'
  • if a command takes 10 args, N is 4, then: $4 => $1, $5 => $2, etc - $10 => $7 and orig $1, $2, $3 are thrown away
  • for N = 0 or N > $#, it does nothing
  • by default N=1

Manual shift example

#!/bin/bash

if [[ $# -lt 3 ]]
then
	echo "Try give more than 3 args..."; 
	exit 1
fi
echo $*
shift 3
echo $*

Creating Menus

• Interactive programs
  • User select
    • PSs

User select

select variable [in list]
do 
 command... 
 break 
done

PS1-4

export PS1="\u@\h \w> "

export PS2="continue-> "

PS3='Choose your favorite vegetable: '

export PS4='$0.$LINENO+ '

Exercise

• Exercise nr 6

Sub-scripts and Functions

function function_name { 
command... 
} 

function_name () { 
command... 
}

• Local variables
• Without parameters
• With parameters
• Examples - /etc/wpa_supplicant/functions.sh
  • https://w1.fi/wpa_supplicant/

Functions Con't

updateMeViaFunc=''

function_name () { 
  local funcVa='I will die outside'
  echo "First param was: $1"
  echo "Called from the inside: $funcVa"
  updateMeViaFunc="I'm returned from function too"
  return 0 
}

function_name "myParam"
echo "I'm returned from function: $?"
echo "Called from the outside: $funcVa"
echo "Global variable changed by function: $updateMeViaFunc"

Sub-scripts

source scriptname.sh
. scriptname.sh

Exercise

• Exercise nr 8

Advanced Commands

• Test operators
• Operators
• Comparison operators

Test file

• -e exists ( file or directory both true )
• -s file not zero size

Test the file type

• -f regular file ( not directory, or device file)
• -d is directory

Test file attributes

• -r file has read permission (for the user running the test)
• -w file has write permission (for the user running the test)
• -x file has execute permission (for the user running the test)

The assignment operator

# assignment
var=27
category=minerals

Arithmetic operators

let "z=5**3"    # 5 * 5 * 5
echo "z = $z"   # z = 125
bash$ expr 5 % 3 # 2

let "var += 5"
let "var -= 4"
let "var *= 5"
let "var /= 4"
let "var %= 4"

Arithmetic Expansion

Translating a string into a numerical expression backticks, double parentheses, or let.

z=`expr $z + 3`
z=$((z+3)) #    Not to be confused with command substitution. $()
let "z = z + 4"
echo $z

Operators operations on bits (bitwise operators)

<< # left shift 1 bit ( x2 )
<<= # left-shift-equal
let "var <<= 2" # results in var left-shifted 2 bits (multiplied by 4)
>> # bitwise right shift (divides by 2 for each shift position)
>>= # right-shift-equal (inverse of <<=)
& # bitwise AND
&= # bitwise AND-equal
| # bitwise OR
|= # bitwise OR-equal
~ # bitwise NOT
^ # bitwise XOR
^= # bitwise XOR-equal

Logical operators

# ! NOT
if [ ! -f $FILENAME ]
then
  ...

# && AND
if [ $condition1 -a $condition2 ]
if [ $condition1 ] && [ $condition2 ]
if [[ $condition1 && $condition2 ]]    # Also works.
#  Note that && operator not permitted inside brackets of [ ... ] construct.

# || OR
if [ $condition1 -o $condition2 ]
if [ $condition1 ] || [ $condition2 ]
if [[ $condition1 || $condition2 ]]    # Also works.
#  Note that || operator not permitted inside brackets

Numeric constants

let "dec = 32"
echo "decimal number = $dec"             # 32

# "0"
let "oct = 032"
echo "octal number = $oct"               # 26

# "0x"
let "hex = 0x32"
echo "hexadecimal number = $hex"         # 50

# BASE#NUMBER
let "bin = 2#111100111001101"
echo "binary number = $bin"              # 31181

Comparing arithmetic using the command if

if [ "$a" -ne "$b" ]
then
  echo "$a is not equal to $b"
  echo "(arithmetic comparison)"
fi

Integer comparison operators

if [ "$a" -eq "$b" ]
if [ "$a" -ne "$b" ]
if [ "$a" -gt "$b" ]
if [ "$a" -ge "$b" ]
if [ "$a" -lt "$b" ]
if [ "$a" -le "$b" ]
(("$a" < "$b"))
(("$a" <= "$b"))
(("$a" > "$b"))
(("$a" >= "$b"))

Comparing strings

Always quote a tested string.

if [ "$a" = "$b" ]
if [ "$a" != "$b" ]
if [ -z "$String" ] # zero length (null)
if [ -n "$String" ] # not null

Comparing the complex (and, or)

if [ "$a" -eq 24 -a "$b" -eq 47 ]

if [ "$a" -eq 98 -o "$b" -eq 47 ]

if [[ "$v1" -gt "$v2" ]] || [[ "$v1" -lt "$v2" ]] && [[ -e "$filename" ]]

Regular Expressions

• Commonly used in scripts, such as grep, sed and awk.
  • As of version 3, Bash has acquired its own RE-match operator: =~.

Know RE

• A character set
• An anchor - ^, $
• Modifiers - *, (), \, []

*,.,^,$

• "1133*" : 113, 1133, 1133333...

.

• "13." : 1133, 11333

*,.,^,$

• "^" : line start

^,$

• "$" : line end "XXX$"
• "^$": empty line

[...]

• "[xyz]"
• "[c-n]"
• "[B-Pk-y]"
• "[a-z0-9]"
• "[^b-d]"
• "[Yy][Ee][Ss]" : yes, Yes, YES, yEs ..
• "[0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9][0-9][0-9]" : NNN-NN-NNNN

\

• "\$"
• "\\"

\<\>

word boundary

"\<the\>" matches "the," but not "them," "there," "other," etc.

Extended RE +

+

"+" matches one or more of previous RE

echo a111b | sed -ne '/a1\+b/p'
echo a111b | grep 'a1\+b'

\{\}

number of occurrences

# 5 digits exact
echo -e "23\n2345\n23413\n234" | grep "[0-9]\{5\}"

if [[ $input =~ ^[0-9]{3}-[0-9]{2}-[0-9]{4}$ ]]

(),|

• () group REs
• | alternatives

echo -e "read\nreed\nraad" | grep -E 'r(ee|aa)d'

getopts Command

• Parses command-line arguments passed to the script.
• It permits passing and concatenating multiple options and associated arguments to a script
  • scriptname -xyz -w /var/spool
• Argument pointer($OPTIND), optional argument attached to an option ($OPTARG)
  • ':' after option name says it has associated argument
• Usually looped with 'while' to process options and arguments one at a time
• Provided arguments must be preceded by dash (-), lacks condition brackets
• External alternatives: getopt external command, getopt library function

getopts Command example

# Here we observe how 'getopts' processes command-line arguments to script.
# The arguments are parsed as "options" (flags) and associated arguments.

# Try invoking this script with:
#   'scriptname -mn'
#   'scriptname -oq qOption' (qOption can be some arbitrary string.)
#   'scriptname -qXXX -r'
#
#   'scriptname -qr'
#+      - Unexpected result, takes "r" as the argument to option "q"
#   'scriptname -q -r' 
#+      - Unexpected result, same as above
#   'scriptname -mnop -mnop'  - Unexpected result
#   (OPTIND is unreliable at stating where an option came from.)
#
#  If an option expects an argument ("flag:"), then it will grab
#+ whatever is next on the command-line.

NO_ARGS=0 
E_OPTERROR=85

if [ $# -eq "$NO_ARGS" ]    # Script invoked with no command-line args?
then
  echo "Usage: `basename $0` options (-mnopqrs)"
  exit $E_OPTERROR          # Exit and explain usage.
                            # Usage: scriptname -options
                            # Note: dash (-) necessary
fi  


while getopts ":mnopq:rs" Option
do
  case $Option in
    m     ) echo "Scenario #1: option -m-   [OPTIND=${OPTIND}]";;
    n | o ) echo "Scenario #2: option -$Option-   [OPTIND=${OPTIND}]";;
    p     ) echo "Scenario #3: option -p-   [OPTIND=${OPTIND}]";;
    q     ) echo "Scenario #4: option -q-\
                  with argument \"$OPTARG\"   [OPTIND=${OPTIND}]";;
    #  Note that option 'q' must have an associated argument,
    #+ otherwise it falls through to the default.
    r | s ) echo "Scenario #5: option -$Option-";;
    *     ) echo "Unimplemented option chosen.";;   # Default.
  esac
done

shift $(($OPTIND - 1))
#  Decrements the argument pointer so it points to next argument.
#  $1 now references the first non-option item supplied on the command-line
#+ if one exists.

exit $?

Exercise

• Exercise nr 9

Using Arrays

• Newer versions of Bash support one-dimensional arrays.

Simple array

• initialized with the variable[xx] notation

area2=( zero one two three four )
area[6]=`expr ${area[11]} + ${area[51]}`
echo ${area2[0]}

Declare array

• a script may introduce the entire array by an explicit declare -a variable statement.

declare -a bigOne=( /home/jusfeel/shell-scripting/Day-1/* )

Dereference array

• To dereference (retrieve the contents of) an array element, use curly bracket notation, that is, ${element[xx]}.

for f in ${bigOne[@]}
do
        echo $f
done

Exercise

• Exercise nr 7

Here document

#!/bin/bash
if [ $# -lt 1 ]; then
  echo "Usage: $0 package(s)"
  exit 1
fi
while (($#)); do
  sudo apt-get install "$1" << CONFIRM
y
CONFIRM
  shift
done

Exercise

• Exercise nr 14

Batch Stream EDitor - sed

• Performs basic transformations on text read from a file or a pipe
• The result is sent to standard output
• Has no output file specification, we can use output redirection
  • sed option 'some/expression' file_to_process > sed_output_in_a_file
• Does not modify the original input
• Filters text that it gets from a pipeline feed (vi and ed don't)
• We don't need to interact with the editor while it is running (batch editor)
• Used in scripts, greatly easing repetitive editing tasks (esp. replacement of text in a large number of files)

Example file content

This is the first line of an example text.
It is a text with erors.
Lots of erors.
So much erors, all these erors are making me sick.
This is a line not containing any errors.
This is the last line.

sed delete, /d

cat -n example
sed '/erors/d' example
sed -n '/^This.*errors.$/p' example
sed '2,4d' example
sed '3,$d' example
sed -n '/a text/,/This/p' example

sed print, /p

sed '/erors/p' example

sed suppress, -n

sed -n '/erors/p' example
sed -n '/a text/,/This/p' example

sed Find and Replace

sed 's/erors/errors/' example
sed 's/erors/errors/g' example
sed 's/^/> /' example
sed 's/$/EOL/' example
sed -e 's/erors/errors/g' -e 's/last/final/g' example

sed Examples

grep sed /etc/init.d/*

Text File Processing - awk

• Yet another popular stream editor
  • a pattern scanning and processing language
• Usually implemented as GUI version like 'gawk', etc
• To search files for lines or other text units containing one or more patterns
  • When a line matches one of the patterns, special actions are performed on that line
• Different from programs in most other languages, which are "procedural"
  • "data-driven": you describe the data you want to work with and then what to do when you find it
  • You have to describe, in great detail, every step the program is to take
  • Easy to read and write

The awk Programming Tool

run awk = specify an awk program = tells awk what to do

• The program consists of a series of rules
• Each rule specifies
  • one pattern to search for
  • and one action to perform upon finding the pattern
• It may also contain advanced features:
  • function definitions, loops, conditions and other programming constructs
• Ways to run awk:
  • for the short program, run it on the command line

    • awk PROGRAM inputfile(s)

  • for regular multiple changes and on multiple files, put the awk commands in a script

    • awk -f PROGRAM-FILE inputfile(s)

Pattern Matching

• The condition may be:
  • a regular expression
  • a test of a variable
  • an arithmetic expression
  • or anything that produces a non-zero or nonempty result
• It may represent a range by giving two conditions separated by a comma
  • once a line matches the first condition, the action is performed until a line matches the second condition
• Special conditions, BEGIN and END.
  • BEGIN is performed before any lines are read
  • END action is performed after all the lines have been read or another action executes an exit statement
• If there is no condition(can be missing/omitted), the action is applied to all lines

Processing input

• Each line is split into fields based on the contents of the variable FS
  • FS by default is any whitespace
  • Fields are numbered: $1, $2, ...
    • $0 contains the entire line
  • Variable NF contains the number of fields in the line

Range Patterns

NR == 10, NR == 20 (NR - current line number)

Action Statements

• Can be any computation task:
  • modify the input line
  • save it in a variable
  • it can perform a calculation on it
  • it can print some or all of the line
  • can actually do anything else you can think of
• Like the condition, action may be missing/omitted
• If there is no action, matching lines are printed

awk Examples

awk example 1

'print' command

df -h | awk '{ print $2 " " $4 }'

Exercise

• Exercise nr 15

References

Executing Commands

Special Shell Variables

Test Operators - Files

Miscellaneous Constructs

Regular Expression

Comparison of Command Shells

Sample .bashrc

Exercises

0. Install 'git' and create new repository in '~/Documents/scripts' folder.
- sudo apt install git
- cd ; mkdir Documents/scripts ; cd Documents/scripts
- git init ; git add . ; git commit -m "my own scripts generator"
- git config --global user.name "foo bar" ; git config --global user.email "foo@bar.com"
- git commit -m "my own scripts generator" ; git log
Keep our generator.sh versioned with git. 


1. Prepare aliases for those commands:
1.1. find /etc/ -name apache
1.2. git log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit


2. Create script 'generator.sh':
- it should add '#!/bin/bash' to the file
- change permission (x)	
- echo message "Done."


3. Extend our generator:
- it should take as an argument the name of the file
- pass as a second argument name of the catalog in which you want to save the file


4. Improve our generator:
- add usage info with if statement


5. Change the generator:
- test how many variables user provided
- allow to provide 1 or 2, but no 0 arguments
- provide folder and file existence checks
- if folder doesn't exist ask the user if he wants to create it or not (HINT: use 'read' command)


6. Add menu to our generator
- '0', '1', '2' and 'Q' should be the choice list
-- 0 no arguments
-- 1 file name only
-- 2 file and folder name
-- Q quit the script


7. Create a variable, which will be an array with names of our scripts:
- echo all elements from this array
- add one more element to this array, but do it in a smart way(check how long this array is)
- echo only last element from this array


8. Prepare even smarter version of generator:
- replace repetitive code with a function(s)
- create 'sources.sh' file and place the function(s) there


8.1. Make another generator
- It should allow user to choose:
-- if he wants to have default constructs in it (if, case, function, loops)
-- which of those constructs
-- how many of them


9. Create script 'howmany' which will use two options:
- '-a' will count how many different users are logged in
- '-b' will show how many cores/processors are available on server


10. 'grep' exercise

Find all users who have no login option via terminal
a) do it without 'grep' command (create small script)
b) use 'grep' 
c) compare speed of both ways ('time' command), what can you say?


10.1. http://ebible.org/kjv/

Prep similar example with the ASCII file from above link.
Find all occurrences of word 'Jesus'. 
Do it with 'grep' and 'sed'.
Compare results with 'time'.


11. Search apache logs for keyword (for example "error")
Do it in loop 'for'


12. Do the script in which you search for all files with specific extension. 
Create function 'search' and it should use parameter with extension.
Script should take extension as an argument.
Create a variable 'testy' inside of the function and save extension in it.
Echo "$testy" outside of the function after invoking it.


13. Add task to crontab. It should 'ls' into file 'list'.


14. Make a script which will allow the user to choose about what file should be removed.
Use 'here document'.


15. Use file 'example' and program 'awk':
a)
- take only the first 2 words from each line
- add 2 tabulators between these 2 words
- add 'Meow!' at the end of each line
b)
- prepare parameter and pass 'Meow!' into awk program via this parameter
- provide comment message after execution of awk program


16. Web-service example
a) client(bash script) should call this end-point 'http://www.youtypeitwepostit.com/api/'
b) ask about post "Squidly!" and get the result in 'json' format
c) search in json response for post's date and get the day and month from it's value
d) print nice message like this: "Oops! Squidly pooped himself ..eee.. made a lot of ink on 23rd of July"
e) (*) migrate the whole 'data' into 'drupal' website
-- enable the 'blog' module and create one blog (monkey job, just clicking)
-- prepare 'MySQL' query (REAL GEEKY STUFF!)
-- use drupal's 'drush' to execute sql query from our bash script (lazy way for lazy bastards like me)


17. Install debugger and play with it ( http://bashdb.sourceforge.net/ )
- do some specific mistakes in one of our generators and test them with debugger
-- syntax error
-- typos
-- missing function definition
- test some of these http://www.tldp.org/LDP/abs/html/abs-guide.html#GOTCHAS