Usp Mod 4
Usp Mod 4
SHELL
PROGRAMMIN
G
MODULE 4
U N I X C O N C EP TS &
AP P L I C ATI O N S 4 T H ED I TI O N
BY S U M I TAB H A D AS
The Shell’s Interpretive Cycle
The shell in UNIX acts as an interpreter between the user and the
operating system. Its role is to take commands from the user,
interpret them, and pass them to the operating system for
execution. Once the commands are executed, it displays the output to
the user. This cycle of interaction is known as the shell's interpretive
cycle.
The Shell’s Interpretive Cycle
Steps in the Shell's Interpretive Cycle
Prompting: The shell displays a prompt (e.g., $) to indicate that it is
ready to accept a command.
Reading: The user enters a command, and the shell reads the input.
Parsing: The shell breaks the command into parts (tokens) such as
the command name, options, and arguments.
The Shell’s Interpretive Cycle
Steps in the Shell's Interpretive Cycle
Substitution/Expansion: It processes variables ($VAR), wildcards (*), and
command substitutions (e.g., $(date)).
Command Lookup:
The shell identifies whether the command is:
A shell built-in (e.g., cd).
An executable file located in directories specified by $PATH.
The Shell’s Interpretive Cycle
Steps in the Shell's Interpretive Cycle
Execution: For external commands: The shell creates a new process using
fork(). The child process replaces itself with the command using exec(). For
built-in commands, the shell directly executes them.
Output: The shell receives and displays the output (or errors) of the executed
command.
Return to Prompt: After command execution, the shell returns to the prompt,
ready for the next command.
The Shell’s Interpretive Cycle
Wildcards:
Wildcards are special characters used in UNIX shell commands to
represent multiple files or directories without typing their names
explicitly. They are part of the shell’s substitution/expansion step
during the interpretive cycle.
The Shell’s Interpretive Cycle
Wildcards: The shell interprets the wildcard (*) and matches it with
filenames in the current directory that meet the pattern.
For example:
*.txt matches all files ending with .txt.
* matches all files.
a?c matches files like abc, aac (where ? matches any single character)
The Shell’s Interpretive Cycle
Common Wildcards in UNIX
Wildcard Meaning Example Matches
* Matches zero or more characters *.txt file.txt,
notes.txt, a.txt
? Matches exactly one character file?.txt file1.txt,
file2.txt
The Shell’s Interpretive Cycle
Common Wildcards in UNIX
Wildcard Meaning Example Matches
[] Matches any one character inside the brackets file[12].txt
file1.txt, file2.txt
[! ] Matches any character not in the brackets file[!1].txt
file2.txt, but not file1.txt
{} Matches a list of patterns separated by commas file{1,2}.txt
file1.txt, file2.txt
The Shell’s Interpretive Cycle
Example: Using Wildcards
Command: ls *.c
Explanation: Lists all files with the .c extension in the current directory.
Expansion: *.c → main.c, utils.c, test.c.
Execution: The shell runs: ls main.c utils.c test.c.
The Shell’s Interpretive Cycle
Example: Using Wildcards
Command: rm file[1-3].txt
Explanation: Deletes files named file1.txt, file2.txt, and file3.txt.
Command: mv data_?.csv backup/
Explanation: Moves files like data_1.csv and data_2.csv (but not data_10.csv)
to the backup/ directory.
The Shell’s Interpretive Cycle
Escaping:
In UNIX, escaping is the process of preventing the shell from
interpreting special characters (like *, ?, $, etc.) in a command. It
tells the shell to treat these characters literally, rather than using
their special meanings (e.g., as wildcards or variable expansions).
The Shell’s Interpretive Cycle
Why Escaping is Needed
The shell interprets special characters like:
* (wildcard for matching multiple files),
$ (variable expansion),
\n (newline), and many others.
If you want the shell to treat these characters as regular text, you use
escaping.
The Shell’s Interpretive Cycle
Ways to Escape Characters
Backslash (\): Place a backslash before the special character to
escape it.
Example: echo Hello\*World
Output: Hello*World
(The * is not treated as a wildcard.)
The Shell’s Interpretive Cycle
Ways to Escape Characters
Single Quotes ('): Enclose the string in single quotes to escape all
special characters.
Example: echo 'Hello $USER'
Output: Hello $USER
(The $USER is not expanded as a variable.)
The Shell’s Interpretive Cycle
Ways to Escape Characters
Double Quotes ("): Enclose the string in double quotes to escape most
special characters, except $, \, and `.
Example: echo "Hello \$USER"
Output: Hello $USER
(Here, $USER is escaped, but double quotes allow expansion by
default.)
The Shell’s Interpretive Cycle
Ways to Escape Characters
Escape Command (\): Use \ as a general escape character for
commands.
Example: touch file\ name.txt
Creates a file named file name.txt (the space is escaped).
The Shell’s Interpretive Cycle
Quoting:
Quoting in the UNIX shell is a way to control how the shell interprets
special characters (such as $, *, ?, and |) within a command. By
enclosing text in quotes, you can either allow or suppress the shell's
interpretation of these characters.
The Shell’s Interpretive Cycle
Types of Quoting
1. Single Quotes (')
Purpose: Preserves the literal meaning of everything inside the quotes. All special characters are
treated as plain text.
Usage: Protects the string from shell interpretation, including variables ($) and commands ($( )).
Example: echo 'Hello $USER'
Output: Hello $USER
(The $USER variable is not expanded.)
The Shell’s Interpretive Cycle
2. Double Quotes (")
Purpose: Allows some interpretation while preserving others. Special characters like $
(variable expansion), ` (command substitution), and \ (escape character) retain their meaning.
Wildcards (*), pipes (|), and others are treated as plain text.
Usage: Used when you want to expand variables or execute commands but treat other special
characters literally.
Example: echo "Hello $USER"
Output: Hello your_username (The $USER variable is expanded.)
echo "Today is $(date)"
Output: Today is Mon Nov 19 12:34:56 UTC 2024 (The $(date) command is executed.)
The Shell’s Interpretive Cycle
3. Backslash (\)
Purpose: Escapes a single character. Prevents the shell from interpreting the
following character.
Usage: Useful for escaping one or two characters instead of quoting an entire string.
Example: echo Hello\*World
Output: Hello*World
(The * is treated as plain text.)
The Shell’s Interpretive Cycle
Three standard files:
In UNIX, every process has three default file descriptors (or standard files) connected to
the terminal by default,
Standard Input (stdin):
File Descriptor: 0
Purpose: Accepts input data (usually from the keyboard).
Example: cat
The cat command waits for user input via stdin.
The Shell’s Interpretive Cycle
Standard Output (stdout):
File Descriptor: 1
Purpose: Displays output data (usually to the terminal screen).
Example: echo "Hello, World!"
The text Hello, World! is displayed on the terminal via stdout.
The Shell’s Interpretive Cycle
Standard Error (stderr):
File Descriptor: 2
Purpose: Displays error messages (usually to the terminal screen).
Example: ls nonexistent_file
Error message: ls: cannot access 'nonexistent_file': No such file or
directory is sent to stderr.
The Shell’s Interpretive Cycle
Redirection:
Redirection allows you to change where stdin, stdout, or stderr are
read from or written to. This is often used to store outputs or handle
errors in files.
The Shell’s Interpretive Cycle
Types of Redirection
Redirecting stdout: Syntax: command > file
Sends the standard output to a file, overwriting its contents.
Example: echo "Hello, World!" > output.txt
Writes Hello, World! to the file output.txt.
Append to the file: echo "New Line" >> output.txt
Adds New Line to the file without overwriting.
The Shell’s Interpretive Cycle
Redirecting stderr:
Syntax: command 2> file
Sends error messages to a file.
Example: ls nonexistent_file 2> errors.txt
Writes the error message to errors.txt.
The Shell’s Interpretive Cycle
Redirecting Both stdout and stderr:
Syntax: command > file 2>&1 or command &> file
Sends both outputs and errors to the same file.
Example:
ls existing_file nonexistent_file > output_and_errors.txt 2>&1
Writes both output and error messages to output_and_errors.txt.
The Shell’s Interpretive Cycle
Redirecting stdin:
Syntax: command < file
Reads input from a file instead of the keyboard.
Example: cat < input.txt
Reads and displays the contents of input.txt.
The Shell’s Interpretive Cycle
A pipe: in Unix is a powerful inter-process communication
mechanism that allows the output of one command to be used as
the input to another command. It enables you to chain multiple
commands together, creating a pipeline for processing data step by
step.
The symbol for a pipe is the vertical bar (|).
The Shell’s Interpretive Cycle
How a Pipe Works
A pipe connects the standard output (stdout) of one process to the
standard input (stdin) of another.
It allows commands to work together in a streamlined and efficient
manner without needing intermediate files.
The Shell’s Interpretive Cycle
Basic Syntax
command1 | command2 | command3
command1: Generates the output.
command2: Takes the output of command1 as input and processes it.
command3: Further processes the output of command2.
The Shell’s Interpretive Cycle
Count the Number of Lines in a File
cat file.txt | wc -l
cat file.txt: Displays the content of the file.
wc -l: Counts the number of lines.
The Shell’s Interpretive Cycle
Sort and Display Unique Words in a File
cat file.txt | tr ' ' '\n' | sort | uniq
tr ' ' '\n': Converts spaces to newlines to separate words.
sort: Sorts the words alphabetically.
uniq: Removes duplicate entries.
The Shell’s Interpretive Cycle
Common Commands Used with Pipes
grep: Search for patterns in text.
sort: Sort lines of text.
uniq: Remove duplicate lines.
wc: Count words, lines, and characters.
awk: Process and analyze text data.
sed: Perform text transformations.
The Shell’s Interpretive Cycle
The tee command: in Unix is used to read from the standard
input (stdin) and simultaneously write the output to both standard
output (stdout) and one or more files. It is especially useful for
logging or saving intermediate results in a pipeline without
interrupting the flow.
The Shell’s Interpretive Cycle
Basic Syntax
command | tee [options] file_name
command: Any command whose output you want to process.
file_name: The file where the output will be saved.
The Shell’s Interpretive Cycle
How tee Works
Input: Takes the output of a command (or any data stream).
Output:
Displays it on the screen (stdout).
Saves it to the specified file(s).
The Shell’s Interpretive Cycle
Save and Display Output Simultaneously
ls -l | tee output.txt
Action: Lists files in long format.
Result:
The output is displayed on the terminal.
The same output is saved to output.txt.
The Shell’s Interpretive Cycle
Append to a File
To append the output to an existing file instead of overwriting it:
ls -l | tee -a output.txt
-a option: Appends to output.txt instead of overwriting.
The Shell’s Interpretive Cycle
Command substitution :
in Unix is a feature that allows the output of a command to be used as
an input to another command. It is a powerful tool for automating
tasks and making scripts more dynamic by using the result of one
command directly in another.
The Shell’s Interpretive Cycle
Syntax of Command Substitution
There are two common syntaxes for command substitution:
Backticks (``):
command_output=`command`
$(...) (Preferred Modern Style):
command_output=$(command)
The Shell’s Interpretive Cycle
Examples of Command Substitution
1. Assign Command Output to a Variable
current_date=$(date)
echo "Today's date is: $current_date"
date: Generates the current date and time.
Result: Stores the output in the variable current_date and prints it.
The Shell’s Interpretive Cycle
Use Command Output as an Argument
echo "Number of files in current directory: $(ls | wc -l)"
ls: Lists files in the current directory.
wc -l: Counts the number of lines (files).
The Shell’s Interpretive Cycle
Embed Command Output in a String
echo "The current working directory is $(pwd)"
pwd: Prints the current working directory.
The Shell’s Interpretive Cycle
Nested Command Substitution
echo "Largest file: $(ls -S | head -1)"
ls -S: Lists files sorted by size (largest first).
head -1: Retrieves the first file from the sorted list.
The Shell’s Interpretive Cycle
Run Command Inside a Script
#!/bin/bash
username=$(whoami)
echo "Hello, $username! Welcome to the system."
whoami: Retrieves the current username.
Shell Programming
Ordinary Variables
These are variables created and used within the shell or a shell script.
They are:
Local to the current shell session or script.
Not accessible by child processes or other shells.
Shell Programming
Key Features
Ordinary variables are used to store temporary data.
They exist only during the execution of the current shell or script.
They do not affect the environment of other shells or programs.
Shell Programming
Syntax :
variable_name=value
Accessing an ordinary variable:
Use the $ symbol.
echo $variable_name
Shell Programming
Examples:
name="Zaheed"
echo $name
Output: Zaheed
The variable name is an ordinary variable, valid only in the current
shell.
Shell Programming
Environment Variables
Environment variables are a type of variable that:
Are global and can be accessed by child processes or subshells.
Define system-wide or shell-wide configurations.
Shell Programming
Key Features
Used to configure the behavior of the shell and system programs.
Passed from the parent process to child processes.
Examples include PATH, HOME, USER, SHELL, etc.
Shell Programming
Syntax
To create an environment variable:
export variable_name=value
Examples:
export name="Zaheed"
echo $name # Output: Zaheed
The export command makes the variable available to child processes.
Shell Programming
Feature Ordinary Variables Environment Variables
result=$(expr 20 / 4)
echo "20 / 4 = $result“
result=$(expr 10 % 3)
echo "10 % 3 = $result"
Shell Programming
String Operations
Finding String Length:
str="Hello"
length=$(expr length "$str")
echo "Length of '$str' is $length"
Shell Programming
Substring Extraction:
str="HelloWorld"
# Extract 5 characters starting from the 4th position
substr=$(expr substr "$str" 4 5)
echo "Substring: $substr"
Shell Programming
Logical Comparisons
Equality and Inequality
if [ $(expr 5 = 5) -eq 1 ]; then
echo "5 equals 5"
fi
if [ $(expr 10 != 5) -eq 1 ]; then
echo "10 is not equal to 5"
fi
Shell Programming
Greater Than, Less Than
num1=8
num2=10
if [ $(expr $num1 \< $num2) -eq 1 ]; then
echo "$num1 is less than $num2"
fi
Shell Programming
Regular Expression Matching
Match a pattern in a string:
str="hello123world"
match=$(expr "$str" : '.*\([0-9][0-9]*\)')
echo "Extracted digits: $match"
Shell Programming
The while loop in shell programming is used to execute a block of code
repeatedly as long as a given condition is true. It is often used when the
number of iterations is not known beforehand and depends on the condition
being evaluated during execution.
Shell Programming
Syntax: while [ condition ]; do
# Commands to execute
done
[ condition ]: The test condition that is checked before each iteration.
do: Indicates the start of the loop body.
done: Indicates the end of the loop body.
Shell Programming
Basic Example
#!/bin/bash
count=1
while [ $count -le 5 ]; do
echo "Count: $count"
count=$((count + 1))
done
Shell Programming
Output:
Count: 1
Count: 2
Count: 3
Count: 4
Count: 5
Shell Programming
The for loop in shell programming is used to iterate over a list of items,
performing a set of commands for each item in the list. It is versatile and
commonly used when the number of iterations or the list of items is known
beforehand.
Shell Programming
Basic Syntax
for variable in list; do
# Commands to execute
done
variable: The loop variable that takes the value of each item in the list.
list: A space-separated list of items to iterate over.
do: Indicates the start of the loop body.
done: Indicates the end of the loop body.
Shell Programming
Iterating Over a List of Strings
#!/bin/bash
for name in Alice Bob Charlie; do
echo "Hello, $name!"
done
Output:
Hello, Alice!
Hello, Bob!
Hello, Charlie!
Shell Programming
Using a Step Value
Output:
#!/bin/bash
Odd Number: 1
for i in {1..10..2}; do
Odd Number: 3
echo "Odd Number: $i"
Odd Number: 5
done
Odd Number: 7
The syntax {start..end..step}
Odd Number: 9
defines the range and step
value.
Shell Programming
Iterating Over Command Output
#!/bin/bash
for file in $(ls *.txt); do
echo "Processing file: $file"
done
This iterates over the list of .txt files in the current directory.
Shell Programming
Iterating Over Files in a Directory
#!/bin/bash
for file in /path/to/directory/*; do
echo "File: $file"
done
This lists all files and directories in /path/to/directory.
Shell Programming
The case control statement in shell programming is used to execute
different blocks of commands based on the value of a variable or expression.
It is similar to the switch statement in languages like C or Java, but with a
syntax tailored for shell scripts.
Shell Programming
Basic Example
#!/bin/bash
read -p "Enter a number (1-3): " num
case $num in
1)
echo "You chose One."
;;
Shell Programming
2)
echo "You chose Two."
;;
3)
echo "You chose Three."
;;
*)
echo "Invalid choice!"
;;
esac
Shell Programming
Input/Output:
Argument: three
set one two three four five
Argument: four
while [ $# -gt 0 ]; do
Argument: five
echo "Argument: $1"
The shift command reduces the number of
shift arguments ($#) until none remain.
done
Shell Programming
Shifting Multiple Positions
#!/bin/bash
set a b c d e f
echo "Before shift: $@"
shift 3
echo "After 3 shifts: $@"
Output: Before shift: a b c d e f
After 3 shifts: d e f