Security in UNIX &

Windows
Security in UNIX Operating Systems
Objectives
• Brief history of UNIX systems
• Description of protection systems of UNIX
• Comparison of requirements of a secure operating
system with UNIX
• Why ordinary operating systems are inherently
insecure.
• Examination of common vulnerabilities
• Types of threats that they will have to overcome.
• Windows Security as compared to UNIX
2
 History of UNIX systems
• UNIX is a multiuser operating system developed
by Dennis Ritchie and Ken Thompson at AT&T
Bell Labs
• Ken Thompson adapted some of the ideas of
Multics and constructed new system that was
named as a pun on the Multics system, UNICS
(UNIplexed Information and Computing Service),
later it was renamed as UNIX.
• UNIX remains a significant operating system
today, embodied in many systems, such as Linux,
Sun Solaris, IBM AIX, the various BSD systems,
etc.
3
 History of UNIX systems (contd..)
• While UNIX adopted many Multics principles, such
as hierarchical file systems, virtual memory, and
encrypted passwords, UNIX was far simpler.
• UNIX adopted several of the Multics security features,
such as password storage, protection ring usage, access
control lists, etc., but most were streamlined as well.
• As we will discuss later, the ordinary UNIX security
mechanisms are not capable of enforcing the
requirements of a secure operating system.
• A variety of efforts have aimed to extend or replace the
insecure mechanisms for ordinary UNIX systems with
mechanisms that may achieve the requirements of a
secure operating system
4
 UNIX SECURITY: Overview
• A running UNIX system consists of an operating system
kernel and many processes each running a program.
• A protection ring boundary isolates the UNIX kernel
from the processes.
• Each process has its own address space that defines the
memory addresses that it can access.
• Modern UNIX systems define address spaces primarily
in terms of the set of memory pages that they can access.
• UNIX uses the concept of a file for all persistent system
objects, such as secondary storage, I/O devices, network,
and inter process communication
5
 UNIX SECURITY: Overview (contd..)
• UNIX process is associated with an identity, based on the
user associated with the process, and access to files is limited
by the process’s identity.
• UNIX security aims to protect users from each other and the
system’s trusted computing base (TCB) from all users.
• Informally, the UNIX TCB consists of the kernel and several
processes that run with the identity of the privileged user,
root or super user.
• These root processes provide a variety of services, including
system boot, user authentication, administration, network
services, etc.
• Both the kernel and root processes have full system access.
• All other processes have limited access based on their
associated user’s identity.
6
 UNIX PROTECTION SYSTEM
• UNIX is a discretionary access control (DAC)
system
• The UNIX protection system defines a
transition state that describes how processes
change between protection domains.
• The labeling state is largely ad hoc.
• All UNIX resources are represented as files
• While directories are not files, they are
represented as files in the UNIX protection
state 7
UNIX PROTECTION SYSTEM (contd..)
• The protection state specifies that subjects may
perform read, write, and execute operations on files
• Files are also associated with an owner UID and an
owner GID
• The limited set of objects and operations enabled
UNIX designers to use a compressed access control
list format called UNIX mode bits, to specify the
access rights of identities to files.
• Mode bits define the rights of three types of subjects:
(1) the file owner UID
(2) the file group GID
(3) all other subjects
8
UNIX PROTECTION SYSTEM (contd..)
• First, the UNIX authorization mechanism checks
whether the process identity’s UID corresponds to the
owner UID of the file, and if so, uses the mode bits
for the owner to authorize access.
• If the process identity’s GID or supplementary groups
correspond to the file’s group GID, then the mode bits
for the group permissions are used. Otherwise, the
permissions assigned to all others are used.
• Example: UNIX mode bits are of the form (owner
bits, group bits, others bits) where each element in the
tuple consists of a read bit, a write bit, and an execute
bit. The mode bits:
• rwxr--r--
9
UNIX PROTECTION SYSTEM (contd..)
• UNIX permits users to change their files owner UID,
group GID and even mode bits, so its discretionary.
• File labeling is also discretionary.
• UNIX processes are labelled by trusted services from
a set of labels (i.e., user UIDs and group GIDs)
defined by trusted administrators, and child processes
inherit their process identity from their parent. This is
mandatory approach, so its secure, but inflexible.
• Setuid bit which is used for protection domain
transitions is a mode bit, it can be set by the file’s
owner, so it is also managed in a discretionary manner
10
 UNIX AUTHORIZATION
• The UNIX authorization mechanism controls each
process’s access to files and implements protection
domain transitions that enable a process to change its
identity.

• The authorization mechanism runs in the kernel, but it

depends on system and user processes for determining
its authorization queries and its protection state.

• UNIX authorization mechanism does not implement a

reference monitor.
11
 UNIX AUTHORIZATION (contd..)
• UNIX authorization occurs when files are opened, and the
operations allowed on the file are verified on each file
access.
• The requesting process provides the name of the file and the
operations that will be requested upon the file in the open
system call.
• If authorized, UNIX creates a file descriptor that represents
the process’s authorized access to perform future operations
on the file.
• File descriptors are stored in the kernel, and only an index
is returned to the process. Thus, file descriptors are a form
of capability.
• User processes present their file descriptor index to the
kernel when they request operations on the files that they
have opened.
12
 UNIX AUTHORIZATION (contd..)
• UNIX authorization controls traditional file
operations by mediating file open for read, write, and
execute permissions.
• However, the use of these permissions does not
always have the expected effect:
(1)these permissions and their semantics do not always
enable adequate control and
(2)some objects are not represented as files, so they
are unmediated.
• If a user has read access to a file, this is sufficient to
perform a wide-variety of operations on the file
besides reading.
13
 UNIX AUTHORIZATION (contd..)
• For example, simply via possession of a file
descriptor, a user process can perform any ad hoc
command on the file using the system calls
ioctl or fcntl, as well as read and modify file
metadata.
• Further, UNIX does not mediate all security-
sensitive objects, such as network
communications.
• Host firewalls provide some control of network
communication, but they do not restrict network
communication by process identity.
14
 UNIX AUTHORIZATION (contd..)
• The UNIX authorization mechanism depends on user-
level authentication services, such as login and sshd,
to determine the process identity.

• When a user logs in to a system, his/her processes are

assigned his/her login identity. All subsequent
processes created in this login session inherit this
identity unless there is a domain transition.

• Such user-level services also need root privileges in

order to change the identity of a process, so they run
with this special UID.
15
 UNIX AUTHORIZATION (contd..)
• However, several UNIX services need to run as root in
order to have the privileges necessary to perform their
tasks.
• These privileges include the ability to change process
identity, access system files and directories, change file
permissions, etc.
• Some of these services are critical to the correct
operation of UNIX authorization, such as sshd and
passwd, but others are not, such as inetd and ftp.
• However,a UNIX system’s trusted computing base must
include all root processes, thus risking compromise of
security critical services and the kernel itself.
16
 UNIX SECURITY ANALYSIS
If UNIX can be a secure operating system, it must
satisfy the secure operating system requirements.
However, UNIX fails to meet any of these requirements:

1. Complete Mediation: How does the reference

monitor interface ensure that all security sensitive
operations are mediated correctly?
• The UNIX reference monitor interface consists of
hooks to check access for file or inode permission on
some system calls. The UNIX reference monitor
interface authorizes access to the objects that the
kernel will use in its operations.
17
 UNIX SECURITY ANALYSIS (contd..)
• A problem is that the limited set of UNIX operations
(read, write, and execute) is not expressive enough to
control access to information.
• UNIX permits modifications to files without the need
for write permission(e.g.,fcntl).

2. Complete Mediation: Does the reference monitor

interface mediate security-sensitive operations on all
system resources?
• UNIX authorization does not provide complete
mediation of all system resources. For some objects,
such as network communications, UNIX itself
provides no authorization at all
18
 UNIX SECURITY ANALYSIS (contd..)
3. Complete Mediation: How do we verify that the
reference monitor interface provides complete
mediation?
• Since the UNIX reference monitor interface is
placed where the security-sensitive operations are
performed, it difficult to know whether all
operations have been identified and all paths have
been mediated. No specific approach has been used
to verify complete mediation.
4. Tamperproof: How does the system protect the
reference monitor, including its protection system,
from modification? 19
 UNIX SECURITY ANALYSIS (contd..)
• The reference monitor and protection system are
stored in the kernel, but this does not guarantee
tamper-protection.
• First, the protection system is discretionary, so it may
be tampered by any running process
• Second, the UNIX kernel is not as protected from
untrusted user processes as the Multics kernel is. Both
use protection rings for isolation, but the Multics
system also explicitly specifies gates for verifying the
legality of the ring transition arguments.
• While UNIX kernels often provide procedures to
verify system call arguments, such procedures may be
misplaced.

20
 UNIX SECURITY ANALYSIS (contd..)
• Finally, user-level processes have a variety of
interfaces to access and modify the kernel itself above
and beyond system calls. Ensuring that these
interfaces can only be accessed by trusted code has
become impractical.

5. Tamperproof: Does the system’s protection system

protect the trusted computing base programs?
In addition to the kernel, the UNIX TCB consists of all
root processes, including all processes run by a user
logged in as a root user. Since these processes could run
any program, guaranteeing the tamper-protection of the
TCB is not possible

21
UNIX SECURITY ANALYSIS (contd..)
6. Verifiable: What is basis for the correctness of the
system’s TCB?
Any basis for correctness in a UNIX system is informal.
The effectively unbounded size of the TCB prevents any
effective formal verification. Further, the size and
extensible nature of the kernel, makes it impractical to
verify its correctness.

7. Verifiable: Does the protection system enforce the

system’s security goals?
Verifiability enforcement of security goals is not
possible because of the lack of complete mediation and
the lack of tamper proofing.
22
 UNIX VULNERABILITIES
Here is a list of vulnerabilities, that aim to provide
some examples of the types of problems encountered
when the system design does not focus on protecting
the integrity of the trusted computing base.

Network-facing Daemons: UNIX has several root

(i.e.,TCB) processes that maintain network ports that
are open to all remote parties (e.g., sshd, ftpd,
sendmail, etc.), called network-facing daemons.

23
 UNIX VULNERABILITIES (contd..)
Rootkits:
Modern UNIX systems support extension via kernel
modules that may be loaded dynamically into the
kernel. However, a malicious or buggy module may
enable an attacker to execute code in the kernel, with
full system privileges.
A variety of malware packages, called rootkits, have
been created for taking advantage of kernel module
loading or other interfaces to the kernel available to
root processes.
24
 UNIX VULNERABILITIES (contd..)
Environment Variables
UNIX systems support environment variables, system
variables that are available to processes to convey state
across applications. One such variable is LIBPATH
whose value determines the search order for dynamic
libraries.
A common vulnerability is that an attacker can change
LIBPATH to load an attacker-provided file as a dynamic
library. Since environment variables are inherited when
a child process is created, an untrusted process can
invoke a TCB program under an untrusted environment
25
 UNIX VULNERABILITIES (contd..)
Shared Resources
• If TCB processes share resources with untrusted processes,
then they may be vulnerable to attack.
• A common problem is the sharing of the/tmp directory.
Since any process can create files in this directory, an
untrusted process is able to create files in this directory and
grant other processes, in particular a TCB process, access to
such files as well.
• If the untrusted process can guess the name of TCB
process’s /tmp file, it can create this file in advance, grant
access to the TCB process, and then have access itself to a
TCB file.
26
UNIX VULNERABILITIES (contd..)
Time-of-Check-to-Time-of-Use (TOCTTOU)
• Finally, UNIX has been prone to a variety of
attacks where untrusted processes may change the
state of the system between the time an operation
is authorized and the time that the operation is
performed.
• The UNIX file system remains susceptible to
TOCTTOU attacks because the mapping between
file names and actual file objects (inodes) can be
manipulated by the untrusted processes.
27
 WINDOWS SECURITY
• Windows operating systems also fail to meet the
requirements of a secure operating system.

• Many of security issues in windows are similar to

UNIX.

• For example, Windows also supports processes

with their own address spaces that are managed by
a ring-protected kernel.

28
 WINDOWS PROTECTION SYSTEM
• The Windows 2000 protection system ,like the
UNIX protection system, provides a discretionary
access control model for managing protection state,
object labeling, and protection domain transitions.
• Subjects in Windows are similar to subjects in
UNIX. In Windows , each process is assigned a
token that describes the process’s identity.
• Unlike UNIX, Windows objects can belong to a
number of different data types besides files.
• Infact, applications may define new datatypes, and
add them to the active directory, the hierarchical
name space for all objects known to the system.29
 WINDOWS PROTECTION SYSTEM (contd..)
• From an access control perspective, object types are
defined by their set of operations.

• Windows defines up to 30 operations per object

type vs read, write, and execute operations in the
UNIX protection state.

• The other major difference between a Windows and

UNIX protection state is that Windows supports
arbitrary access control lists(ACLs) rather than the
limited mode bits approach of UNIX.
30
 WINDOWS AUTHORIZATION
• Windows authorization queries are processed by a specific
component called the Security Reference Monitor (SRM).

• The SRM is a kernel component that takes a process token,

an object SID, and a set of operations, and it returns a
Boolean result of an authorization query.

• The SRM uses the object SID to retrieve its ACL from
which it determines the query result.

• Because of the negative permissions, the way that the SRM

processes authorization queries is more complicated than in
the UNIX case
31
WINDOWS SECURITY ANALYSIS
• Windows is effectively no more or less secure than ordinary
UNIX—they are both insecure.
• Despite the additional expressive power offered by the
Windows access control model, it also does not satisfy any of
the reference monitor guarantees either.

1. Complete Mediation: How does the reference monitor

interface ensure that all security sensitive operations are
mediated correctly?
• In Windows, mediation is provided by object managers.
Without the source code, it is difficult to know where
mediation is performed, but we would presume that object
managers would authorize the actual objects used in the
security-sensitive operations, similarly to UNIX. 32
WINDOWS SECURITY ANALYSIS (contd..)
2.Complete Mediation: Does the reference monitor interface mediate
security-sensitive operations on all system resources?

• Object managers provide an opportunity for complete mediation,

but provide no guarantee of mediation.

• Further, the set of managers may be extended, resulting in the

addition of potentially insecure object managers.

• Without a formal approach that defines what each manager does

and how it is to be secured, it will not be possible to provide a
guarantee of complete mediation.

33
WINDOWS SECURITY ANALYSIS (contd..)
3. Complete Mediation: How do we verify that the reference
monitor interface provides complete mediation?
• As for UNIX, no specific approach has been used to verify
complete mediation.

4. Tamperproof: How does the system protect the reference

monitor, including its protection system, for modification?
Windows suffers from the same problems as UNIX when it
comes to tampering. First, the protection system is
discretionary, so it may be tampered by any running process.
Untrusted user processes can modify permissions to their
user’s data arbitrarily, so enforcing security goals on user data
is not possible
34
WINDOWS SECURITY ANALYSIS (contd..)
5. Tamperproof: Does the system’s protection
system protect the trusted computing base
programs?

• The TCB of Windows system is no better

than that of UNIX. Nearly any program may
be part of the Windows TCB, and any
process running these programs can modify
other TCB programs invalidating the TCB.
35
WINDOWS SECURITY ANALYSIS (contd..)
6. Verifiable: What is basis for the
correctness of the system’s trusted
computing base?
• As for UNIX, any basis for correctness
is informal. Windows also has an
unbounded TCB and extensible kernel
system that prevent any effective formal
verification.
36
WINDOWS SECURITY ANALYSIS (contd..)
7. Verifiable: Does the protection system enforce the
system’s security goals?

• The general Windows model enables any

permission combination to be specified, but no
particular security goals are defined in the system.
Thus, it is not possible to tell whether a
systemissecure.Sincethemodelismorecomplexthant
heUNIXmodelandcanbeextended arbitrarily, this
makes verifying security even more difficult.
37
 WINDOWS VULNERABILITIES
Not surprisingly given its common limitations, Windows
suffers from the same kinds of vulnerabilities as the UNIX
system:
• The Windows Registry:
is a global, hierarchical database to store data for all programs.
When a new application is loaded it may update the registry
with application specific, such as security-sensitive
information.
• Administrator Users:
We mentioned in the Windows security evaluation that
traditionally users ran under the identity Administrator or at
least with administrative privileges enabled. The reason for
this is similar to there as on that broad access is granted to
registry entries

38
 WINDOWS VULNERABILITIES (contd..)
Enabled By Default:
• Like users and software vendors, Windows
deployments also came with full permissions and
functionality enabled. This resulted in the famous
Code Red worms which attacked the SQL server
component of the Microsoft IIS webserver.

• Subsequent versions of IIS are now “locked

down” , such that software has to be manually
enabled to be accessible.
39
 Summary
• The UNIX and Windows protection systems shows that it is
not enough just to design an operating system to enforce
security policies.
• Security enforcement must be comprehensive (i.e., mediate
completely),mandatory(i.e., tamper proof),and verifiable.
• Both UNIX and Windows originated in an environment in
which security requirements were very limited.
• For UNIX, the only security requirement was protection
from other users, and for Windows, users were assumed to
be mutually-trusted on early home computers.
• The connection of these systems to untrusted users and
malware on the Internet changed the security requirements
for such systems, but the systems did not evolve.
40