Linux

Click to edit Performance

Master Optimization
title style
PollClick
Question 1
to edit Master title style
Rate your Current Linux Knowledge and Experience
• None
• Beginner
• Comfortable
• Basic Certification level: RHCSA/LPIC-1/Linux+/LFCS
• Advanced Certification level: RHCE/LPIC-2/LFCE
• Expert Certification level: RHCA/LPIC-3
• Beyond all that
PollClick
Question
to edit Master title style
What is your main Linux distribution?
• Red Hat / CentOS / Fedora
• Oracle
• SUSE
• Debian / Ubuntu / Mint
• Arch or some other geek distributions
PollClick
Question 3
to edit Master title style
Do you like systemd
• yes
• no
PollClick
Question 5
to edit Master title style
Which Geographical area are you from?
• India
• Asia (not India)
• Europe
• Africa
• North America
• South America
• Asia / Pacific
• Netherlands
 Linux
Click to edit Performance
Master Optimization
title style

Agenda
Agenda
Click to edit Master title style
• Using common tools to analyze performance
• Using /proc and sysctl
• Using tuned
• Control Groups
• Analyzing and Tuning I/O Performance
• Analyzing and Tuning Memory Performance
• Analyzing and Tuning Network Performance
 Linux
Click to edit Performance
Master Optimization
title style

1. Using Common Tools to

Analyze Performance
Using Common Tools
Click to edit Master title style
• Common tools do great for analyzing performance
• Just know what to look for
• load average
• zombie tasks
• cpu utilization
• memory related parameters
• swap usage
• most active processes
• Recommended: use distribution standard tools
• Sophisticated tools may provide more information but are not
always available
• Also consider using systemd-analyze
Understanding Processes and Threads
Click to edit Master title style
• Processes are the default managed entities by the Linux kernel
• Processes can use threads to implement multitasking
• Kernel threads are managed as processes, and for each kernel
routine one thread is started on each CPU
• User threads are not individually managed, but can be shown and
are scheduled independently
• The Linux kernel schedules tasks (not processes) and a task can be
either a single-threaded process, or a thread in a multi-threaded
process
• Use ps -C command -L to see specific command threads
• Use ps -efL to see threads for all processes
Understanding Zombies
Click to edit Master title style
• A process becomes a Zombie when it has completed its task, but
the parent process hasn't collected its execution status
• Zombies are already dead so they can't an don't have to be killed
• The most important disadvantage, is that Zombies occupy a PID
• To get rid of the Zombie, the parent process must collect the child
execution status
• Send SIGCHLD to the parent to ask the parent to reap the Zombie
• Kill the parent process
• When the parent is killed, the Zombie becomes an orphan and will be
adopted by the init process
Demo: killing a Zombie
Click to edit Master title style
• Run zombie from
https://github.com/sandervanvugt/performance/zombie
• Use ps aux | grep zombie and note the PID of the child as well as
the parent
• Use kill <childpid>; it will fail
• Use kill -SIGCHLD <parentpid>; it will be ignored
• Use kill <parentpid>; the Zombie will get adopted and init will reap
it after a few seconds
 Linux
Click to edit Performance
Master Optimization
title style

2. Using /proc and sysctl to

Optimize Performance
Understanding /proc/sys
Click to edit Master title style
• Hundreds of kernel tunables can be set through files in /proc/sys
• Modifications in here are run-time only
• To make them persistent, use sysctl
• /proc/sys/vm/swappiness → vm.swappiness
• /proc/sys/net/ipv6/conf/all/disable_ipv6 →
net.ipv6.conf.all.disable_ipv6
• /etc/sysctl.conf is where you apply settings, distributions may have
a specific location for the configuration files
 Linux
Click to edit Performance
Master Optimization
title style

3. Using tuned to Select

Performance Profiles
Understanding Tuned
Click to edit Master title style
• Introduced by Red Hat, also available on recent Ubuntu
• tuned process applies tuned profiles
• Use tuned-adm to manage default profiles
• Create custom profiles through files in the tuned configuration
directory
• Default profiles are in /usr/lib/tuned
• Custom profiles are in /etc/tuned
LabClick
3: Creating a Custom Tuned
to edit Master title styleProfile
• Create a custom Tuned profile with the name stress
• In this profile set all tcp_rmem and tcp_wmem buffers to 4096
• Use tuned-adm list to verify that the profile is active
 Linux
Click to edit Performance
Master Optimization
title style

4. Using Cgroups
Understanding Cgroups
Click to edit Master title style
• Cgroups places resources in controllers that represent the type of
resource, like cpu, memory and blkio
• By default, processes get an equal amount of resources
• Cgroups are used to limit the availability of resources using limits
and weights
• Cgroups currently are integrated in systemd
• Systemd uses 3 slices to identify resource types
• system for system services and daemons
• machine for virtual machines and containers
• user for user sessions
• By default, each slice has access to an equal amount of resources
Enabling Accounting
Click to edit Master title style
• Enable accounting in the [Service] section of the unit file
• CPUAccounting=true
• MemoryAccounting=true
• BlockIOAccounting=true
• Or better: enable it in /etc/systemd/system.conf
• Use drop-in files to take care of this
• e.g. the SSH service would use a drop-in
/etc/systemd/system/sshd.service.d/*.conf
• man 5 systemd.resource-control for all parameters
• CPUShares=512
• MemoryLimit=512M
• BlockIO*=
Monitoring Cgroups
Click to edit Master title style
• Use systemd-cgtop to show a top-alike overview of cgroups
• Use systemd-cgls for a complete list of all slices, cgroups and their
associated processes
Putting Commands into a Slice
Click to edit Master title style
• To put a command into a slice, you can use the systemd-run
command with the --slice= option
• systemd-run --slice=example.slice sleep 10d
• show with systemd-cgls /example.slice/<servicename>
• If the --slice option is not used, commands started with systemd-run
will be put into the system.slice
Applying Runtime Resource Limits
Click to edit Master title style
• systemctl set-property allows you to set specific properties at
runtime
• Changes are applied immediately and stored persistently
• systemctl set-property sshd.service MemoryAccounting=yes
• systemctl set-property sshd.service MemoryLimit=2048M
• cat
/sys/fs/cgroup/memory/system.slice/sshd.service/memory.limit_in_
bytes
Using Custom Slices
Click to edit Master title style
• Put a service in a custom slices using Slice=my.slice; if the slice
doesn't yet exist it will be created when the service is started
• You can also pre-create custom slices by creating a *.slice file in
/etc/systemd/system. Put the tunables in the [Slice] section
• To make a slice a child of another slice, give it the name <parent>-
<child>.slice; it will inherit all settings of the parent slice
• Note that the slice will only be created once the first process is
started within
 Linux
Click to edit Performance
Master Optimization
title style

5. Analyzing and Tuning I/O

Performance
Indicators of bad I/O
Click to edit Master title style
• High wa parameter in top
• High usage in iotop
• Slow writes
• Simple tests using dd
• Complex tests using bonnie++
I/O-schedulers
Click to edit Master title style
• New multi-queue schedulers have been introduced to allow I/O
operations to be mapped to multiple hardware or software request
queues
• mq-deadline: requests are sorted by expiration time to ensure that
operations are executed before expiration
• kyber: specific to support fast throughput on SSD
• bfq: recommended on slower I/O devices, and where interactive
system response is a high priority
• none: use on very fast I/O devices to not optimize anything
• Set in tuned profile, using:
[disk]
/sys/block/sda/queue/iosched/key=value
Testing I/O Performance
Click to edit Master title style
• Use dd for simple tests
• Use fio for more advanced tests
• fio --name=test1 --ioengine=libaio --iodpeth=1 --rw-randwrite--bs=4k -
-direct=1 --size=512M --numjobs=2 --group-reporting--
filename=/tmp/testfile
• --direct=1 bypasses cache usage
• Monitor the value for clat (used) in the output: lower is better
• To test, work with I/O scheduler parameters, for instance set
fifo_batch higher (16) or lower (1) to allow more I/O jobs to be
executed simultaneously
• Make persistent in systemd:
[sysfs]
/sys/block/sda/queue/iosched/fifo_batch=1
Fixing Bad I/O Performance
Click to edit Master title style
• Architecture: separates disk devices and SAN storage
• File System choice
• File System Journaling options
• I/O Scheduler choice
LabClick
4: Optimizing I/O Performance
to edit Master title style
• Pre-check: dd if=/dev/zero of=/bigfile bs=1M count=10000
• Use iotop while the dd command is running
• Change I/O scheduler options through sysfs and repeat the dd
command. Any changes?
• Change data= journaling options. Any changes?
• Run on a different file system. Any differences?
 Linux
Click to edit Performance
Master Optimization
title style

6. Analyzing and Tuning

Memory Performance
Understanding Memory
Click to edit Master title style
• Virtual versus Resident Memory
• Swap versus Cache
• Huge pages
• Memory overcommitting
Cache, swap and Paging
Click to edit Master title style
• Unused memory is used for caching
• Some applications need direct I/O to bypass cache
• When using direct I/O significant amounts of swap are needed to
guarantee that contiguous memory can be assigned
• Recommended to use SSD for swapping
• When under memory pressure, applications reclaim page cache
and/or move anonymous memory to swap
• when vm.swappiness is high, moving to swap is preferred
• when vm.swappiness is low, dropping page cache is preferred
• When a page from swap is needed, a major page fault is raised
Understanding OOM
Click to edit Master title style
• When out of memory, the kernel raises an OOM and a random
process is killed
• Consider using vm.panic_on_oom
• Processes with a higher /proc/PID/oom_score are more likely to get
killed
• Tune /proc/PID/oom_score_adj: 1000 is more likely, -1000 make a
process immune for the OOM killer
• /proc/PID/oom_adj is deprecated
• Systemd can be used to set this parameter:
[Service]
OOMAdjust=-17
Demo: Managing OOM Score
Click to edit Master title style
• for i in $(ls /proc/*/oom_score); do echo $(cat $i) score $i "; done
| sort -n
• ps aux grep | nnnn
• echo -17 > nnnn/oom_adj
• echo f > /proc/sysrq_trigger
Indicators of Memory Issues
Click to edit Master title style
• Swap usage
• Low available memory
• Low buffers and cache
• OOM messages in logs
• Slow disk/memory performance
• Memory leaks: use valgrind
Fixing Memory Issues
Click to edit Master title style
• Ultimately: add more RAM
• Use swap efficiently
• Use Cgroups
• Consider using huge pages
• Tune disk and/or network
LabClick
5: Huge Pages
to edit Master title style
• Use free -m to get an overview of current memory usage
• Use grep -i /proc/meminfo to get information about current
memory usage
• Use echo 200 > /proc/sys/vm/nr_hugepages. Repeat the
commands above
 Linux
Click to edit Performance
Master Optimization
title style

7. Analyzing and Tuning

Network Performance
Analyzing Network Performance
Click to edit Master title style
• Use ip -s link for packet statistics
• Use qperf to monitor bandwidth
Fixing Network Performance
Click to edit Master title style
• Work your way up in the stack
• Involve networking hardware as well!
• Consider modifying MTU
• Analyze availability of network buffers in memory
LabClick
6: Analyzing Network Performance
to edit Master title style
• On two servers, install qperf
• On server1, run qperf without any arguments
• On server2, run qperf server1 tcp_bw tcp_lat
• This is the baseline information you need to know about
• Start nc -l 2345 > bigfile on server1
• Use time cat /bigfile | nc server1 2345 and see how fast it is
• Change any parameter you want, test again and observe
• Notice that it will be hard to find a parameter that makes a difference
• echo 1 > /proc/sys/net/core/netdev_max_backlog
• tuned-adm profile stress to active the stress profile you've created
earlier
 Linux
Click to edit Performance
Master Optimization
title style

Additional Resources
Related Courses
Click to edit Master title style
• Below courses are available as live courses as well as recorded
courses
• Use the search feature at learning.oreilly.com to find them
• Linux Performance Optimization
• Linux Under the Hood
• Linux Troubleshooting