MhdAli

Restore HAProxy config from memory

2017-05-30T16:00:00+00:00

Why?

We had corrupted haproxy config file on disk but the running config was in a correct state.

HAProxy was working fine because the way haproxy works it read the config file from disk and import it into the memory.

So we tried to restore the running config from memory.

Here how we did it…

NOTE: If you’re haproxy inside docker try to execute above commands inside docker container.

Find HAProxy memory addresses

We need to find haproxy PID first

bash-4.3# ps axjf|grep haproxy
      6     249       6       6 ?             -1 S        0   0:00  \_ /usr/local/sbin/haproxy -p /run/haproxy.pid -f /usr/local/etc/haproxy/nl-haproxy.cfg -Ds -sf 248
    249     250     250     250 ?             -1 Ss       0   0:00      \_ /usr/local/sbin/haproxy -p /run/haproxy.pid -f /usr/local/etc/haproxy/nl-haproxy.cfg -Ds -sf 248

HAproxy uses 249 PID. Then let’s cat memory map of haproxy

bash-4.3# cat /proc/249/maps
7f0abbbf4000-7f0abc014000 rw-p 00000000 00:00 0
7f0abc014000-7f0abc26b000 r-xp 00000000 fd:01 394011                     /usr/lib/libpcre.so.1.2.6
7f0abc26b000-7f0abc26c000 r--p 00057000 fd:01 394011                     /usr/lib/libpcre.so.1.2.6
7f0abc26c000-7f0abc26d000 rw-p 00058000 fd:01 394011                     /usr/lib/libpcre.so.1.2.6
7f0abc26d000-7f0abc46e000 r-xp 00000000 fd:01 394013                     /usr/lib/libpcreposix.so.0.0.3
7f0abc46e000-7f0abc46f000 r--p 00001000 fd:01 394013                     /usr/lib/libpcreposix.so.0.0.3
7f0abc46f000-7f0abc470000 rw-p 00002000 fd:01 394013                     /usr/lib/libpcreposix.so.0.0.3
7f0abc470000-7f0abc862000 r-xp 00000000 fd:01 393376                     /lib/libcrypto.so.1.0.0
7f0abc862000-7f0abc87e000 r--p 001f2000 fd:01 393376                     /lib/libcrypto.so.1.0.0
7f0abc87e000-7f0abc889000 rw-p 0020e000 fd:01 393376                     /lib/libcrypto.so.1.0.0
7f0abc889000-7f0abc88d000 rw-p 00000000 00:00 0
7f0abc88d000-7f0abcaeb000 r-xp 00000000 fd:01 393377                     /lib/libssl.so.1.0.0
7f0abcaeb000-7f0abcaf0000 r--p 0005e000 fd:01 393377                     /lib/libssl.so.1.0.0
7f0abcaf0000-7f0abcaf6000 rw-p 00063000 fd:01 393377                     /lib/libssl.so.1.0.0
7f0abcaf6000-7f0abcd0a000 r-xp 00000000 fd:01 393379                     /lib/libz.so.1.2.8
7f0abcd0a000-7f0abcd0b000 r--p 00014000 fd:01 393379                     /lib/libz.so.1.2.8
7f0abcd0b000-7f0abcd0c000 rw-p 00015000 fd:01 393379                     /lib/libz.so.1.2.8
7f0abcd0c000-7f0abcd93000 r-xp 00000000 fd:01 393374                     /lib/ld-musl-x86_64.so.1
7f0abcf82000-7f0abcf92000 rwxp 00000000 00:00 0
7f0abcf92000-7f0abcf93000 r--s 00000000 fd:01 393326                     /etc/localtime
7f0abcf93000-7f0abcf94000 r--p 00087000 fd:01 393374                     /lib/ld-musl-x86_64.so.1
7f0abcf94000-7f0abcf95000 rw-p 00088000 fd:01 393374                     /lib/ld-musl-x86_64.so.1
7f0abcf95000-7f0abcf98000 rw-p 00000000 00:00 0
7f0abcf98000-7f0abd079000 r-xp 00000000 fd:01 394026                     /usr/local/sbin/haproxy
7f0abd279000-7f0abd27c000 r--p 000e1000 fd:01 394026                     /usr/local/sbin/haproxy
7f0abd27c000-7f0abd286000 rw-p 000e4000 fd:01 394026                     /usr/local/sbin/haproxy
7f0abd286000-7f0abd293000 rw-p 00000000 00:00 0
7f0abe0b2000-7f0abe26c000 rw-p 00000000 00:00 0                          [heap]
7fff0f12b000-7fff0f140000 rw-p 00000000 00:00 0                          [stack]
7fff0f1ff000-7fff0f200000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

We only need to dump the one that has write permission, ex: rw-p. Memory address is the first column, ex: 7f0abbbf4000-7f0abc014000

dump memory

After we know the memory address that we need to dump, let’s dump using gdb tool.

gdb --pid 249
GNU gdb (GDB) 7.10.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-alpine-linux-musl".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 249
Reading symbols from /usr/local/sbin/haproxy...done.
Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done.
Reading symbols from /lib/libssl.so.1.0.0...(no debugging symbols found)...done.
Reading symbols from /lib/libcrypto.so.1.0.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libpcreposix.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libpcre.so.1...(no debugging symbols found)...done.
Reading symbols from /lib/ld-musl-x86_64.so.1...(no debugging symbols found)...done.
0x00007f0abcd61cb2 in ?? () from /lib/ld-musl-x86_64.so.1
(gdb)

Now we need to dump memory

(gdb) dump memory /tmp/7f0abc889000-7f0abc88d000 0x7f0abc889000 0x7f0abc88d000

Automate gdb command

Because so many memory addresses and we don’t know which one is the address that hold haproxy configuration so we had to use this script: https://gist.github.com/gabrielgrant/1235659

bash-4.3# python dumpmaps.py 249
dump memory 7f0abbbf4000 0x7f0abbbf4000 0x7f0abc014000
dump memory 7f0abc889000 0x7f0abc889000 0x7f0abc88d000
dump memory 7f0abcf82000 0x7f0abcf82000 0x7f0abcf92000
dump memory 7f0abcf95000 0x7f0abcf95000 0x7f0abcf98000
dump memory 7f0abd286000 0x7f0abd286000 0x7f0abd293000
dump memory 7f0abe0b2000 0x7f0abe0b2000 0x7f0abe26c000
dump memory 7fff0f12b000 0x7fff0f12b000 0x7fff0f140000
dump memory 7fff0f1ff000 0x7fff0f1ff000 0x7fff0f200000
dump memory ffffffffff600000 0xffffffffff600000 0xffffffffff601000


Run the commands above with `gdb --pid 249`

Read the output file

Use strings command to read the binary file

strings 7f0abc889000

[TIL] Centos 7 with MySQL 5.6 open file limit

2016-03-07T08:47:31+00:00

What?

Today I tried to upgrade MySQL 5.6.27 to 5.6.29 on one of our database servers, that running on CentOS version 7.

However after I upgraded the package I started seeing this error many times:

[ERROR] /usr/sbin/mysqld: Can't open file: '<TABLE_NAME>' (errno: 24 - Too many open files)

Why?

So even if you go and extend the limit on /etc/security/limits.conf like this one below it wont work

mysql  soft  nofile  49152
mysql  hard  nofile  65536

Because CentOS 7 using systemd that came to me that it cause this issue, because systemd control open files as well!!

How to fix?

So I went to /etc/systemd/system/mysql.service and added this line

LimitNOFILE=40000

Then refresh systemd with

systemctl daemon-reload

and make sure also MySQL restarted as well

systemctl restart mysqld

Final tip

This issue should sorted by MySQL team because the default package doesn’t come with a bigger number of open files.

perp openfile limits

2015-05-21T00:00:00+00:00

If you’re using perp to control your processes instead of systemd, upstart, xinit …etc, you may hit open file limitation issue, today post is to descripe more deep into this issue and how to solve it.

##Explaination

CentOS 6.x using upstart as it’s init daemon, so whenever you install perp you need to configure it to work under upstart. check [] for more info how to implement perp with upstart.

However after we implement our perp setup and configure it to control our service by using runtool command to pass the service arguments and options, in this case the service is logstash with emabaded elasticsearch, after we run it with perp we start facing too many opened files issue.

Our system already has high number of big files configured on it on /etc/security/limits.conf file, by these two lines:

root soft nofile 48000
root hard nofile 48000

BUT the problem still appears in logstash logs.

Checking process limits file show different figure than the one mentioned in limits.conf file:

# cat /proc/$(pgrep java)/limits | grep "open files"
Max open files            4096               4096               files

##Solving

###runtool

Checking runtool manual pages to find about -R option, quoting man pages:

-R rlims
  runlimit(8).  Sets up the soft resource limits for program where rlims is given as a single contiguous string in the form:

    <rlim> = <value> [: ...]

  Where:

  rlim:  single character in the set ‘a’, ‘c’, ‘d’, ‘f’, ‘m’, ‘o’, ‘p’, ‘r’, ‘s’, or ‘t’.

  ‘=’:   the literal character ‘=’.

  value: A  positive  numeric  value for the resource limit, or the special character ‘!’ used to specify ‘‘unlimited’’, that is, sets the soft limit equal to the
         current hard limit.

We configure runtool with -R o:999999 option to increase open files limit to it’s maximum.

BUT after doing so noting changed, too many opened files still in there, and process limits file still having 4096 limits of open files.

# cat /proc/$(pgrep java)/limits | grep "open files"
Max open files            4096               4096               files

###upstart

Since Upstart is controlling perp it could be open file limitation coming from it, adding the following line to our init perp.conf file SOLVE our issue.

limit nofile 999999 999999

Now checking logstash process limits gives us.

# cat /proc/$(pgrep java)/limits | grep "open files"
Max open files            999999               999999               files

Finally no more too many opened files issue.

Strace output

2015-02-25T00:00:00+00:00

Every Sysadmin knows about strace command and we’re using it in daily basis, but never come to me once to redirect the output to different file for more analyzing “you know strace output it’s tremendously fast”, so I thought why not use bash redirection

$ sudo strace -p 4148 > /tmp/strace.out

But it didn’t work still output appeared in my screen, checking strace man page will fine the answer which is use -o option, for example

$ sudo strace -p 4148 -o /tmp/strace.out 
Process 4148 attached

checking around finding out that when you run the strace command it’ll run another strace process as child of it, so apparently the second process is the one who send the output to our stdout, so the only option is to use -o option.

 5110  5301  5301  5110 pts/4     5850 S     1000   0:00      \_ bash
 5301  5850  5850  5110 pts/4     5850 S+       0   0:00          \_ sudo strace -p 4148 -o /tmp/strace.out
 5850  5851  5850  5110 pts/4     5850 S+       0   0:00              \_ strace -p 4148 -o /tmp/strace.out

Welcome to my blog

2015-01-23T07:28:56+00:00

I’d like to welcome you on my personal blog, hope you’ll find what you’re looking for.

If you want to contact you can find my email in my details, please don’t hesitate to drop email by.

Wish you a happy day and good luck.

Thank you