HTTPD Semaphore/Mutex Lock Problem

Introduction

Httpd failing due to not having enough space to make semaphore/mutex locks.

When doing systemctl restart httpd, semaphores are not destroyed properly and the locks are accumulated until it reaches the max capacity.

Lock Error

Check the error_log in /etc/httpd/logs/error_log. If this error exists then follow the solution:

[Wed Nov 25 12:27:57.636689 2020] [core:emerg] [pid 24252] (28)No space left on device: AH00023: Couldn't create the mpm-accept mutex

Solution

Do the following:

* vim /usr/lib/systemd/system/httpd.service
add this line before ExecStart: ExecStartPre=/bin/sh -c "for i in `ipcs -s | awk '/httpd/ {print $2}'`; do (ipcrm -s $i); done" //clear httpd semaphores each restart
comment this line: KillSignal=SIGKILL //This does not gracefully destroy locks
save
ipcs -s | awk -v user=apache '$3==user {system("ipcrm -s "$2)}' //clear previous httpd locks
systemctl restart httpd

Double Check

Do the following commands:

ipcs -s | grep apache | wc -l //displays number of locks used by apache
systemctl restart httpd
ipcs -s | grep apache | wc -l //again to see if locks increased unnecessarily

If the locks amount stay the same you are good to go.

Increase Semaphore Locks capacity =

vim /etc/sysctl.d/99-sysctl.conf
add this line: kernel.sem = 250 256000 32 1024
save
sysctl -p //reload rules

Check if rules applied with:

ipcs -ls
------ Semaphore Limits --------
max number of arrays = 1024
max semaphores per array = 250
max semaphores system wide = 256000
max ops per semop call = 32
semaphore max value = 32767