Mysqld and httpd can no longer create their directories in /run

MySQL (mariadb) and Apache cannot start now, and it seems to be because they can’t create their own directories in /run. If I create the directories myself and change the owners, I can start them manually. It breaks immediately the next boot though. I’m not sure what other info is relevant here.

mariadbd[6188]: 2022-04-17 19:36:50 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
mariadbd[6188]: 2022-04-17 19:36:50 0 [Note] Server socket created on IP: '0.0.0.0'.
mariadbd[6188]: 2022-04-17 19:36:50 0 [Note] Server socket created on IP: '::'.
mariadbd[6188]: 2022-04-17 19:36:50 0 [ERROR] Can't start server : Bind on unix socket: No such file or directory
mariadbd[6188]: 2022-04-17 19:36:50 0 [ERROR] Do you already have another server running on socket: /run/mysqld/mysqld.sock ?
mariadbd[6188]: 2022-04-17 19:36:50 0 [ERROR] Aborting
[Sun Apr 17 19:26:16.435270 2022] [core:error] [pid 673] (2)No such file or directory: AH00099: could not create /run/httpd/httpd.pid.yOWYIN
[Sun Apr 17 19:26:16.435293 2022] [core:error] [pid 673] AH00100: httpd: could not log pid to file /run/httpd/httpd.pid

Did you change the service?

sudo systemctl cat httpd.service
sudo systemctl cat mariadb.service

I didn’t, maybe something else did but they look normal to me.

# /usr/lib/systemd/system/httpd.service
[Unit]
Description=Apache Web Server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=simple
ExecStart=/usr/bin/httpd -k start -DFOREGROUND
ExecStop=/usr/bin/httpd -k graceful-stop
ExecReload=/usr/bin/httpd -k graceful
PrivateTmp=true
LimitNOFILE=infinity
KillMode=mixed

[Install]
WantedBy=multi-user.target

MariaDB has a pretty large config but I can post it too. Looks like I think it should though.

I installed it and it runs just fine. Maybe just delete the pid file?

$ LANG=C stat /run/httpd/httpd.pid                                             
  File: /run/httpd/httpd.pid
  Size: 7         	Blocks: 8          IO Block: 4096   regular file
Device: 0,23	Inode: 4430        Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-04-18 05:13:44.022258765 +0200
Modify: 2022-04-18 05:13:19.331897267 +0200
Change: 2022-04-18 05:13:19.331897267 +0200
 Birth: -

From where? The issue is there is no directory for it in /run, where I’m pretty sure the pid file would be if it was working correctly.

No, for me it looks like that apache and mysql crashed and the pid file stayed. Now you want to start it again and it is blocked because there is already a pid file. If a pid file is there, then one assumes that the progam is running.

Btw… /run dir does not survive a reboot and will be recreated on every boot.

Is this a manual start or at boot?

What does systemctl status show?

If at boot, try starting at command line?

At boot and manually starting I get this issue. As stated, if I set up the directories manually they start fine, but it only works for that time. I would have to do it manually every time if I want to use them now.

It shows the outputs in the first post, with a failed state. Do you want all the details from both?

@megavolt It is kinda crashing (fails to start with an error) but it mostly just seems like it can’t start at all. I thought it might be an issue with the permissions of the /run directory (since I can start it manually after making the directories myself), but I don’t really know what it should be anyway to check.

Did you run the initial mysql setup script?

There is no general issue with mysql or mariadb.

I run a mariadb service and it creates the pid file without problems as /run/mysqld/mysqld.sock

The webservice in my case is nginx - that pid is /run/ngins.pid - at some point I have played with apache as well - no issues - other than my lack of knowledge.

The /run tree is a volatile tmpfs tree which exist only on memory

$ mount
 ...
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
 ...

Both were set up a few months ago, and were both working correctly until now when they stopped. I’m not sure what I’m supposed to do with that code block (if anything)

It is just to show that the /run tree is a tmpfs and thus recreated on every boot.

There must be something else - but I have no idea what.

It is not a general issue - otherwise I would have faced it too - I run a web service on Manjaro using mariadb as backend - and the server was updated and restarted this morning.

I just took out some stats on it - the only thing you can use this for is to verify if it is not a general issue.

$ uname -a
Linux web 5.10.111-1-MANJARO #1 SMP PREEMPT Thu Apr 14 00:00:28 UTC 2022 x86_64 GNU/Linux
$ lsb_release -a
LSB Version:    n/a
Distributor ID: ManjaroLinux
Description:    Manjaro Linux
Release:        21.2.6
Codename:       Qonos
$ ls /run/mysqld -l
total 0
srwxrwxrwx 1 mysql mysql 0 Apr 18 08:37 mysqld.sock
$ systemctl status mysql
● mariadb.service - MariaDB 10.7.3 database server
     Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
     Active: active (running) since Mon 2022-04-18 08:37:59 CEST; 10h ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 871 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    Process: 891 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_>
    Process: 950 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
   Main PID: 924 (mariadbd)
     Status: "Taking your SQL requests now..."
      Tasks: 8 (limit: 38113)
     Memory: 112.6M
        CPU: 7.972s
     CGroup: /system.slice/mariadb.service
             └─924 /usr/bin/mariadbd

Apr 18 08:37:59 web mariadbd[924]: 2022-04-18  8:37:59 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
Apr 18 08:37:59 web mariadbd[924]: 2022-04-18  8:37:59 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
Apr 18 08:37:59 web mariadbd[924]: 2022-04-18  8:37:59 0 [Note] InnoDB: 10.7.3 started; log sequence number 945574969; transaction id 6159
Apr 18 08:37:59 web mariadbd[924]: 2022-04-18  8:37:59 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Apr 18 08:37:59 web mariadbd[924]: 2022-04-18  8:37:59 0 [Note] Server socket created on IP: '0.0.0.0'.
Apr 18 08:37:59 web mariadbd[924]: 2022-04-18  8:37:59 0 [Note] Server socket created on IP: '::'.
Apr 18 08:37:59 web mariadbd[924]: 2022-04-18  8:37:59 0 [Note] InnoDB: Buffer pool(s) load completed at 220418  8:37:59
Apr 18 08:37:59 web mariadbd[924]: 2022-04-18  8:37:59 0 [Note] /usr/bin/mariadbd: ready for connections.
Apr 18 08:37:59 web mariadbd[924]: Version: '10.7.3-MariaDB'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Arch Linux
Apr 18 08:37:59 web systemd[1]: Started MariaDB 10.7.3 database server.

Yeah this is how mine looks

$ uname -a
Linux andersmmg-82k2 5.15.32-1-MANJARO #1 SMP PREEMPT Mon Mar 28 09:16:36 UTC 2022 x86_64 GNU/Linux
$ lsb_release -a
LSB Version:    n/a
Distributor ID: ManjaroLinux
Description:    Manjaro Linux
Release:        21.2.6
Codename:       Qonos
 ls /run/mysqld -l
ls: cannot access '/run/mysqld': No such file or directory
$ systemctl status mysql
× mariadb.service - MariaDB 10.7.3 database server
     Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Sun 2022-04-17 19:36:51 MDT; 16h ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
   Main PID: 6188 (code=exited, status=1/FAILURE)
     Status: "MariaDB server is down"
        CPU: 185ms

Apr 17 19:36:50 andersmmg-82k2 mariadbd[6188]: 2022-04-17 19:36:50 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Apr 17 19:36:50 andersmmg-82k2 mariadbd[6188]: 2022-04-17 19:36:50 0 [Note] Server socket created on IP: '0.0.0.0'.
Apr 17 19:36:50 andersmmg-82k2 mariadbd[6188]: 2022-04-17 19:36:50 0 [Note] Server socket created on IP: '::'.
Apr 17 19:36:50 andersmmg-82k2 mariadbd[6188]: 2022-04-17 19:36:50 0 [ERROR] Can't start server : Bind on unix socket: No such file or directory
Apr 17 19:36:50 andersmmg-82k2 mariadbd[6188]: 2022-04-17 19:36:50 0 [ERROR] Do you already have another server running on socket: /run/mysqld/mysqld.sock ?
Apr 17 19:36:50 andersmmg-82k2 mariadbd[6188]: 2022-04-17 19:36:50 0 [ERROR] Aborting
Apr 17 19:36:51 andersmmg-82k2 mariadbd[6188]: Warning: Memory not freed: 280
Apr 17 19:36:51 andersmmg-82k2 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Apr 17 19:36:51 andersmmg-82k2 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Apr 17 19:36:51 andersmmg-82k2 systemd[1]: Failed to start MariaDB 10.7.3 database server.

Just brainstorming …

When the service fails because it cannot access the /run tree the service has a permission problem.

$ ls -l /run
 ...
drwxr-xr-x  2 mysql   mysql     60 Apr 18 08:37 mysqld
 ...

The user and group should exist as system user and system group

$ cat /etc/group | grep mysql
mysql:x:965:
$ cat /etc/passwd | grep mysql
mysql:x:965:965:MariaDB:/var/lib/mysql:/usr/bin/nologin

It must be the same listed in the service file

$ systemctl cat mysql.service
 ...
##############################################################################
## Package maintainers
##

User=mysql
Group=mysql
 ...

Logentries

$ journalctl -u mariadb

My service file has the same user and group as yours. (mysql for both)

$ cat /etc/passwd | grep mysql
mysql:x:961:961:MariaDB:/var/lib/mysql:/usr/bin/nologin
$ cat /etc/group | grep mysql
mysql:x:961:

My /run directory doesn’t have a mysql entry since the service is unable to create one.

But did the journalctl -u mariadb produce any additional messages that might help. Or just view the journal and search for the pattern.

It worked and then didn’t work, what happened in-between? Was there an upgrade or an installation of another package, or a configuration change…

Looks like there were problems with httpd too. If the journal doesn’t help, check the logs in /var/log/httpd.

Does systemctl --failed produce anything helpful? Does it just show httpd and mariadb?

Is /var/run symlinked to /run, execute ls -l /var/run.

Systemd handles temp files these days. If you execute systemd-analyze cat-config tmpfiles.d do you see

#
d /run/httpd 0755 root root -

# 
d /run/mysqld 0755 mysql mysql -
d /var/lib/mysql 0700 mysql mysql -
h /var/lib/mysql - - - - +C

The journal didn’t have much info, here it is:

Apr 20 00:23:19 andersmmg-82k2 systemd[1]: Starting MariaDB 10.7.3 database server...
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [Note] /usr/bin/mariadbd (server 10.7.3-MariaDB) starting as process 981 ...
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [Warning] WSREP: Failed to guess base node address. Set it explicitly via wsrep_node_address.
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [Warning] WSREP: Failed to guess base node address. Set it explicitly via wsrep_node_address.
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [Warning] WSREP: Guessing address for incoming client connections failed. Try setting wsrep_node_incoming_address explicitly.
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [Note] WSREP: Node addr:
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [Note] InnoDB: Number of transaction pools: 1
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [Note] InnoDB: Using Linux native AIO
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [Note] InnoDB: Completed initialization of buffer pool
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [Note] InnoDB: 128 rollback segments are active.
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [Note] InnoDB: Creating shared tablespace for temporary tables
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [Note] InnoDB: 10.7.3 started; log sequence number 659021; transaction id 1043
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [Note] InnoDB: Buffer pool(s) load completed at 220420  0:23:19
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [Note] Server socket created on IP: '0.0.0.0'.
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [Note] Server socket created on IP: '::'.
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [ERROR] Can't start server : Bind on unix socket: No such file or directory
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [ERROR] Do you already have another server running on socket: /run/mysqld/mysqld.sock ?
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: 2022-04-20  0:23:19 0 [ERROR] Aborting
Apr 20 00:23:19 andersmmg-82k2 mariadbd[981]: Warning: Memory not freed: 280
Apr 20 00:23:19 andersmmg-82k2 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Apr 20 00:23:19 andersmmg-82k2 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Apr 20 00:23:19 andersmmg-82k2 systemd[1]: Failed to start MariaDB 10.7.3 database server.

It just lists mariadb and apache, yeah.

Yes, and yes.

Randomly noticed my / directory was owned by my user, not root as it usually is. I tried to find out why this could have happened, but haven’t found a single thing, even in my command history. Changing the owner back to root (and double-checking the other root directories) fixed both of these services, as well as snapd which I had also been having issues with. Maybe someone else can find this useful too.

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.