I have a complicated problem, so complicated I am not sure how to troubleshoot.
My home server (which is a hypervisor) has two gigabit connections that are bonded and bridged to KVM/QEMU VMs via nm-connection-editor. This makes both the hypervisor and the VMs it contains have local static IPs in the LAN (192.168.1.0/24). It runs Manjaro KDE edition and is up-to-date with the last LTS kernel. I have been using this setup for about 4.5 years now without issues. However, lately the server started dropping from my LAN rather regularly, every 4-5 days uptime. It is not reachable by ping or ssh from LAN, and it is not discoverable by arp-scan. This also applies to all of the VMs it contains, i.e. they are not reachable or discoverable by all of the above. Other hosts on the LAN can continue to ping and ssh each other without problems. When I use the server during this downtime, it cannot ping any hosts in the LAN and reports that they are unreachable. The same response is what I get if I open a VM it contains through QEMU/spice and try pinging other hosts in the LAN. The only ones they can reach is each other, in other words, the hypervisor can reach the VMs and vice versa, as well as VMs can reach each other; however all of them are under the same hypervisor. The issue can be resolved by rebooting the server. This makes me believe that the networking/routing between the hypervisor/server and LAN breaks regularly, but I am not sure where to start to find the underlying problem causing this. Any pointers will be greatly appreciated. I am more than happy to relay output of recommended commands etc., however, with a delay, since I cannot afford to keep the server in this downtime state very long, and have to wait for it to happen again to test those commands. I am not sure if I can find clues in any log files, and would appreciate input about that as well.
Edit1: After posting this, I tried disconnecting the server/hypervisor from its network connection from network manager applet in KDE and reconnecting it, and it did resolve the issue, i.e. the server/hypervisor joined the LAN again and became discoverable as well as being able to reach other hosts in LAN. However, why should this be needed? Is this a bug at some level and how to pinpoint the problem and perhaps report it?
Edit2: Restarting the network connection for the server/hypervisor did not seem to fix the issue for the VMs. They are not reachable or discoverable from LAN and they cannot ping hosts in the LAN. Interestingly, restarting the network as described in Edit1 above broke the connection between the hypervisor and VMs, as well as between the VMs; they can no longer ping each other. Therefore it was remote from being a fix. The only thing that fixes my issues is rebooting the server/hypervisor, which is obviously not ideal to do every 4-5 days.
Please find the output of journalctl --catalog --priority=3 --boot=-2 command below (boot=-2 was when the machine had the above-described network errors).
Feb 09 07:56:11 myserver kernel: pstore: crypto_comp_decompress failed, ret = -22!
Feb 09 07:56:11 myserver kernel: pstore: crypto_comp_decompress failed, ret = -22!
Feb 09 07:56:11 myserver kernel: pstore: crypto_comp_decompress failed, ret = -22!
Feb 09 07:56:11 myserver kernel: pstore: crypto_comp_decompress failed, ret = -22!
Feb 09 07:56:11 myserver kernel: pstore: crypto_comp_decompress failed, ret = -22!
Feb 09 07:56:11 myserver kernel:
Feb 09 07:56:21 myserver libvirtd[1363]: cannot open directory '/home/hakayova/Downloads/guix-system-vm-image': No such file or directory
Feb 09 07:56:21 myserver libvirtd[1363]: internal error: Failed to autostart storage pool 'guix-system-vm-image': cannot open directory '/home/hakayova/Downloads/guix-system-vm-image': No such file or directory
Feb 09 07:56:21 myserver libvirtd[1363]: cannot open directory '/home/hakayova/Downloads/Gentoo': No such file or directory
Feb 09 07:56:21 myserver libvirtd[1363]: internal error: Failed to autostart storage pool 'Gentoo': cannot open directory '/home/hakayova/Downloads/Gentoo': No such file or directory
Feb 09 07:56:21 myserver libvirtd[1363]: cannot open directory '/video/ISOlar/rehberz/20180825/rehberz': No such file or directory
Feb 09 07:56:21 myserver libvirtd[1363]: internal error: Failed to autostart storage pool 'rehberz': cannot open directory '/video/ISOlar/rehberz/20180825/rehberz': No such file or directory
Feb 09 08:00:07 myserver sSMTP[8668]: Error setting verify location: No such file or directory
Feb 09 08:00:07 myserver sSMTP[8668]: Cannot open smtp.gmail.com:587
Feb 09 18:19:25 myserver pulseaudio[24439]: GetManagedObjects() failed: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
Feb 09 21:11:19 myserver kernel: usb 3-9-port2: Cannot enable. Maybe the USB cable is bad?
Feb 09 21:14:06 myserver kernel: usb 3-9-port2: Cannot enable. Maybe the USB cable is bad?
Feb 10 01:00:07 myserver sSMTP[86930]: Error setting verify location: No such file or directory
Feb 10 01:00:07 myserver sSMTP[86930]: Cannot open smtp.gmail.com:587
Feb 10 01:10:30 myserver sSMTP[88443]: Error setting verify location: No such file or directory
Feb 10 01:10:30 myserver sSMTP[88443]: Cannot open smtp.gmail.com:587
Feb 10 01:13:32 myserver libvirtd[1363]: Guest agent is not responding: QEMU guest agent is not connected
Feb 10 01:15:32 myserver libvirtd[1363]: invalid argument: top '/var/lib/libvirt/images/izlem.qcow2' in chain for 'vda' has no backing file
Feb 10 01:20:07 myserver sSMTP[89936]: Error setting verify location: No such file or directory
Feb 10 01:20:07 myserver sSMTP[89936]: Cannot open smtp.gmail.com:587
Feb 10 01:30:26 myserver sSMTP[91525]: Error setting verify location: No such file or directory
Feb 10 01:30:26 myserver sSMTP[91525]: Cannot open smtp.gmail.com:587
Feb 10 02:16:06 myserver sSMTP[97301]: Error setting verify location: No such file or directory
Feb 10 02:16:06 myserver sSMTP[97301]: Cannot open smtp.gmail.com:587
Feb 10 02:26:06 myserver sSMTP[97301]: Connection lost in middle of processing
Feb 10 08:00:03 myserver sSMTP[147139]: Error setting verify location: No such file or directory
Feb 10 08:00:03 myserver sSMTP[147139]: Cannot open smtp.gmail.com:587
Feb 10 18:37:46 myserver kernel: usb 3-9-port2: Cannot enable. Maybe the USB cable is bad?
Feb 11 01:00:06 myserver sSMTP[294604]: Error setting verify location: No such file or directory
Feb 11 01:00:06 myserver sSMTP[294604]: Cannot open smtp.gmail.com:587
Feb 11 01:10:27 myserver sSMTP[296233]: Error setting verify location: No such file or directory
Feb 11 01:10:27 myserver sSMTP[296233]: Cannot open smtp.gmail.com:587
Feb 11 01:13:54 myserver libvirtd[1363]: Guest agent is not responding: QEMU guest agent is not connected
Feb 11 01:15:38 myserver libvirtd[1363]: invalid argument: top '/var/lib/libvirt/images/izlem.qcow2' in chain for 'vda' has no backing file
Feb 11 01:20:08 myserver sSMTP[297738]: Error setting verify location: No such file or directory
Feb 11 01:20:08 myserver sSMTP[297738]: Cannot open smtp.gmail.com:587
Feb 11 01:30:26 myserver sSMTP[299374]: Error setting verify location: No such file or directory
Feb 11 01:30:26 myserver sSMTP[299374]: Cannot open smtp.gmail.com:587
Feb 11 02:17:27 myserver sSMTP[305682]: Error setting verify location: No such file or directory
Feb 11 02:17:27 myserver sSMTP[305682]: Cannot open smtp.gmail.com:587
Feb 11 02:27:27 myserver sSMTP[305682]: Connection lost in middle of processing
Feb 11 08:00:03 myserver sSMTP[358376]: Error setting verify location: No such file or directory
Feb 11 08:00:03 myserver sSMTP[358376]: Cannot open smtp.gmail.com:587
Feb 12 01:00:04 myserver sSMTP[510604]: Error setting verify location: No such file or directory
Feb 12 01:00:04 myserver sSMTP[510604]: Cannot open smtp.gmail.com:587
Feb 12 01:10:26 myserver sSMTP[512280]: Error setting verify location: No such file or directory
Feb 12 01:10:26 myserver sSMTP[512280]: Cannot open smtp.gmail.com:587
Feb 12 01:13:59 myserver libvirtd[1363]: Guest agent is not responding: QEMU guest agent is not connected
Feb 12 01:15:34 myserver libvirtd[1363]: invalid argument: top '/var/lib/libvirt/images/izlem.qcow2' in chain for 'vda' has no backing file
Feb 12 01:20:08 myserver sSMTP[513809]: Error setting verify location: No such file or directory
Feb 12 01:20:08 myserver sSMTP[513809]: Cannot open smtp.gmail.com:587
Feb 12 01:30:27 myserver sSMTP[515516]: Error setting verify location: No such file or directory
Feb 12 01:30:27 myserver sSMTP[515516]: Cannot open smtp.gmail.com:587
Feb 12 02:18:23 myserver sSMTP[522065]: Error setting verify location: No such file or directory
Feb 12 02:18:23 myserver sSMTP[522065]: Cannot open smtp.gmail.com:587
Feb 12 02:28:23 myserver sSMTP[522065]: Connection lost in middle of processing
Feb 12 04:05:01 myserver sSMTP[539540]: Error setting verify location: No such file or directory
Feb 12 04:05:01 myserver sSMTP[539540]: Cannot open smtp.gmail.com:587
Feb 12 08:00:02 myserver sSMTP[577519]: Error setting verify location: No such file or directory
Feb 12 08:00:02 myserver sSMTP[577519]: Cannot open smtp.gmail.com:587
Feb 12 12:59:25 myserver libvirtd[1363]: End of file while reading data: Input/output error
Feb 12 15:07:13 myserver kernel: usb 3-9-port2: Cannot enable. Maybe the USB cable is bad?
Feb 12 15:24:17 myserver kernel: Buffer I/O error on dev sdg3, logical block 16385, lost sync page write
Feb 12 15:24:17 myserver kernel: JBD2: Error -5 detected when updating journal superblock for sdg3-8.
Feb 12 15:24:17 myserver kernel: Aborting journal on device sdg3-8.
Feb 12 15:24:17 myserver kernel: Buffer I/O error on dev sdg3, logical block 16385, lost sync page write
Feb 12 15:24:17 myserver kernel: JBD2: Error -5 detected when updating journal superblock for sdg3-8.
Feb 12 15:24:17 myserver kernel: FAT-fs (sdg1): unable to read boot sector to mark fs as dirty
Feb 12 15:24:17 myserver kernel: Buffer I/O error on dev sdg2, logical block 557056, lost sync page write
Feb 12 15:24:17 myserver kernel: JBD2: Error -5 detected when updating journal superblock for sdg2-8.
Feb 12 15:24:17 myserver kernel: Aborting journal on device sdg2-8.
Feb 12 15:24:17 myserver kernel: Buffer I/O error on dev sdg2, logical block 557056, lost sync page write
Feb 12 15:24:17 myserver kernel: JBD2: Error -5 detected when updating journal superblock for sdg2-8.
Feb 12 15:24:30 myserver kernel: usb 3-9-port2: Cannot enable. Maybe the USB cable is bad?
Feb 13 01:00:07 myserver sSMTP[667292]: Error setting verify location: No such file or directory
Feb 13 01:00:07 myserver sSMTP[667292]: Cannot open smtp.gmail.com:587
Feb 13 01:10:28 myserver sSMTP[667926]: Error setting verify location: No such file or directory
Feb 13 01:10:28 myserver sSMTP[667926]: Cannot open smtp.gmail.com:587
Feb 13 01:12:58 myserver libvirtd[1363]: Guest agent is not responding: QEMU guest agent is not connected
Feb 13 01:14:39 myserver libvirtd[1363]: invalid argument: top '/var/lib/libvirt/images/izlem.qcow2' in chain for 'vda' has no backing file
Feb 13 01:20:08 myserver sSMTP[668553]: Error setting verify location: No such file or directory
Feb 13 01:20:08 myserver sSMTP[668553]: Cannot open smtp.gmail.com:587
Feb 13 01:30:01 myserver sSMTP[669130]: Error setting verify location: No such file or directory
Feb 13 01:30:01 myserver sSMTP[669130]: Cannot open smtp.gmail.com:587
Feb 13 02:20:50 myserver sSMTP[671322]: Error setting verify location: No such file or directory
Feb 13 02:20:50 myserver sSMTP[671322]: Cannot open smtp.gmail.com:587
Feb 13 02:30:50 myserver sSMTP[671322]: Connection lost in middle of processing
Feb 13 08:00:02 myserver sSMTP[688052]: Error setting verify location: No such file or directory
Feb 13 08:00:02 myserver sSMTP[688052]: Cannot open smtp.gmail.com:587
Feb 13 11:47:26 myserver kernel: bond0: (slave enp5s0): speed changed to 0 on port 1
Feb 13 13:14:26 myserver systemd[24094]: Failed to start Synchronize calendars and contacts.
░░ Subject: A start job for unit UNIT has failed
░░ Defined-By: systemd
░░ Support: https://forum.manjaro.org/c/support
░░
░░ A start job for unit UNIT has finished with a failure.
░░
░░ The job identifier is 11904 and the job result is failed.
Feb 13 13:45:01 myserver libvirtd[1363]: End of file while reading data: Input/output error
Feb 13 13:45:04 myserver pulseaudio[706209]: Lost I/O connection in module "module-gsettings"
On an unrelated note I saw the vulnerabilities listed in the output of inxi command above. I would appreciate if you can point me to the right direction to resolve those, if possible.
I am so sorry, I did not notice that remark due to not reading that section since I didn’t consider myself a total noob. Boy was I wrong! I tried fixing it, I hope it is not too late.