I blindly overwrote instead of carefully merging pacnews: Tty login incorrect and lightdm won't start now

Support System Updates
1

Hello!

Today i updated to 6.6.10-1 using yay. Then i was reading the arch wiki and i also ran a program suggested that replaced old files eith pacnew files. I never did that before and not sure if it is a coincidence or not. It replaced a bunch of files.

After a reboot, i cannot log in to the system at all. The boot starts with unencrypting the disk, grub and console booting as normal. It stops with several lines repeating of:

[FAILED] failed to start light display manager.

Then a [TIME] and [DEPEND] error for an external disk i do not usually have plugged in so probably isnt stopping anything.

I can get to the tty terminal but my credentials dont work. It just says “Login incorrect”. I try with different user accounts and sudo and root. Cannot proceed.

I am confident i know the passwords and tried 2 keyboards in case something was going on with hardware input.

Also tried getting in via ssh which is usually running but connection refused.

I can boot to a usb and access files, but all i had available is xubuntu so if i need a manjaro usb i need to make a new one.

I have been on my phone all evening reading but i got lost in the tabs tbh. I read other threads where people have similar problems. Somehow they are able to run commands on their system like doing pacman updates, obtaining logs, running commands. How?? How do i bypass the login to perform admin tasks?

Many of the threads mention potential driver issues, especially graphics cards. But i dont understand how to detemine this for my particular computer? Which is a thinkpad x260.

When booted to usb i tried to look in my file system to find file settings thst would correspond to what helped other people. but either i didnt find anything at all, or my existing settings were substantially different from what was discussed so not wanting to make random changes on that basis.

I see threads about pam, lightdm.conf, mhwd and various systemctls, none of which i was confident about. I tried changing to runlevel 3 for the first couple of kernels. But not sure if doing it properly as my config lines include have encryption so are much longer. Where does the “3” go? At the end or after the “rw” like in thr example?

In my pam config file i found a line approx “pam auth requisite no login so” (sorry i closed the computer i dont have it to transcribe). Could it be a problem? But i also cant login with root.

Somewhere i read something about some kind if login lock you can delete but i couldnt find a matching file on my system. Does anyone know how i got locked out? I didnt put in the wrong password or anything.

Many of the threads i read the poster is told they didnt do enough work prior to posting so this is why i made it so long. I am really stuck with not being able to get past the login. I hope you enjoyed my story but my hands hurt so i cant make it nicer with links im sorry due to only using mobile. And no logs because no login.

Any hint about what to do?

Thank you!

2

What was supposed to happen, or rather, what you where supposed to do with these files, was not replace one with the other, but merge the content of the two.
Meaning:
look at the differences and only apply what is needed while leaving the rest alone.

If you did replace the file /etc/passwd with the file /etc/passwd.pacnew for instance,
that means that you now have lost all passwords and even the user associated with them.
Same goes for /etc/shells vs /etc/shells.pacnew - but this is easier to repair.

If you don’t have a backup of all these from before you replaced them it will be pretty hard for you to rectify the situation.
It can be done, though.

at the very end of the long line, as the last parameter, separated by a space, like all the others before

2 Likes
3

Oh I see. Hmm. How up to date do the backups need to be? I probably have something but not from yesterday. I migrated sdd a few months ago and probably have clone images from that.

Should i try dropping whatever is found for etc passwd? Or could it further mangle things if it is not new enough?

I couldnt figure out how to use the merging interface, the interface was very strange and no indication on the packages webpage or man page or the arch wiki on how to use it. I have mostly use gui diff tools before but this didnt resemble the text diff tools either.

So… do i need to reinstall or should i attempt something else first? I have time right now to try to salvage my system which is a couple years old.

4

As recent as possible - but you are not likely to have changed or added or removed users very often, if at all.
So: any old copy will do.

Of course:
first compare the current one with your older copy
to be sure that this is the cause for your issue.
Then: just replace the current one with the older one.

What you want to look for is the user with the UID 1000
It is likely not even present in your current passwd file.

Check - or ask.
Because: I don’t really know - this is so far only my speculation based upon your description of the problem.

/etc/shadow is the companion file - I don’t know whether it would be affected as well.
If yes: replace both at the same time

/etc/group is also a candidate

meld is one tool - pacdiff another
there are many ways - just plain old looking and editing yourself included
There are guides/descriptions “everywhere” - here and on the Arch Wiki …

ps:
it may be easier:
/etc/shadow and /etc/gshadow and /etc/passwd and /etc/group
will have backup files with the old content
the respective names are:
/etc/shadow- and /etc/gshadow- and /etc/passwd- and /etc/group-

so you may not even need your backup - just copy the old to the new
cp /etc/shadow- /etc/shadow
for instance

5

these files in /etc/ have versions ending in a -: group, gshadow, passwd, shadow, subgid, subuid

when I compare the active version vs the *- version in meld, they are identical except for 1-2 lines at the bottom related to recent test user account and an account created by a package I installed (and IIRC actually uninstalled) that runs a service. All else appears to be as it should be including user 1000.

The most easily available backup is from May 2023 so I compared for the above files and found nothing too exciting. Everything accounted for by known changes to system:

May 2023 vs current differences

subgid, subuid

  • backup missing a bunch of lines at the end which correspond to user accounts added since May 2023

group, passwd, gshadow

  • as previous, + backup missing a bunch of lines at the end which appear to correspond to packages installed since then

shadow

  • as previous + line for my user is different which makes sense as I have changed password since that time

as for files not having a *- version but either being mentioned here or which I noticed to be potentially relevant when I was websearching, I also compared May 2023 vs current for /etc/grub.d/, /etc/lightdm/, /etc/pam.d/, /etc/plymouth/, /etc/shells, /etc/ssh/. what sticks out to me is:

  • /etc/ssh/sshd_config: my custom file is completely replaced
  • /etc/shells: shells I have installed like zsh and fish no longer present
  • throughout: lots of things having to do with authorization have been changed; but I do not know the significance

didn’t include minor difference such as modified comments:

May 2023 vs current differences

:file_folder: /etc/grub.d/

  • identical: 00_header, 10_linux, 30_os-prober, 30_uefi-firmware, 40_custom, 41_custom, 60_memtest86+, README

new: :page_facing_up: 25_bli a short shell script

:page_facing_up: 41_snapshots-btrfs: script as been changed a bunch, not going to post unless useful. I notice the variable grub_btrfs_directory has been renamed to grub_directory but that is not the only change

:page_facing_up: 20_linux_xen: the variable xen_version is renamed to entry_xen_version; I believe that is all.

:file_folder: /etc/lightdm/

  • identical: keys.conf, lightdm.conf.save, lightdm-gtk-backgroundmatch.conf, lightdm-webkit2-greeter.conf, users.conf, web-greeter.yml, Xgreeter, Xsession

  • removed: deepin/ (uninstalled), lightdm.conf.pacnew (modified Sept 2022)

  • modified:

    • :page_facing_up: lightdm-gtk-greeter.conf, slick-greeter.conf: minor config changes I made
    • :page_facing_up: lightdm.conf: the line [Seat:*] has been moved down the file
    • :page_facing_up: lightdm.conf: commented out or missing in new file:
logind-check-graphical=true
greeter-session=lightdm-gtk-greeter
greeter-show-remote-login=true
user-session=xfce

:file_folder: /etc/pam.d/

  • identical: chfn, chsh, cups, doas, groupmems, lightdm-greeter, lxdm, other, polkit-1, rlogin, rsh, runuser-l, runuser, samba, screen, su-l, sudo, system-local-login, system-remote-login, system-services, systemd-user, vlock, webdavd, xfce4-screensaver, xpra, sshd

removed: chage, chgpasswd, deepin-auth-keyboard, groupadd, groupdel, groupmod, shadow, useradd, userdel, usermod

modifed:
:page_facing_up: /etc/pam.d/chpasswd:

auth		required	pam_unix.so
account		required	pam_unix.so
session		required	pam_unix.so
password 	required 	pam_unix.so sha512 shadow

is now

password	include		system-auth

:page_facing_up: /etc/pam.d/crond:

auth      required    pam_unix.so
auth      required    pam_env.so

account   required    pam_access.so
account   required    pam_unix.so
account   required    pam_time.so

session    required   pam_loginuid.so
session   required    pam_limits.so
session   required    pam_unix.so

is now

auth       include    system-auth
account    required   pam_access.so
account    include    system-auth
session    required   pam_loginuid.so
session    include    system-auth

:page_facing_up: /etc/pam.d/lightdm and lightdm-autologin lines added:

-session    optional    pam_kwallet5.so auto_start
-auth       optional    pam_kwallet5.so

:page_facing_up: /etc/pam.d/login line removed:

auth       required     pam_securetty.so

:page_facing_up: /etc/pam.d/newusers:

auth		required	pam_unix.so
account		required	pam_unix.so
session		required	pam_unix.so
password 	required 	pam_unix.so sha512 shadow

is now

account		required	pam_permit.so
password	include		system-auth

:page_facing_up: /etc/pam.d/passwd:

password	required	pam_unix.so sha512 shadow nullok

is now

auth		include		system-auth
account		include		system-auth
password	include		system-auth

:page_facing_up: /etc/pam.d/su:

session	        required        pam_unix.so

is now

session optional pam_xauth.so

:page_facing_up: /etc/pam.d/system-auth:

password   required                    pam_unix.so          try_first_pass nullok shadow sha512

is now

password   required                    pam_unix.so          try_first_pass nullok shadow

:page_facing_up: /etc/pam.d/system-login removed line:

session    optional   pam_umask.so

:page_facing_up: /etc/pam.d/xscreensaver added line:

account include system-auth

new (of any interest?)
:page_facing_up: /etc/pam.d/cockpit
:page_facing_up: /etc/pam.d/kde
:page_facing_up: /etc/pam.d/postgresql
:page_facing_up: /etc/pam.d/remote

:file_folder: /etc/plymouth/ - identical

:page_facing_up: /etc/shells - current file current file doesn’t have shells I have installed like zsh, fish. Only bash. sh and rbash

:file_folder: /etc/ssh/

:page_facing_up: /etc/ssh/sshd_config: this file has been completed replaced; my customization such as non standard port number + x over ssh removed

modified:
:page_facing_up: /etc/ssh/moduli: completely different. Meld can’t read it. Old file has lines that start with 202207, new file’s lines start with 202310

:page_facing_up: /etc/ssh/ssh_config: new file includes the line

Include /etc/ssh/ssh_config.d/*.conf

new:
:file_folder: /etc/ssh/ssh_config.d/ - an empty directory
:file_folder: /etc/ssh/sshd_config.d/: contains 1 file, :page_facing_up: 99-archlinux.conf who’s content is:

# sshd_config defaults on Arch Linux
KbdInteractiveAuthentication no
UsePAM yes
PrintMotd no

unreadable: all /etc/ssh/*key files (makes sense)

identical: all /etc/ssh/*key.pub files, /etc/ssh/sshd_config.save

I don’t know what to make of all this? Files which were backed up are the same, but other files (like ssh configs) were not backed up and totally different.

6

This points to a problem with /etc/passwd /etc/shadow … which you might have altered somehow.

That does not appear to be the case.
But only you can know.

You mention /etc/shells being different.
I’d fix that.
If your system shell (and your users shell) is still bash, things should still work.

But they obviously don’t.

We don’t have system information.
inxi -Fazy

Look at the system logs
journalctl ...

I have no further idea at the moment.

7

ok that is fair enough! I am also puzzled; I wonder what I have done.

but still I have the original question which is how do I run things like journalctl when I am unable to login to the system? using a fresh manjaro USB it is only showing information from the current boot, which doesn’t seem useful because booting from USB works just fine. Am not really sure how useful is inxi in this context either? But here it is.

inxi -Fazy

System:
  Kernel: 6.6.10-1-MANJARO arch: x86_64 bits: 64 compiler: gcc v: 13.2.1
    clocksource: tsc available: hpet,acpi_pm
    parameters: BOOT_IMAGE=/boot/vmlinuz-x86_64 lang=en_US keytable=us tz=UTC
    misobasedir=manjaro misolabel=MANJARO_XFCE_2313 quiet
    systemd.show_status=1 splash apparmor=1 security=apparmor driver=free
    nouveau.modeset=1 i915.modeset=1 radeon.modeset=1
  Desktop: Xfce v: 4.18.1 tk: Gtk v: 3.24.36 info: xfce4-panel wm: xfwm
    v: 4.18.0 vt: 7 dm: LightDM v: 1.32.0 Distro: Manjaro Linux base: Arch Linux
Machine:
  Type: Laptop System: LENOVO product: 20F5S42N00 v: ThinkPad X260
    serial: <superuser required> Chassis: type: 10 serial: <superuser required>
  Mobo: LENOVO model: 20F5S42N00 v: SDK0J40705 WIN
    serial: <superuser required> UEFI-[Legacy]: LENOVO v: R02ET53W (1.26 )
    date: 02/20/2017
Battery:
  ID-1: BAT0 charge: 18.9 Wh (95.5%) condition: 19.8/23.2 Wh (85.5%)
    volts: 12.1 min: 11.1 model: SONY 45N1111 type: Li-poly serial: <filter>
    status: not charging
CPU:
  Info: model: Intel Core i5-6300U bits: 64 type: MT MCP arch: Skylake
    gen: core 6 level: v3 note: check built: 2015 process: Intel 14nm family: 6
    model-id: 0x4E (78) stepping: 3 microcode: 0x9E
  Topology: cpus: 1x cores: 2 tpc: 2 threads: 4 smt: enabled cache:
    L1: 128 KiB desc: d-2x32 KiB; i-2x32 KiB L2: 512 KiB desc: 2x256 KiB
    L3: 3 MiB desc: 1x3 MiB
  Speed (MHz): avg: 2900 min/max: 400/3000 scaling: driver: intel_pstate
    governor: powersave cores: 1: 2900 2: 2900 3: 2900 4: 2900 bogomips: 20004
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3
  Vulnerabilities:
  Type: gather_data_sampling status: Vulnerable: No microcode
  Type: itlb_multihit status: KVM: VMX unsupported
  Type: l1tf mitigation: PTE Inversion
  Type: mds status: Vulnerable: Clear CPU buffers attempted, no microcode;
    SMT vulnerable
  Type: meltdown mitigation: PTI
  Type: mmio_stale_data status: Vulnerable: Clear CPU buffers attempted, no
    microcode; SMT vulnerable
  Type: retbleed status: Vulnerable
  Type: spec_rstack_overflow status: Not affected
  Type: spec_store_bypass status: Vulnerable
  Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer
    sanitization
  Type: spectre_v2 mitigation: Retpolines, STIBP: disabled, RSB filling,
    PBRSB-eIBRS: Not affected
  Type: srbds status: Vulnerable: No microcode
  Type: tsx_async_abort status: Vulnerable: Clear CPU buffers attempted, no
    microcode; SMT vulnerable
Graphics:
  Device-1: Intel Skylake GT2 [HD Graphics 520] vendor: Lenovo driver: i915
    v: kernel arch: Gen-9 process: Intel 14n built: 2015-16 ports: active: eDP-1
    empty: DP-1, DP-2, HDMI-A-1, HDMI-A-2 bus-ID: 00:02.0 chip-ID: 8086:1916
    class-ID: 0300
  Device-2: Lite-On Integrated Camera driver: uvcvideo type: USB rev: 2.0
    speed: 480 Mb/s lanes: 1 mode: 2.0 bus-ID: 1-8:5 chip-ID: 04ca:7058
    class-ID: 0e02
  Display: x11 server: X.org v: 1.21.1.10 compositor: xfwm v: 4.18.0 driver:
    X: loaded: modesetting alternate: fbdev,vesa dri: iris gpu: i915
    display-ID: :0.0 screens: 1
  Screen-1: 0 s-res: 1366x768 s-size: <missing: xdpyinfo>
  Monitor-1: eDP-1 model: AU Optronics 0x106c built: 2012 res: 1366x768
    hz: 60 dpi: 126 gamma: 1.2 size: 276x155mm (10.87x6.1") diag: 317mm (12.5")
    ratio: 16:9 modes: 1366x768
  API: EGL v: 1.5 hw: drv: intel iris platforms: device: 0 drv: iris
    device: 1 drv: swrast surfaceless: drv: iris x11: drv: iris
    inactive: gbm,wayland
  API: OpenGL v: 4.6 compat-v: 4.5 vendor: intel mesa v: 23.3.3-manjaro1.1
    glx-v: 1.4 direct-render: yes renderer: Mesa Intel HD Graphics 520 (SKL GT2)
    device-ID: 8086:1916 memory: 30.05 GiB unified: yes
Audio:
  Device-1: Intel Sunrise Point-LP HD Audio vendor: Lenovo
    driver: snd_hda_intel v: kernel alternate: snd_soc_skl,snd_soc_avs
    bus-ID: 00:1f.3 chip-ID: 8086:9d70 class-ID: 0403
  API: ALSA v: k6.6.10-1-MANJARO status: kernel-api with: aoss
    type: oss-emulator tools: alsactl,alsamixer,amixer
  Server-1: JACK v: 1.9.22 status: off tools: N/A
  Server-2: PipeWire v: 1.0.0 status: active with: 1: pipewire-pulse
    status: active 2: wireplumber status: active 3: pipewire-alsa type: plugin
    tools: pactl,pw-cat,pw-cli,wpctl
Network:
  Device-1: Intel Ethernet I219-LM vendor: Lenovo driver: e1000e v: kernel
    port: N/A bus-ID: 00:1f.6 chip-ID: 8086:156f class-ID: 0200
  IF: enp0s31f6 state: down mac: <filter>
  Device-2: Intel Wireless 8260 driver: iwlwifi v: kernel pcie: gen: 1
    speed: 2.5 GT/s lanes: 1 bus-ID: 04:00.0 chip-ID: 8086:24f3 class-ID: 0280
  IF: wlp4s0 state: down mac: <filter>
Bluetooth:
  Device-1: Intel Bluetooth wireless interface driver: btusb v: 0.8 type: USB
    rev: 2.0 speed: 12 Mb/s lanes: 1 mode: 1.1 bus-ID: 1-7:3 chip-ID: 8087:0a2b
    class-ID: e001
  Report: rfkill ID: hci0 rfk-id: 1 state: up address: see --recommends
Drives:
  Local Storage: total: 1.51 TiB used: 933.5 GiB (60.4%)
  SMART Message: Unable to run smartctl. Root privileges required.
  ID-1: /dev/sda maj-min: 8:0 vendor: Timetec model: SD09 size: 953.87 GiB
    block-size: physical: 512 B logical: 512 B speed: 6.0 Gb/s tech: SSD
    serial: <filter> fw-rev: 3B0 scheme: MBR
  ID-2: /dev/sdb maj-min: 8:16 vendor: SanDisk model: USB 3.2Gen1
    size: 114.61 GiB block-size: physical: 512 B logical: 512 B type: USB
    rev: 3.2 spd: 5 Gb/s lanes: 1 mode: 3.2 gen-1x1 tech: N/A serial: <filter>
    fw-rev: 1.00 scheme: MBR
  SMART Message: Unknown USB bridge. Flash drive/Unsupported enclosure?
  ID-3: /dev/sdc maj-min: 8:32 vendor: Silicon Power model: SPCC M.2 PCIe SSD
    size: 476.94 GiB block-size: physical: 512 B logical: 512 B type: USB
    rev: 2.1 spd: 480 Mb/s lanes: 1 mode: 2.0 tech: SSD serial: <filter>
    fw-rev: 1.00 scheme: GPT
Partition:
  Message: No partition data found.
Swap:
  Alert: No swap data was found.
Sensors:
  System Temperatures: cpu: 78.0 C pch: 62.5 C mobo: N/A
  Fan Speeds (rpm): fan-1: 4531
Info:
  Processes: 205 Uptime: 10m wakeups: 0 Memory: total: 32 GiB note: est.
  available: 30.77 GiB used: 1.86 GiB (6.0%) Init: systemd v: 255
  default: graphical tool: systemctl Compilers: gcc: 13.2.1 clang: 16.0.6
  Packages: pm: pacman pkgs: 1159 libs: 321 tools: pamac pm: flatpak pkgs: 0
  Shell: Bash v: 5.2.21 running-in: xfce4-terminal inxi: 3.3.31
8

You did ask this in your first post.

If you write “init=/bin/bash” instead of the “3” you will be root without a password.
But the system will be very minimal, no services available, no network … you can’t even shut it down easily …

You can also use the installation medium (USB) to chroot into your system and inspect and repair it. (check logs, see what was recently installed, perform another update …)

No need to log in in both cases.
The latter case gives you a more functional system than the former - but you’ll still be limited to the command line.

chroot is the much better option IMO

My recommendation:
install and use “mc” (a console file manager) to make it easier to navigate around and do what would be more difficult when not very familiar with the command line.

9

ah! I forgot all about that. got totally side tracked by diffing all these files.

you are correct, I can get into the system by correctly editing the grub line with 3 at the end instead of in the middle like I had before. :smiley:

I think I’d like to try to change my password in this tty environment as a first step. Just in case that would solve it. Is there any reason not to do that like causing some kind of worse mess?

Then after if that doesn’t work I will do as you say with the chroot and… run an update? See if I have any nvidia drivers which I saw mentioned many times as a potential cause of this kind of error? I do not have any nvidia hw but when I was looking through all those files above I did notice some sort of nvidia service or user account was present so who knows.

and I will make a journalctl

11

ok fixed

that’s nice how it puts the code block into an iframe so it doesn’t go on for pages and pages when scrolling

12

I deleted my post - this inxi mentions nvidia because you booted the using the live USB.
Your actual systems Grub command line will be different.

13

It does? I can’t find it

I found nvidia in the users accounts that were listed in the /etc/*- files that I looked through earlier. Something is/was installed on my system having to do with nvidia.

edit: nvidia-persistenced but not sure if currently installed. no idea why it would ever have been.

14

grep GRUB_CMDLINE_LINUX_ /etc/default/grub
from inside your system (after chroot)
will show you your actual Grub command line

I see no Nvidia hardware present from what inxi shows. :man_shrugging:

15

Between @Nachlese’s very kind attention and various other posts, most particularly [Fix] Can’t login - black screen & chroot guide I think I am most of the way towards solving this. knock on wood

  • I had to use a bit more sudo than the above link suggests but was able to enter chroot per #5 in the instructions
  • sudo pacman -Syyu which didn’t find much
  • used runuser to switch to my usual user to run yay updates for AUR packages
  • used passwd to reset my password
  • tried to force reinstall of lightdm but not sure if i was able to do it or not
  • used mhwd to verify that I do not have any nvidia drivers installed (nor should I)

I think that is everything which may have helped. Did everything in a single session so not sure what had effect or not. The result is that when I boot into the newest kernel it still gives the same error about failing to start display manager. But now I can login via the tty and use startxfce4 to access the GUI; am no longer locked out completely. So I feel heartened that I should be able to resolve whatever issue is going on and the system is not hosed.

I am getting some error about a keyring on login to xfce so maybe it is this PAM thing or some other related issue I created by destroying the configurations.

Am going to try uninstalling and reinstalling lightdm + maybe clear the cache if needed to see if it’ll fix the original itself that way.

16

Look at the system logs.
They probably tell you why lightdm fails.

17

gkr-pam: unable to locate daemon control file - for which I only find threads of other lost people

18

If you don’t know how to interpret what is in the logs you should probably post more context.
The message you posted is not an error message (!)
and does not tell us anything.

journalctl -r
could be the easiest approach
it will list the log in reverse order
you can scroll from the most recent back to what came before …

19

Topic title edited to reflect what actually caused the issue. Please search the forum and Arch wiki for pacnew.

I don’t want to see tens or hundreds of replies in this thread because no one understands what happened. That should never happen, however it does–a lot. Let’s not waste everyone’s time.

20

Have you put them back yet?

AFAIK if you use zsh or fish as a login shell and it’s not listed in that file, it will cause a failed login.