Backspace deletes only part of a unicode character

Jazz · 20 January 2022 13:41

I just discovered a really annoying issue with my terminal.

$ locale
LANG=hr_HR.UTF-8
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=C

$ stty -a | sed -n -e 's/^.*\; erase \= //p' | cut -d\; -f1
^?

If you type any number of non-English Unicode characters, e.g. “öäüčđž” and then use your backspace key to delete those, then immediately start typing any of valid commands, the result will be quite unexpected:

$ ls -al
bash: $'\303\266\303\244\303\274\304ls': command not found

Only parts of each Unicode character are deleted due to the setting stty erase ^?.
I tried to set stty erase ^H, by using Ctrl+V+backspace, but that didn’t resolve my issue. Any ideas?

ishaan2479 · 20 January 2022 14:28

Change LC_CTYPE="C" to LC_CTYPE="hr_HR.UTF-8" and reboot:

sudo localectl set-locale LC_CTYPE="hr_HR.UTF-8"

More info:

https://unix.stackexchange.com/questions/180706/bash-ls-command-not-found-when-typing-ñ-by-mistake

Jazz · 20 January 2022 15:11

Unfortunately changing LC_CTYPE with localectl didn’t fix it.

$ sudo localectl set-locale LC_CTYPE="hr_HR.UTF-8"^C
$ localectl 
   System Locale: LANG=en_US.UTF-8
                  LC_CTYPE=hr_HR.UTF-8
                  LC_NUMERIC=hr_HR.UTF-8
                  LC_TIME=hr_HR.UTF-8
                  LC_MONETARY=hr_HR.UTF-8
                  LC_PAPER=hr_HR.UTF-8
                  LC_NAME=hr_HR.UTF-8
                  LC_ADDRESS=hr_HR.UTF-8
                  LC_TELEPHONE=hr_HR.UTF-8
                  LC_MEASUREMENT=hr_HR.UTF-8
                  LC_IDENTIFICATION=hr_HR.UTF-8
       VC Keymap: us
      X11 Layout: us
       X11 Model: pc105
$ locale   
LANG=hr_HR.UTF-8
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=C
$ locale -a
C
POSIX
en_US.utf8
hr_HR.utf8

ishaan2479 · 20 January 2022 15:33

Try adding LC_CTYPE=hr_HR.UTF-8 to /etc/locale.conf

Jazz · 20 January 2022 18:56

I didn’t have to, since my locale.conf already contained it.

$ cat /etc/locale.conf  
LANG=en_US.UTF-8
LC_CTYPE=hr_HR.UTF-8
LC_NUMERIC=hr_HR.UTF-8
LC_TIME=hr_HR.UTF-8
LC_MONETARY=hr_HR.UTF-8
LC_PAPER=hr_HR.UTF-8
LC_NAME=hr_HR.UTF-8
LC_ADDRESS=hr_HR.UTF-8
LC_TELEPHONE=hr_HR.UTF-8
LC_MEASUREMENT=hr_HR.UTF-8
LC_IDENTIFICATION=hr_HR.UTF-8

Jazz · 21 January 2022 17:44

It doesn’t help even if I place export LC_CTYPE="hr_HR.UTF-8" to my .bash_profile. My LC_CTYPE stays as C as an output of locale.

This bug is very frustrating, since every single typo with an accented character key in my command line, followed by backspace is punished with a “command not found” error.

Jazz · 18 September 2022 20:49

Anyone else could verify this issue?

In the new line type exactly this (using “[” and “]” to mark separate keystrokes):

[č] [Backspace] [l] [s] [Enter]

If your keyboard doesn’t have the “č” character, you may copy-paste it from here.

Expacted behavior: Should execute the ls command.
What actually happens: Otputs the bash: $'\304ls': command not found error.

lukewhite · 18 September 2022 21:09

Just tried this – I’m using Manjaro Gnome with Gnome Terminal 3.44.1. It works fine on my system - typing the string of characters just gives me the output of the ls command - no errors or the like.

edit: oh, and my shell is zsh 5.9

Jazz · 19 September 2022 01:20

The link you shared was a wonderful trace, since that’s exactly what’s happening in my case: “a terminal (emulator) which understands multibyte encodings (UTF-8), but at the same time - the shell doesn’t”.

Unfortunately, I couldn’t resolve my issue with the solution proposed over there. My configs below:

$ localectl
   System Locale: LANG=en_US.UTF-8
                  LC_CTYPE=hr_HR.UTF-8
                  LC_NUMERIC=hr_HR.UTF-8
                  LC_TIME=hr_HR.UTF-8
                  LC_MONETARY=hr_HR.UTF-8
                  LC_PAPER=hr_HR.UTF-8
                  LC_NAME=hr_HR.UTF-8
                  LC_ADDRESS=hr_HR.UTF-8
                  LC_TELEPHONE=hr_HR.UTF-8
                  LC_MEASUREMENT=hr_HR.UTF-8
                  LC_IDENTIFICATION=hr_HR.UTF-8
       VC Keymap: croat
      X11 Layout: hr
       X11 Model: pc105

$ locale
LANG=hr_HR.UTF-8
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=C

$ cat /etc/default/locale
LANG=en_US.UTF-8
LC_ADDRESS=hr_HR.UTF-8
LC_IDENTIFICATION=hr_HR.UTF-8
LC_MEASUREMENT=hr_HR.UTF-8
LC_MONETARY=hr_HR.UTF-8
LC_NAME=hr_HR.UTF-8
LC_NUMERIC=hr_HR.UTF-8
LC_PAPER=hr_HR.UTF-8
LC_TELEPHONE=hr_HR.UTF-8
LC_TIME=hr_HR.UTF-8

$ cat /etc/X11/xorg.conf.d/00-keyboard.conf
# Written by systemd-localed(8), read by systemd-localed and Xorg. It's
# probably wise not to edit this file manually. Use localectl(1) to
# instruct systemd-localed to update it.
Section "InputClass"
        Identifier "system-keyboard"
        MatchIsKeyboard "on"
        Option "XkbLayout" "hr"
        Option "XkbModel" "pc105"
EndSection

$ cat /etc/vconsole.conf
KEYMAP=croat
FONT=
FONT_MAP=

Any ideas on how to resolve this issue?

Jazz · 5 February 2023 18:45

I still didn’t figure this out. As far as I understood, the conflict appears when either terminal emulator OR shell does not understand multibyte encodings of the other party.

I use the default Konsole + Yakuake (under the same profile) and all the profile settings seem right:

Settings > Edit the Current profile… > General > Environment > added LANG=hr_HR.UTF-8
Settings > Edit the Current profile… > Advanced > Default character encoding > selected UTF-8

My shell seems to understand the UTF-8 as well:

$ locale -v | grep 'LANG='
LANG=hr_HR.UTF-8

$ echo -e '\xe2\x82\xac'
€

$ cat /etc/locale.conf
LANG=en_US.UTF-8
LC_CTYPE=hr_HR.UTF-8
LC_NUMERIC=hr_HR.UTF-8
LC_TIME=hr_HR.UTF-8
LC_MONETARY=hr_HR.UTF-8
LC_PAPER=hr_HR.UTF-8
LC_NAME=hr_HR.UTF-8
LC_ADDRESS=hr_HR.UTF-8
LC_TELEPHONE=hr_HR.UTF-8
LC_MEASUREMENT=hr_HR.UTF-8
LC_IDENTIFICATION=hr_HR.UTF-8

$ cat /etc/default/locale 
LANG=en_US.UTF-8
LC_ADDRESS=hr_HR.UTF-8
LC_IDENTIFICATION=hr_HR.UTF-8
LC_MEASUREMENT=hr_HR.UTF-8
LC_MONETARY=hr_HR.UTF-8
LC_NAME=hr_HR.UTF-8
LC_NUMERIC=hr_HR.UTF-8
LC_PAPER=hr_HR.UTF-8
LC_TELEPHONE=hr_HR.UTF-8
LC_TIME=hr_HR.UTF-8

However, when I type:

€ENTER

€BackspaceLSENTER

… the following still happens:

$ €
bash: $'\342\202\254': command not found
$ ls
bash: $'\342\202ls': command not found

Something still doesn’t understand multibyte encodings here.

Nachlese · 5 February 2023 19:12

I may have missed it in the thread but did you check which locales are actually enabled and then try regenerating them?

grep -v ^# /etc/locale.gen

sudo locale-gen

I just tried this and it works as it should - for me.
(Xfce with bash in xfce4-terminal)

and:

gives the same result for me:

Jazz · 5 February 2023 19:30

$ grep -v ^# /etc/locale.gen

en_US.UTF-8 UTF-8
hr_HR.UTF-8 UTF-8

$ stty -a | sed -n -e 's/^.*\; erase \= //p' | cut -d\; -f1
^?

I just searched for “backspace in terminal only deletes one byte of multibyte character stty iutf8” and found many interesting results:

Seems like this rabbit hole is a bit deeper than I expected. Just tried stty iutf8 without success.

Jazz · 5 February 2023 19:39

Could be due to kernel version as well:

$ uname -r 
5.15.91-1-MANJARO

Nachlese · 5 February 2023 19:48

I run the same (in a VM)
all works there, as well as here:

uname -r
6.1.9-zen1-1-zen

Arch (EOS)

first of your above links got me to /etc/inputrc which is in the “readline” package
but I don’t suppose you touched that

Is it terminal specific? (tried a different terminal?)

I had some weird behaviour when I installed Archlabs a couple of years ago - they used zsh and some strange configuration
which made it impossible to use mc properly.

Perhaps I’ll find that issue again and what caused it …

Jazz · 5 February 2023 20:17

I just tried my newest Linux Mint MATE as a VM, running zsh and I can’t reproduce that issue over there. I also ran bash there and same thing - all good.

❯ uname -r
5.15.0-58-generic

❯ grep -v ^# /etc/locale.gen


en_US.UTF-8 UTF-8
hr_HR.UTF-8 UTF-8

I just ran zsh on my main Manjaro - I can still reproduce the same issue, which may indicate that the issue could be really terminal related. So, I ran my last available terminal emulator in Kate:

$ basename "/"$(ps -o cmd -f -p $(cat /proc/$(echo $$)/stat | cut -d \  -f 4) | tail -1 | sed 's/ .*$//')
kate

But I could reproduce the same issue there.

qruqs · 6 February 2023 06:58

Just an idea I got while skimming this topic.
I don’t see any info mentioned about which Terminal emulator you use. Could there be a difference in how they behave, like settings etc?
Konsole I believe is standard in Plasma (it’s what I use anyway) but it can be replaced if one has other needs/interests for features that only exist in other emulators.
Just a thought.

Nachlese · 6 February 2023 10:02

No one seems to be able to reproduce the issue.

Don’t know whether @Jazz tried to use the default profile in Konsole - which is,
at least when the system was installed fairly recently,
running the zsh shell.

Perhaps something global in /etc
/etc/profile, for instance
or some change to some file in ~/.config

Most of those are auto-generated
can be removed (or backed up) and will be recreated

/etc/skel is another place to look - those are the defaults for a fresh user
and can be compared to what is currently in ~/.config

I suggested checking and regenerating the enabled locales.
Feedback was: he checked and it was ok -
but did he, just to be sure, also regenerate them? Don’t know.

Jazz · 6 February 2023 22:48

I did mention it here - I use the default profile in Konsole, Yakuake and my IDE’s terminal emulators (Kate and VSCodium).

I never changed the global settings in /etc. Speaking of ~/.config, same things happen when I sudo su which completely has its own default settings - I could always recreate this issue on Manjaro. Hence I doubt I’m the only one facing this issue, since the sample in this thread is too small.

I did. No effect.

Jazz · 6 February 2023 22:58

An interesting plot twist: I just created a new user, logged in there, opened the default Konsole and… the issue is not there! What kind of sorcery is this?

Nachlese · 6 February 2023 23:03

Not to be contrarian - just observations/facts:

as of now/currently - the default profile is read only and is using zsh
your installation is likely from before that (to me rather silly) change - so you still can have Bash without “contorsions”
(it involves creating a new profile and setting this as default …)

I totally believe you!
Why would you …

it’s not quite accurate (IMO)
sudo su -
will get root’s environment
sudo su
will not

I could not.
And, as you know from your research, similar problems are rarely reported and working advice is hard or impossible to find.

But I don’t have installed any IDE.

the kind of plot twist you might have had earlier
when you had backed up and then deleted all your settings in ~/.config
like I suggested

So:
the issue is as close to be resolved as it can get I recon …

Good on you!