How to fix high CPU usage of localsearch-3?

Support GNOME
1

I’m using Gnome and until recently I used to have issues with tracker-miner-fs-3 causing high CPU usage out of nowhere all the time. I was never able to fix that problem, but according to some reports on the Internet, NodeJS applications with huge node_modules directories containing tens if not hundreds of thousands of small files can cause this. Unfortunately as a developer I have about a hundred of those on my system (constantly adding more) and could not find a way to exclude node_modules all directories in tracker-miner. So I ended up permanently disabling the service by running:

systemctl --user mask tracker-miner-fs-3.service
systemctl --user stop tracker-miner-fs-3.service

Now, after a recent update my understanding is that tracker-miner was replaced by localsearch and now I’m having CPU usage issues again, only this time with process localsearch-3 and in more regular but shorter bursts.

Do you have any suggestions on how to configure localsearch properly so that I don’t have this issue anymore?

This is my log btw:

$ journalctl --user -u localsearch-3.service
...
Dez 09 11:46:50 tvk-xps159575 systemd[1715]: localsearch-3.service: Main process exited, code=exited, status=1/FAILURE
Dez 09 11:46:50 tvk-xps159575 systemd[1715]: localsearch-3.service: Failed with result 'exit-code'.
Dez 09 11:46:50 tvk-xps159575 systemd[1715]: Failed to start Tracker file system data miner.
Dez 09 11:46:50 tvk-xps159575 systemd[1715]: localsearch-3.service: Consumed 4.632s CPU time, 1G memory peak.
Dez 09 11:46:51 tvk-xps159575 systemd[1715]: localsearch-3.service: Scheduled restart job, restart counter is at 1359.
Dez 09 11:46:51 tvk-xps159575 systemd[1715]: Starting Tracker file system data miner...
Dez 09 11:46:56 tvk-xps159575 localsearch-3[77634]: Could not create store: ambiguous column name: ROWID
Dez 09 11:46:56 tvk-xps159575 systemd[1715]: localsearch-3.service: Main process exited, code=exited, status=1/FAILURE
Dez 09 11:46:56 tvk-xps159575 systemd[1715]: localsearch-3.service: Failed with result 'exit-code'.
Dez 09 11:46:56 tvk-xps159575 systemd[1715]: Failed to start Tracker file system data miner.
Dez 09 11:46:56 tvk-xps159575 systemd[1715]: localsearch-3.service: Consumed 4.499s CPU time, 1G memory peak.
Dez 09 11:46:56 tvk-xps159575 systemd[1715]: localsearch-3.service: Scheduled restart job, restart counter is at 1360.
Dez 09 11:46:56 tvk-xps159575 systemd[1715]: Starting Tracker file system data miner...
Dez 09 11:47:02 tvk-xps159575 localsearch-3[77690]: Could not create store: ambiguous column name: ROWID
Dez 09 11:47:02 tvk-xps159575 systemd[1715]: localsearch-3.service: Main process exited, code=exited, status=1/FAILURE
Dez 09 11:47:02 tvk-xps159575 systemd[1715]: localsearch-3.service: Failed with result 'exit-code'.
Dez 09 11:47:02 tvk-xps159575 systemd[1715]: Failed to start Tracker file system data miner.
Dez 09 11:47:02 tvk-xps159575 systemd[1715]: localsearch-3.service: Consumed 4.526s CPU time, 1.1G memory peak.
Dez 09 11:47:02 tvk-xps159575 systemd[1715]: localsearch-3.service: Scheduled restart job, restart counter is at 1361.
Dez 09 11:47:02 tvk-xps159575 systemd[1715]: Starting Tracker file system data miner...
Dez 09 11:47:08 tvk-xps159575 localsearch-3[77749]: Could not create store: ambiguous column name: ROWID
Dez 09 11:47:08 tvk-xps159575 systemd[1715]: localsearch-3.service: Main process exited, code=exited, status=1/FAILURE
Dez 09 11:47:08 tvk-xps159575 systemd[1715]: localsearch-3.service: Failed with result 'exit-code'.
Dez 09 11:47:08 tvk-xps159575 systemd[1715]: Failed to start Tracker file system data miner.
Dez 09 11:47:08 tvk-xps159575 systemd[1715]: localsearch-3.service: Consumed 4.581s CPU time, 1.1G memory peak.
Dez 09 11:47:08 tvk-xps159575 systemd[1715]: localsearch-3.service: Scheduled restart job, restart counter is at 1362.
Dez 09 11:47:08 tvk-xps159575 systemd[1715]: Starting Tracker file system data miner...
Dez 09 11:47:14 tvk-xps159575 localsearch-3[77807]: Could not create store: ambiguous column name: ROWID
Dez 09 11:47:14 tvk-xps159575 systemd[1715]: localsearch-3.service: Main process exited, code=exited, status=1/FAILURE
Dez 09 11:47:14 tvk-xps159575 systemd[1715]: localsearch-3.service: Failed with result 'exit-code'.
Dez 09 11:47:14 tvk-xps159575 systemd[1715]: Failed to start Tracker file system data miner.
Dez 09 11:47:14 tvk-xps159575 systemd[1715]: localsearch-3.service: Consumed 4.839s CPU time, 1G memory peak.
Dez 09 11:47:14 tvk-xps159575 systemd[1715]: localsearch-3.service: Scheduled restart job, restart counter is at 1363.
Dez 09 11:47:14 tvk-xps159575 systemd[1715]: Starting Tracker file system data miner...

I could of course do systemctl --user mask localsearch-3.service, hoping that this doesn’t cause new issues, but I’d really like to just actually fix the underlying problem.

2

Hi @0x0,

See:

https://wiki.archlinux.org/title/GNOME

Specifically:

5.1.8: Search

[…]
localsearch does not automatically recurse into all directories under the user’s home directory, so you may need to add custom paths via the Search > Search locations menu item. To exclude a directory from the indexing, create an empty .nomedia file.
[…]

Hope this helps!

3

Something like:

touch /path/to/node_modules/.nomedia

It’s a shame this doesn’t work in a way similar to .hidden files; then it might have been possible to list all those paths in one file.

1 Like
4

I was looking for exactly this! But ALAS…

1 Like
5

Maybe it’s an undocumented feature :smile_cat: :man_shrugging:

1 Like
6

Thank you, it does help to an extend, but since the node_modules directories are generated automatically by package managers like npm and yarn, it is not good practice to modify them (as in adding a .nomedia file). Some projects explicitly delete their node_modules every time a new build is created. It would be a tedious task to always keep up with adding .nomedia files to all the node_modules directories, especially when it comes to those projects that I do not maintain myself.

7

Other than telling you to look here:

It can be configured with the Search menu item in Settings. It is started automatically by gnome-session when the user logs in.

…that’s all I can offer you, sorry.

Edit:

BUT! This might help:

https://stackoverflow.com/a/64186312

https://www.yld.io/blog/using-npm-pre-and-post-hooks

Have the hook ensure the file is there…

8

A possible way around that might be to place a .nomedia file in the root of your projects directory, for example;

/projects/.nomedia
/projects/app/appname/
/projects/web/sitename/

I don’t see any inherent problem with this approach unless you have a need to search the project subdirectories themselves. Of course, this presumes that the presence of the .nomedia file has a recursive effect; of this, I’m uncertain.

1 Like
9

Pre/Post install hooks would have to be added to every single project and injecting the .nomedia in general would in some cases break reproducible builds.

Adding .nomedia to my projects directory would not suffice for me because I also have NodeJS projects in other locations including inside of a backups directory that I need to be searchable and includes a lot of old backups with horrible folder structures.

It would be really nice if I could simply tell it to ignore all folders that are named “node_modules” no matter where they are located.

10

Maybe this can give you a clue:

https://askubuntu.com/a/1341997

11

Yes, that would be nice.

12

I see, so I guess I could add node_modules here:

image
image1070×703 46.7 KB

But do we have any reason to believe that localsearch would read the configuration of tracker-miner? I couldn’t find a specific entry to localsearch unfortunately.

13

:man_shrugging:

Best way to know: test and find out.

14

It seems like it is already able to identify directories containing a node_modules directory:

$ /usr/lib/localsearch-3 --eligible /home/tvk/Documents/test
Data object “/home/tvk/Documents/test” currently exists
  File is NOT eligible to be indexed (not an indexed folder)

I get the same result for all directories that I tested that include a node_modules dir. And It doesn’t matter if I have set ‘node_modules’ in the mentioned dconf option or not.

So I guess the reason is a different one after all.

I read through the man page and then ran the following to debug the issue:

$ systemctl --user mask localsearch-3.service
$ systemctl --user stop localsearch-3.service
$ export G_MESSAGES_DEBUG=all
$ export TRACKER_DEBUG=config,miner-fs-events,monitors,statistics,status
$ /usr/lib/localsearch-3
Tracker-Message: 17:16:23.418: Set scheduler policy to SCHED_IDLE
Tracker-Message: 17:16:23.418: Setting priority nice level to 19
(localsearch-3:212665): GLib-GIO-DEBUG: 17:16:23.441: Using cross-namespace EXTERNAL authentication (this will deadlock if server is GDBus < 2.73.3)
(localsearch-3:212665): GLib-GIO-DEBUG: 17:16:23.442: _g_io_module_get_default: Found default implementation gvfs (GDaemonVfs) for ‘gio-vfs’
Tracker-Message: 17:16:23.457: Mount monitors set up for to watch for added, removed and pre-unmounts...
(localsearch-3:212665): Tracker-DEBUG: 17:16:23.458: No mounts found to iterate
(localsearch-3:212665): GLib-GIO-DEBUG: 17:16:23.460: _g_io_module_get_default: Found default implementation dconf (DConfSettingsBackend) for ‘gsettings-backend’
(localsearch-3:212665): dconf-DEBUG: 17:16:23.460: watch_fast: "/org/freedesktop/tracker/extract/" (establishing: 0, active: 0)
(localsearch-3:212665): GLib-GIO-DEBUG: 17:16:23.461: Using cross-namespace EXTERNAL authentication (this will deadlock if server is GDBus < 2.73.3)
(localsearch-3:212665): dconf-DEBUG: 17:16:23.462: watch_established: "/org/freedesktop/tracker/extract/" (establishing: 1)
(localsearch-3:212665): dconf-DEBUG: 17:16:23.463: watch_fast: "/org/freedesktop/tracker/miner/files/" (establishing: 0, active: 0)
(localsearch-3:212665): dconf-DEBUG: 17:16:23.463: watch_established: "/org/freedesktop/tracker/miner/files/" (establishing: 1)
Tracker-Message: 17:16:23.464: General options:
Tracker-Message: 17:16:23.464:   Initial Sleep  ........................  15
Tracker-Message: 17:16:23.464: Indexer options:
Tracker-Message: 17:16:23.464:   Throttle level  .......................  0
Tracker-Message: 17:16:23.464:   Indexing while on battery  ............  yes (first time only = yes)
Tracker-Message: 17:16:23.464:   Low disk space limit  .................  Disabled
(localsearch-3:212665): Tracker-DEBUG: 17:16:23.464: tracker-miner-fs-3 running as org.freedesktop.LocalSearch3
(localsearch-3:212665): dconf-DEBUG: 17:16:23.464: watch_fast: "/org/freedesktop/tracker/fts/" (establishing: 0, active: 0)
(localsearch-3:212665): dconf-DEBUG: 17:16:23.464: watch_established: "/org/freedesktop/tracker/fts/" (establishing: 1)
(localsearch-3:212665): Tracker-DEBUG: 17:16:28.944: Current and DB locales match: 'en_US.UTF-8'

(localsearch-3:212665): Tracker-CRITICAL **: 17:16:30.719: Could not create store: ambiguous column name: ROWID

Then I tried to remove the filesystem indexer db as well as the RSS indexer db which can be done quite easily like this:

$ localsearch reset -sr 
(tracker reset:214015): GLib-GIO-DEBUG: 17:23:49.771: Using cross-namespace EXTERNAL authentication (this will deadlock if server is GDBus < 2.73.3)
Found 0 PIDs…
(tracker reset:214015): GLib-GIO-DEBUG: 17:23:49.773: Using cross-namespace EXTERNAL authentication (this will deadlock if server is GDBus < 2.73.3)
(tracker reset:214015): GLib-GIO-DEBUG: 17:23:49.775: _g_io_module_get_default: Found default implementation gvfs (GDaemonVfs) for ‘gio-vfs’

And unmasked the service again:

$ systemctl --user unmask localsearch-3.service

Now it seems to work:

$ journalctl --user --follow -u localsearch-3.service
Dez 09 17:23:56 tvk-xps159575 systemd[1715]: Starting Tracker file system data miner...
Dez 09 17:23:56 tvk-xps159575 systemd[1715]: Started Tracker file system data miner.

Now it idles at 0.0% and I haven’t seen it spike yet.

2 Likes