Testers needed: Manjaro Data Donor

Manjaro Development QA
1

I worked the last few days on a new Manjaro project: Manjaro Data Donor - short MDD.

It is a way for us to gather a few usage statistics about Manjaro.

The motivation for that at the start was to improve our user counting. Until now what has been done, was counting systems via ping.manjaro.org. These pings are sent from Manjaro systems via the NetworkManager.

There were some problems with this approach though:

  • Individual systems were only distinguished on basis of IP address. This doesn’t allow statistics over time and systems behind the same NAT are counted as one. Also one needs to store the IP address at least for a short period of time. The analysis software that was used for that was Matomo, and they promise IP addresses are masked, but we still had to rely on this promise.
  • Matomo is a rather bulky tool and wasn’t really made for system telemetry. It is meant for website analysis. The setup therefore was kind of hacky while the results rather meager and the data was only available to few people.
  • Using NetworkManager pings to check the online status for user counting is acceptable but also not what it was meant for, nor was it communicated as such. I think it’s better to be explicit and transparent about these kind of things.

So I wanted to improve upon that for quite some time now. MDD is now the tool for that and some more as it will also provide interesting hardware and environment statistics about the systems that Manjaro is being used on.

How to Help

You can install the tool simply as a package: sudo pacman -S mdd

It’s a simple Python script. You can check out the source code on GitHub. For the actual data retrieval it mostly uses the excellent inxi internally. Shoutout to @h2-1!

If you can, please try it out and tell me about any bugs you encounter. From preliminary tests we did, it works most often fine on x86, but ARM is not yet handled correctly.

Your data will be sent to a ClickHouse database I deployed on one of our Hetzner servers (Nuremberg, Germany). After the testing is done I’ll delete all data again.

Before sending the data, you can dry-run the tool to see what would be sent: mdd --dry-run. If you’re fine with the data being transmitted then just run MDD again, this time without any arguments to submit the data: mdd

If you encounter bugs, then run mdd --log DEBUG to get additional logs in your terminal.

You can study visualization of the data we received at: Manjaro Metrics

Next Steps

In the next few days we’ll do some more testing and if results are positive, I plan on installing it on all Manjaro systems and adding a systemd service to submit the data automatically.

As a reminder: Right now you have to install MDD manually and there is no systemd service yet.

With this systemd service later in place, sending the hardware data with MDD will be opt-out because I believe, if you do opt-in, the data you gather will be so heavily skewed you can just leave it be.

Let me know what you think. I know telemetry is a contentious subject, but we need at least some data about how Manjaro is being used by so many people around the world in order to show that the project has a future and also to plan for that future.

11 Likes
2

I’ll definitely be checking it out once my mirrors are synced. (I believe that’s why the address can’t be found…)

3

If nothing else, interesting because I couldn’t remember how long ago I first installed Manjaro. Now I know: 6 years ago.

4 Likes
4

Another way:

stat / | grep -i "birth:" | awk '{print $2}'
1 Like
5

Yea, mdd does it very similar:

1 Like
6

Works a treat. Went through and I don’t think it’s a lot of information, or a breach of privacy. So I think it’s perfectly, amazingly wonderful and very refreshing. Submitted and I hope it’s as cool for everyone else as it’s for me.

4 Likes
7

Hi,

The sending data are looking like the KDE telemetric, done on x86, not tested on ARM

[nls@lap ~]$ mdd
Welcome to MDD - The Manjaro Data Donor
Preparing data submission...

------------------------------------------
        Sending the following data
------------------------------------------
{
    "meta": {
        "version": 1,
        "timestamp": "2024-11-02T14:02:29.754011+00:00",
        "device_id": "939bf6e1-8e22-5927-9c01-a8cff7f4d01d",
        "distro_id": "manjaro",
        "release": "24.1.1",
        "inxi": true
    },
    "system": {
        "kernel": "6.6.54-2-MANJARO",
        "form_factor": "laptop",
        "install_date": "2023-04-07T07:35:41+00:00",
        "product_name": "NJ50_70CU",
        "product_family": "Not Applicable",
        "sys_vendor": "Notebook",
        "board_name": "NJ50_70CU"
    },
    "boot": {
        "uefi": true,
        "uptime_seconds": 24921
    },
    "cpu": {
        "arch": "x86_64",
        "model": "Intel Core i7-10510U",
        "cores": 4,
        "threads": 8
    },
    "memory": {
        "ram_gb": 15.319877624511719,
        "swap_gb": 7.812496185302734
    },
    "graphics": {
        "comp": "kwin_wayland",
        "dri": "iris",
        "gpus": [
            {
                "vendor": "CLEVO/KAPOK",
                "model": "Intel CometLake-U GT2 [UHD Graphics]",
                "driver": "i915"
            }
        ],
        "outputs": [
            {
                "model": null,
                "res": "1920x1080",
                "refresh": 0,
                "dpi": 0,
                "size": "N/A"
            }
        ]
    },
    "audio": {
        "servers": [
            {
                "name": "PipeWire",
                "active": true
            }
        ]
    },
    "disk": {
        "disks": [
            {
                "size_gb": 931.5133895874023,
                "root": {
                    "size_gb": 292.96875,
                    "fstype": "ext4",
                    "crypt": false
                },
                "home": null
            }
        ],
        "windows": true
    },
    "locale": {
        "region": "en_GB.UTF-8",
        "language": "en",
        "timezone": "Europe/Paris"
    },
    "package": {
        "last_update": "2024-11-01T23:02:47+01:00",
        "branch": "stable",
        "pkgs": 1938,
        "foreign_pkgs": 43,
        "pkgs_update_pending": 0,
        "flatpaks": 0,
        "pacman_mirrors": {
            "total": 2,
            "ok": 1,
            "country_config": "France"
        }
    },
    "desktop": {
        "cli": "/bin/bash",
        "gui": "KDE Plasma",
        "dm": "SDDM",
        "wm": "kwin_wayland",
        "display": "wayland",
        "display_with": "Xwayland"
    }
}
------------------------------------------

Succesful sent at 2024-11-02 15:02:32
[nls@lap ~]$

@romangg
Edit 1 : not available on ARM ? I updated an ARM device right now.

[nls@c2-1 ~]$ sudo pacman -S mdd
[sudo] password for nls: 
error: target not found: mdd
[nls@c2-1 ~]$
1 Like
8

Woah, so you’re asking if I will let you spy on me…

Ok, done.

10 Likes
9

I appreciate this kind of transparency.

I’m curious how the opt-out will be implemented, if that’s been determined.

I realize that making it too easy to opt-out will potentially skew data too much, while making it hidden or difficult to opt-out will potentially trigger privacy concerned people, especially those who are ignorant to what the collected datum entails.

Regardless, thank you for being forthcoming in the approach. It is refreshing to see in this day and age.

4 Likes
10

I am perfectly fine with it the way it is now: source available, prewiew of the data, public website for the statistics, clear communication.

As for the opt out, it belongs to manjaro settings manajer. A simple app with that explanation, link to source, preview of data, link to statistic website, a button to disable the service.

3 Likes
11

You mean… the OPT-IN?

I sent data, and looked at it beforehand, the way it collects the monitor(s) refresh rate is unreliable. You can see that also on the website, almost a third of people did not send refresh rate data.

//EDIT: for the OPT-IN, I guess the tool could be a “dependency” of Manjaro (with a service regularly collecting data), and a popup could ask user if he wants to help send data after installation on an update, and a reboot, and if user says NO, no more popup and disable the service collecting data regularly.

3 Likes
12

Yes, I believe the problem is inxi on Wayland. Here inxi doesn’t seem to reliable yet gather the data.

I looked into it an hour ago and I could try to get the data via wayland-info or even xrandr in case Xwayland is installed. Or inxi gets updated to support Wayland better in this regard.

13

I use X11 and the result for both monitors is “0” in the JSON.

omano  ~ $  echo $XDG_SESSION_TYPE
x11

For the refresh rate issue, probably dual monitors cause the issue?

//EDIT: no disabled second monitor, still 0. xrandr is accurate though on X11.

14

No, dual monitors should be handled. We simply pick the largest refresh rate.

Can you run inxi -Gxxx and xrandr locally and paste the output here?

15

Only the Graphics part (command gave full system)

Graphics:
  Device-1: Advanced Micro Devices [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT /
    6900 XT] vendor: XFX driver: amdgpu v: kernel arch: RDNA-2 code: Navi-2x
    process: TSMC n7 (7nm) built: 2020-22 pcie: gen: 4 speed: 16 GT/s lanes: 16
    ports: active: DP-3,HDMI-A-1 empty: DP-1,DP-2 bus-ID: 09:00.0
    chip-ID: 1002:73bf class-ID: 0300
  Display: x11 server: X.Org v: 21.1.14 with: Xwayland v: 24.1.4
    compositor: kwin_x11 driver: X: loaded: amdgpu unloaded: modesetting,radeon
    alternate: fbdev,vesa dri: radeonsi gpu: amdgpu display-ID: :0 screens: 1
  Screen-1: 0 s-res: 3840x1200 s-dpi: 96 s-size: 1014x317mm (39.92x12.48")
    s-diag: 1062mm (41.83")
  Monitor-1: DP-3 mapped: DisplayPort-2 pos: primary,left model: PA248QV
    serial: <filter> built: 2021 res: 1920x1200 dpi: 94 gamma: 1.2
    size: 518x324mm (20.39x12.76") diag: 611mm (24.1") ratio: 16:10 modes:
    max: 1920x1200 min: 720x400
  Monitor-2: HDMI-A-1 mapped: HDMI-A-0 pos: right model: FUNAI TV built: 2009
    res: 1920x1080 dpi: 3048 gamma: 1.2 size: 16x9mm (0.63x0.35")
    diag: 6mm (0.2") ratio: 15:9 modes: max: 1920x1080 min: 640x480
  API: EGL v: 1.5 hw: drv: amd radeonsi platforms: device: 0 drv: radeonsi
    device: 1 drv: swrast gbm: drv: kms_swrast surfaceless: drv: radeonsi x11:
    drv: radeonsi inactive: wayland
  API: OpenGL v: 4.6 compat-v: 4.5 vendor: amd mesa v: 24.2.6-arch1.1
    glx-v: 1.4 direct-render: yes renderer: AMD Radeon RX 6800 (radeonsi navi21
    LLVM 18.1.8 DRM 3.54 6.6.58-1-MANJARO) device-ID: 1002:73bf
    memory: 15.62 GiB unified: no
  API: Vulkan v: 1.3.295 layers: 8 device: 0 type: discrete-gpu name: AMD
    Radeon RX 6800 (RADV NAVI21) driver: mesa radv v: 24.2.6-arch1.1
    device-ID: 1002:73bf surfaces: xcb,xlib


omano  ~ $  xrandr
Screen 0: minimum 320 x 200, current 3840 x 1200, maximum 16384 x 16384
DisplayPort-0 disconnected (normal left inverted right x axis y axis)
DisplayPort-1 disconnected (normal left inverted right x axis y axis)
DisplayPort-2 connected primary 1920x1200+0+0 (normal left inverted right x axis y axis) 518mm x 324mm
   1920x1200     59.95 +  74.93*   72.00    60.01    50.02  
   1920x1080     60.00    50.00    59.94  
   1600x1200     60.00  
   1680x1050     59.95  
   1280x1024     75.02    60.02  
   1440x900      59.89  
   1280x960      60.00  
   1280x800      59.95  
   1152x864      75.00  
   1280x720      60.00    50.00    59.94  
   1024x768      75.03    70.07    60.00  
   832x624       74.55  
   800x600       75.00    60.32    56.25  
   720x576       50.00  
   720x480       60.00    59.94  
   640x480       75.00    72.81    66.67    60.00    59.94  
   720x400       70.08  
HDMI-A-0 connected 1920x1080+1920+0 (normal left inverted right x axis y axis) 16mm x 9mm
   1920x1080     50.00 +  60.00*   59.94    30.00    25.00    24.00    29.97    23.98  
   1680x1050     50.00  
   1280x1024     50.00  
   1440x900      50.00  
   1360x768      60.02  
   1280x800      50.00  
   1280x768      59.99  
   1280x720      60.00    50.00    59.94  
   1024x768      60.00  
   800x600       60.32  
   720x576       50.00  
   720x480       60.00    59.94  
   640x480       60.00    59.94
16

Thank you. So while xrandr picks up the refresh data quite fine, inxi doesn’t seem to.

@h2-1: Do you have an idea why that could be the case?

As an alternative I can implement a fallback path to go to xrandr (and to wayland-info) for data that inxi wasn’t able to obtain.

1 Like
18

Got this error when trying to install mdd:
error: target not found: mdd.

But due to others reporting issues with the latest stable releases, I have not installed the latest one. Maybe that’s my problem.

19

Having the same error. Everything else downloads except the MDD package. Received “failed retrieving file” 404 error.

I’m in USA

1 Like
20

Mirror not up-to-date, perhaps?

21

Wait for your mirrors to sync…