TempFS or ZRAM for Web Browsing (Firefox/Chrome)?

johntdavis84 · 28 December 2020 21:23

Hello!

I’m planning on setting up some sort of RAM disk to support Firefox and Chrome while browsing on my Pi 4b. They both really hit my SD card, and I’m seeing massive CPU usage from what I assume are repeated disk I/O operations that are taking too long. Trying to use speediest.net in Chromium pretty much pegs all 4 cores.

I’m planning on putting Chrome and Safari’s data into a RAM disk, but I’m not sure if I should use TempFS or ZRAM. I know ZRAM has an additional compression function, but does it actually perform any better? I don’t anticipate using more than 500M of system RAM. I don’t have that much to go around.

Thanks!

Razathorn · 28 December 2020 22:56

I honestly think you’re barking up the wrong tree here. I did a LOT of investigation on sd card access when using firefox and chrome and my determination was that it was simply nowhere close to blocked on io. Sd cards and the underlying filesystem have so much buffering that it simply isn’t blocking on the disk unless you have a really bad sd card. I recommend you try running iotop while browsing and you’ll see it’s nowhere close to exceeding bandwidth or the buffers.

Try running iozone without the -e param and you’ll find it doing GB/s of random writes, and even after rebooting, it will be pulling 8-12 MB/s random 4k reads on a 500M file. When you see CPU pegged, it is CPU. Blockage on io shows up as IOwait. It’s just the single core speed of these arm SBCs is pretty darn slow. I daily drive a pi4b as well, and I’m not even considering adding an SSD because it just doesn’t block on the sd card for daily activity at all in my experience.

These tests were done AFTER a reboot on an existing 500M file. Note the speed of the usb spinning disk next to it. I suggest consulting iotop and watching the io rates and you’ll note it’s just cpu blocked, no io blocked, unless you’ve got something else going on.

johntdavis84 · 28 December 2020 23:16

Thanks! Any suggestions for speeding up Firefox and Chrome? Or at the very least keeping them from trying to murder the Pi?

I don’t use XFCE very often–I’m mostly headless, with various web-admin services running–and it’s a slog every time I have to go into the GUI. The UI lags 2-3 seconds from clicking a menu or an icon and getting a new window or menu to open, for example. But I can deal with that. Chrome being nearly unusable is a whole other thing–I have to go to another computer to run google searches when troubleshooting.

EDIT: You mentioned “iozone,” but I cannot find that. Did you mean iotop again?

Razathorn · 28 December 2020 23:52

The iozone program is a disk benchmarking tool (iozone3). Chromium should not be that slow. I’m literally building software in the background right now on an SD card while typing this to you on my pi4 and I can click on a menu item or make a new tab in chromium -instantly-. I mean… maybe you have a bad sd card or controller? that’s insane. It’s not a fast experience, but I’ve literally got 7 tabs open right now while building in the background and it is faster than you’re describing by a lot. I’d be interested to see what iotop says and what else is going on on your system. I’m also running under plasma (with composition turned off so 60fps 1080 video runs.)

johntdavis84 · 29 December 2020 01:52

The iozone program is a disk benchmarking tool (iozone3).

Odd. I just tried to install iozone from the AUR and got an error about it not being available for the aarch64 architecture.

Do I have to modify the build files to install this on an RPi4?

Chromium should not be that slow. I’m literally building software in the background right now on an SD card while typing this to you on my pi4 and I can click on a menu item or make a new tab in chromium -instantly-. I mean… maybe you have a bad sd card or controller? that’s insane. It’s not a fast experience, but I’ve literally got 7 tabs open right now while building in the background and it is faster than you’re describing by a lot. I’d be interested to see what iotop says and what else is going on on your system. I’m also running under plasma (with composition turned off so 60fps 1080 video runs.)

What’s the best way to capture iotop output for the forum?

Using XFCE, the whole system is sluggish, even when Firefox/Chrome is closed. I’ve gotten used to clicking and waiting 1-3 seconds for something to happen, even when top isn’t showing any sort of CPU constraint. It’s when I’m hitting something like speediest.net or fast.com in either browser in XFCE that all 4 cores completely peg. From the command line, the speedtest-cli tool takes about ~25 percent of the CPU and still underreports my speed.

I’m starting to wonder, as you said, if there’s something wrong with the SD Card. It’s rated for 100MB/s, and even with the improved disk IO in the 5.10 kernel, I still can’t break ~80 MB/s, and usually hit somewhere between 75-80.

I had assumed that was just from the cheap USB SD Card reader I’m using, but now I’m wondering about the card itself. (Still, the reader is much faster than the built-in microSD slot, which never broke 40 MB/s. I actually switched to booting and running off a USB 3 card reader to try to address the CPU throttling issue, because I thought it was disk I/O.)

Drive info:

~]$ sudo hdparm -I /dev/sdd

/dev/sdd:
SG_IO: bad/missing sense data, sb[]: 70 00 05 00 00 00 00 0a 00 00 00 00 24 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

ATA device, with non-removable media
Standards:
Likely used: 1
Configuration:
Logical max current
cylinders 0 0
heads 0 0
sectors/track 0 0
–
Logical/Physical Sector size: 512 bytes
device size with M = 10241024: 0 MBytes
device size with M = 10001000: 0 MBytes
cache/buffer size = unknown
Capabilities:
IORDY not likely
Cannot perform double-word IO
R/W multiple sector transfer: not supported
DMA: not supported
PIO: pio0

Here’s a benchmark using hdparm: ~]$ clear && lsblk && sudo hdparm -t --direct /dev/sdd

I’m not yet sure how reliable the various benchmarking tools are, so here’s a test using dd.
Two things I immediately notice:

The first test is the slowest reported number I’ve ever seen off this card.
The microSD card seems very inconsistent from run to run. I’d expect minor variances, but I did these one after the other with maybe 1-2 seconds in between.

[~]$ dd if=/dev/zero of=~/2G-test-file.txt oflag=direct bs=128k count=16k
16384+0 records in
16384+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 68.2323 s, 31.5 MB/s

[~]$ dd if=/dev/zero of=~/2G-test-file2.txt oflag=direct bs=128k count=16k
16384+0 records in
16384+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 42.5517 s, 50.5 MB/s

[~]$ dd if=/dev/zero of=~/2G-test-file2.txt oflag=direct bs=128k count=16k
16384+0 records in
16384+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 44.0087 s, 48.8 MB/s

[~]$ dd if=/dev/zero of=~/2G-test-file2.txt oflag=direct bs=128k count=16k
16384+0 records in
16384+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 36.1134 s, 59.5 MB/s

[~]$ dd if=/dev/zero of=~/2G-test-file2.txt oflag=direct bs=128k count=16k
16384+0 records in
16384+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 40.2269 s, 53.4 MB/s

0n0w1c · 29 December 2020 04:01

I am all for caching, I even cache nfs mounts. But what browsers consume is memory. Browsers use disk caching for web content that ideally would be held in memory, were it available. I am not sure using ram to cache a disk cache is best use.

But then again, we use ram as swap space!

johntdavis84 · 29 December 2020 04:59

So, I watched my process monitor closer when I ran the speediest again, using speediest.net. 3.71 GiB free is what I normally see when idle. I believe the rest is taken up by reserved VRAM and various tempFS mounts.

First, stats at login screen (LightDM’s default greeter):

Memory usage during the test:

CPU usage during the test. Behold the redlining. It’s choking so badly the test results aren’t even close to accurate.

Oddly enough, The observed amount of CPU usage in the far right column doesn’t come close to enough to peg the CPU (~ 76 percent). That’s not great, but I’m not seeing about 25 percent of the load in the process list. I’m missing something.

About the best thing I can say about this is the Ice Tower is doing its job and keeping temps cool when the CPU is choking.

Razathorn · 29 December 2020 05:48

Honestly, with those DD numbers, you’re making me want to try the 5.10 kernel. I’m on the LTS kernel (5.4.81). I only get around 45MB/s on sustained write at most, generally closer to 30-40.

Makes me wonder if you’re just getting wrecked by your kernel selection on pi. 5.4 has full acceleration and everything. Might give it a try. Even Raspberry Pi OS 64 bit is still on 5.4 and still trying to get 5.10 working as well.

Razathorn · 29 December 2020 05:51

If you use yay to install iozone, you can elect to ignore the architecture warning and it will build and install just fine. I use the following scripts:

[raz@nibble iozone]$ cat test-throughput.sh  
iozone -e -s 500000 -r 8192 -i 0 -i 1
[raz@nibble iozone]$ cat test-4k-random-prep.sh  
iozone -e -s 500000 -r 4 -i 0 -i 1 -w -f testfile
[raz@nibble iozone]$ cat test-4k-random-after-reboot.sh  
iozone -e -s 500000 -r 4 -i 2 -w -f testfile
[raz@nibble iozone]$

I reboot after the prep script and then run the after-reboot script so the file isn’t cached.

johntdavis84 · 29 December 2020 06:06

Thanks for the iozone install instructions. That worked perfectly. I’ll set up the test scripts and try them and report back with results.

I’m glad you brought this up. I was having this throttling issue before switching to 5.10. I was still on whatever the default is (5.4?) at that point, and switching the kernel was just one of the things I tried to help. So far, I’ve:

Moved my microSD card out of the Pi’s onboard slot, and onto a USB 3.0 Card Reader plugged into the Pi. I was capped at 45 MB/s before I did this. It had no impact on the CPU throttling I observe when running a speed test in Chromium/Firefox, or the generally high CPU usage when they’re open.
Moved to the 5.10 kernel;
Switched to the picom compositor; and
Enabled a bunch of “shift this to the GPU” stuff in Chromium in hopes of taking pressure off the 4 primary cores.

I am getting superior file I/O numbers in various benchmarks now. Still not coming close to the top speed this microSD card is supposed to get (100MB/s), but I suspect overhead from the reader/the USB 3 pipe is to blame for that. I’ve also got a 1Gbps Ethernet adapter plugged in to hardware isolate my publicly exposed microservices++, so the USB3 pipe is kind of full.

Based on the results I’ve posted so far and your reaction, is it fair to say my microSD card is probably functioning correctly?. The initial low performance with the dd test vs. later runs made me wonder.

EDIT: @Razathorn, I should mention as well that the CPU is overclocked to 2 GHz. It never crashes/fails any benchmarks, and its temp stays low, but could this be the source of the problem? Maybe I’m just pushing it too hard. I’ve got no prior experience with overclocking, so it didn’t occur to me that this might be it until late last night.

++ And to avoid having to figure out, for now, how to assign a second set of v4 and v6 IPs+++ to the internal interface–all the tutorials assume I know a lot more about this than I do.

+++ Second IP is to avoid a port conflict between docker containers, which I can deal with once I get a bit smarter with my reverse proxy manager.

Razathorn · 29 December 2020 16:48

I’m also OC at 2.0 ghz. You will never get much north of the 30-40 range on sd card for REAL performance. The os cache and sd card cache are always going to trick you. The Pi sd card slot just isn’t capable of it. I get 300MB/s if I write a 500MB file to disk, but if I write a 2000MB file, I get 31.8 MB/s just now. This is why the iozone stuff has the -e param, but sync doesn’t kill read buffers, and your read numbers are just always inflated until a reboot on a pre-existing file, and even then, the random read of that 500MB file is going to have some read ahead cache in there.

I’ve done a LOT of operating system trials on the pi, and the only thing I do to make chrome fast is, well, install chromium, turn off the GPU blacklist flag, install h264ify, linux scroll wheel fix, and disable composition (although it’s fast before disabling composition, just won’t do 60fps 1080 video.)

I’m concerned there’s something going on in your video drivers or something maybe? Are you running with vc4-fkms-v3d ?

Wayne

johntdavis84 · 29 December 2020 17:12

I haven’t done any of the bolded things. I’ll look into those. OTOH, I’m seeing slowdowns outside of Chrome, so I don’t think that’s the whole of it, though I’ll definitely implement those tweaks.

inxi -Fazy (excerpts):

System:
Kernel: 5.10.1-1-MANJARO-ARM aarch64 bits: 64 compiler: N/A
parameters: coherent_pool=1M 8250.nr_uarts=0
snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1
video=HDMI-A-1:1920x1080M@60 smsc95xx.macaddr=DC:A6:32:67:EC:54
vc_mem.mem_base=0x3eb00000 vc_mem.mem_size=0x3ff00000 root=LABEL=ROOT_MNJRO
rw rootwait console=ttyS0,115200 console=tty1 selinux=0 plymouth.enable=0
smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 kgdboc=ttyS0,115200 elevator=noop
usbhid.mousepoll=8 snd-bcm2835.enable_compat_alsa=0 audit=0
cgroup_enable=memory swapaccount=1
Console: tty 1 DM: LightDM 1.30.0 Distro: Manjaro ARM

CPU:
Speed: 2000 MHz min/max: 600/2000 MHz Core speeds (MHz): 1: 2000 2: 2000
3: 2000 4: 2000

Graphics:
Device-1: bcm2711-vc5 driver: vc4_drm v: N/A bus ID: N/A chip ID: brcm:gpu
Device-2: bcm2711-hdmi0 driver: N/A bus ID: N/A chip ID: brcm:soc
Device-3: bcm2711-hdmi1 driver: N/A bus ID: N/A chip ID: brcm:soc
Display: server: X.org 1.20.10 driver: modesetting alternate: fbdev
tty: 255x63
Message: Advanced graphics data unavailable in console. Try -G --display

Drives:
Local Storage: total: 238.50 GiB used: 12.51 GiB (5.2%)
SMART Message: Required tool smartctl not installed. Check --recommends
ID-1: /dev/sdd type: USB vendor: Kingston model: Multi-Reader -3
size: 238.50 GiB block size: physical: 512 B logical: 512 B serial:
rev: 1.00 scheme: MBR

Partition:
ID-1: / raw size: 238.26 GiB size: 234.51 GiB (98.43%)
used: 12.46 GiB (5.3%) fs: ext4 dev: /dev/sdd2
ID-2: /boot raw size: 213.6 MiB size: 213.4 MiB (99.89%)
used: 52.4 MiB (24.6%) fs: vfat dev: /dev/sdd1

Swap:
Kernel: swappiness: 60 (default) cache pressure: 100 (default)
ID-1: swap-1 type: zram size: 5.57 GiB used: 0 KiB (0.0%) priority: 100
dev: /dev/zram0

Sensors:
System Temperatures: cpu: 41.9 C mobo: N/A
Fan Speeds (RPM): N/A

Info:
Processes: 201 Uptime: 12h 21m Memory: 3.78 GiB used: 656.0 MiB (17.0%)
gpu: 64.0 MiB Init: systemd v: 246 Compilers: gcc: 10.2.0 Packages:
pacman: 793 lib: 206 Shell: Bash v: 5.0.18 running in: tty 1 (SSH)
inxi: 3.1.08

Here’s what I’m booting up with:

─────
│ File: /boot/config.txt
───────┼
1 │ # See /boot/overlays/README for all available options
2 │
3 │ initramfs initramfs-linux.img followkernel
4 │ kernel=kernel8.img
5 │ arm_64bit=1
6 │ enable_gic=1
7 │ disable_overscan=1
8 │
9 │ #enable sound
10 │ dtparam=audio=on
11 │ hdmi_drive=2
12 │
13 │ gpu_mem=64
14 │ #enable vc4
15 │ dtoverlay=vc4-fkms-v3d
16 │ max_framebuffers=2
17 │
18 │ #Monitor Configuraiton
19 │ #hdmi_enable_4kp60=1 # Enable 4K60 display
20 │
21 │ #overclock (ETA Prime @ How to Overclock Raspberry Pi 4 To 2.0Ghz! All 4 Cores - With Benchmarks - YouTube )
22 │ over_voltage=5 # Overvolt CPU
23 │ arm_freq=2000 # Overclock to 2 GHz from 1.5 GHz
24 │ gpu_freq=620

Razathorn · 29 December 2020 18:04

Honestly, I’m really suspicious of that usb card reader. This feels like latency. The 4k read test with iozone after reboot will tell the story.

johntdavis84 · 29 December 2020 20:35

Okay, here we go:

Prep:

After Reboot:

I have no idea what the random scores should be for this microSD card, but that random read is surprisingly slow.

I should note that I have experienced these lag issues before and after switching to the external USB card reader. I actually switched over because I was attempting to get faster speeds than I was getting from the internal slot. (Also I like the convenience of having the microSD card within easier reach on my desk.

Razathorn · 29 December 2020 22:25

It’s not. The random read scores from other pi 4b tests I’ve seen online are in the 4MB/s range, so these are about right. It’s about where my sandisk a2 64GB card was. I’d also suggest running a dd if=testfile of=/dev/null bs=4M after a fresh reboot to see what your fs read speed without any possible caching is.

Honestly, my impression is this shouldn’t cause the issues you’re describing.

The samsung evo select was significantly faster on random read, though. The a2 ratings are bs and require special reader and drivers.

Razathorn · 29 December 2020 23:06

So, in an effort to ensure you don’t have unrealistic expectations of web browsing on pi4 or the pi4 in general as a desktop, here’s a video of mine in action. This is a 10 FPS capture, which slows it down a little, but not much, and it certainly isn’t this choppy in real life–it’s just the 10 fps capture since a 1080p 30fps capture really affects performance a lot more.

johntdavis84 · 30 December 2020 05:51

After a harrowing moment where the external card reader just froze and took down the entire computer, I have moved the SD card back into the Pi. I don’t think that card reader is meant for continuous operation.

I think I may have killed it.

Re: the dd if=testfile of=/dev/null bs=4M test: I first ran dd if=/dev/zero of=~/2G-test-file.txt oflag=direct bs=128k count=16k to generate a test file; rebooted; and ran the test.

~]$ dd if=/dev/zero of=~/2G-test-file.txt oflag=direct bs=128k count=16k
16384+0 records in
16384+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 323.794 s, 6.6 MB/s

The read test, on fresh reboot:

]$ dd if=2G-test-file.txt of=/dev/null bs=4M
512+0 records in
512+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 49.4755 s, 43.4 MB/s

So that seems to be normal, right?

Maybe my Pi just doesn’t like XFCE.

EDIT: Re: Chromium, turns out I had already disabled the blocklist for the GPU and shifted a bunch of stuff over to GPU processing. I did notice that Chrome was set to always stay open in the background and turned that off.

I’ll need to experiment a bit more and observe closer, but I think that’s already giving me an improvement system-wide.

(Why would that be on by default on a Pi?)

system · 14 January 2021 05:51

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.