CO-RE (Compile Once - Run Everywhere) technology is more than a year old now. It allows observing systems’ structs through eBPF without installing kernel headers (kernel-devel package) & recompiling tools on every machine. It’s done by including premade & compressed debugging info directly into the kernel. This procedure requires compiling with the flag “CONFIG_DEBUG_INFO_BTF=y” enabled, and a kernel’s size increases only roughly up to 1.5 MB.
BPF is the future of Linux observability tooling, and CO-RE has incredibly reduced an overhead. Would it be possible to include the previously mentioned flag into Manjaro’s kernel building process?
More info here (remove spaces):
facebookmicrosites. github. io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html
According to this doc, some major Linux distributions come with kernel BTF already built in:
- Fedora 31+
- RHEL 8.2+
- OpenSUSE Tumbleweed (in the next release, as of 2020-06-04)
- Arch Linux (from kernel 5.7.1.arch1-1)
- Ubuntu 20.10
- Debian 11 (amd64/arm64)
Kindly ping and ask.
Is there any plan for manjaro to follow up turning CONFIG_DEBUG_INFO_BTF on ?
Pinging @nightmare-2021 , the master of the manjaro kernels
Will be included with the next update for the kernels 5.10 + 5.11 + 5.12
Thanks a lot, that will definitely be a fantastic feature!
I’m using kernel 5.10.42 which doesn’t seem to have BTF enabled (There is no entry for /sys/kernel/btf)
Should it be added to future 5.10.x versions or should I upgrade my kernel to have it?
BTF will not be added, cause it growes up the kernel for more then 120 MB plus !!!
That’s too bad.
The future of Linux is eBPF, and CO-RE is the future of eBPF.
I am maintaining an open source project named Tracee, which is eBPF based, and we are about to add CO-RE support to it. With CO-RE, the compiled binary will be able to run on kernels which have BTF enabled out of the box, just like a regular binary. Without it, the users will have to compile the bpf code with the kernel headers of the specific kernel it runs on.
Tracee is not the only project that will use CO-RE. Cilium is another example and more to come in the future.
Most major distros already have BTF enabled as @oji mentioned above, and by not enabling it now, I think that manjaro users (and I’m one of them) will be left behind.
So we are talking about this project:
If we look at Arch and our config of our kernel I see this:
--- config-Manjaro 2021-06-17 00:40:50.642243000 +0200
+++ config-Arch 2021-06-17 06:59:52.432230512 +0200
@@ -1,6 +1,6 @@
# Automatically generated file; DO NOT EDIT.
-# Linux/x86 5.12.7-2 Kernel Configuration
+# Linux/x86 5.12.11-arch1 Kernel Configuration
CONFIG_CC_VERSION_TEXT="gcc (GCC) 11.1.0"
@@ -260,7 +260,9 @@ CONFIG_BPF_SYSCALL=y
-# CONFIG_BPF_PRELOAD is not set
@@ -449,7 +451,6 @@ CONFIG_X86_PMEM_LEGACY_DEVICE=y
@@ -10140,7 +10139,14 @@ CONFIG_DEBUG_BUGVERBOSE=y
# Compile-time checks and compiler options
-# CONFIG_DEBUG_INFO is not set
+# CONFIG_DEBUG_INFO_REDUCED is not set
+# CONFIG_DEBUG_INFO_COMPRESSED is not set
+# CONFIG_DEBUG_INFO_SPLIT is not set
+# CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT is not set
+# CONFIG_GDB_SCRIPTS is not set
# CONFIG_READABLE_ASM is not set
So we may try it by not enabling the whole
CONFIG_DEBUG_INFO if that is possible @nightmare-2021. However, it would be good that @yanivagman or @chendotjs try to modify the config file of one of our kernels, like linux512 and compile it locally on their systems to see the size
Please test and give feedback:
If extramodules like nvidia are needed, let me know.
However, we might not enable it for the regular kernel. It blows up the kernel a lot:
5724103 254M -rw-r–r-- 1 phil phil 254M 17. Jun 13:21 linux512-5.12.11-2-x86_64.pkg.tar.zst
1103490 87M -rw-r–r-- 1 root root 87M 10. Jun 20:12 linux512-5.12.10-1-x86_64.pkg.tar.zst
1104025 87M -rw-r–r-- 1 root root 87M 16. Jun 21:41 linux512-5.12.11-1-x86_64.pkg.tar.zst
1102769 87M -rw-r–r-- 1 root root 87M 27. Mai 23:06 linux512-5.12.7-2-x86_64.pkg.tar.zst
1102892 87M -rw-r–r-- 1 root root 87M 29. Mai 00:41 linux512-5.12.8-1-x86_64.pkg.tar.zst
1103093 87M -rw-r–r-- 1 root root 87M 3. Jun 20:37 linux512-5.12.9-1-x86_64.pkg.tar.zst
Here are the changes so far. So if we are near 90 MB in size we might enable it …
I guess you guys can make it an optional kernel like the real-time kernels. Of course, if the impact is minimal in management and for the repos?
EDIT: No longer needed/valid. See @nightmare-2021 's post below.
If someone from the community can be found who usually builds the kernel and the extramodules on Wednesdays and Saturdays, this is possible.
We will definitely not do this due to the additional time required.
I tested the kernel you compiled and it works great!
I don’t understand why adding CONFIG_DEBUG_INFO_BTF=y adds an extra of 120MB
From your commits I can see that you also included the CONFIG_DEBUG_INFO=y -
is this a must?
According to the link you provided before (BPF binaries: BTF, CO-RE, and the future of BPF perf tools) setting CONFIG_DEBUG_INFO_BTF=y should only add 1.5MB to the image size
We’ve found the cause and are in the process of changing the kernel build.
Everything should work with the next kernel update.
@nightmare-2021 thats great!
Thank you guys for looking into it and figuring this out.
It’s already at unstable - so you can test it.
We’ll include it for all Kernel >= 5.4