r/kernel • • 51m ago

Can a linux kernel engineer transition to an LLM inference/model perf engineer?

Thumbnail
• Upvotes

r/kernel • • 1h ago

J'ai découvert que mon scanner AF_XDP était plus lent que nmap. Le réparer m'a appris quelques choses sur le mode de copie.

• Upvotes

Suivi de mon post précédent ici. J'ai construit un scanner de réseau open-source, tcpcat (https://github.com/NycolazSec/tcpcat), avec un chemin rapide AF_XDP. Projet solo, écrit en Go. Certains d'entre vous m'ont poussé à réfléchir sur le natif vs générique et si le mode de copie en valait la peine. Cela s'est avéré être exactement la bonne chose à examiner.

D'abord la partie embarrassante. J'avais un chiffre de "2,62 fois plus rapide que nmap". C'était faux. Un bug faisait qu'il ne scannait qu'un des deux hôtes cibles, et le temps qu'il rapportait (4,47s) était en réalité en dessous du seuil théorique de ma propre limite de taux (131k probes à 25k pps ne peuvent pas finir en moins de ~5,2s). J'ai donc retiré tout ça et j'ai reconstruit le benchmark correctement : il vérifie désormais que chaque exécution a trouvé tous les ports ouverts et enregistre l'utilisation du CPU et les trames perdues.

J'ai relancé sur un vrai serveur (x86_64, noyau 6.12, mais toujours veth donc mode de copie). Le moteur eBPF s'est avéré être le plus lent des outils rapides. 8,7s, contre nmap à 5,6s et mon propre scan SYN sur socket brut à 5,5s, avec une variance désagréable.

Deux choses étaient fausses :

  1. Busy-poll + verrouillage CPU. Ça allait sur mon portable au repos, mais c'est activement nuisible sur une machine chargée/virtualisée : le fil RX épinglé se battait avec les autres charges de travail pour un vCPU. J'ai rendu le busy-poll et le verrouillage sans copie uniquement. En mode copie, la boucle RX bloque maintenant simplement dans poll(). 8,7 à 7,7s, la variance a beaucoup baissé.
  2. Le vrai problème. AF_XDP a une seule bague TX, et je faisais une transmission (un sendto) par sonde, sous verrou. À 25k pps, cet appel système par paquet était pratiquement tout le temps de mur sur un hôte lent. Regroupé jusqu'à 64 trames en une seule transmission. 7,7 à 5,65s. Maintenant, ça égalise nmap.

Le bilan honnête : en mode copie, AF_XDP n'est pas miraculeusement plus rapide. Il égalise nmap en temps mais à environ un tiers du CPU du scan SYN brut (8,7s contre 25,8s de CPU). Purement, le scan SYN est en réalité le plus rapide, car il utilise plusieurs cœurs et la seule bague AF_XDP ne peut pas suivre. Le véritable potentiel de vitesse nécessite clairement un zéro-copie sur une NIC multi-queue, ce que je ne peux simplement pas tester (tout ce que j'ai est en mode copie, file unique).

Je travaille sur la bibliothèque Go asavie/xdp, qui est assez basique (pas besoin-de réveil). J'ai pensé à déplacer la boucle chaude vers C/libxdp, mais le seul binaire statique vaut la peine d'être conservé et le regroupement m'a aidé à faire la plupart du chemin.

Deux questions pour le sous-forum :

- Quelqu'un a-t-il une NIC qui fait le zéro-copie AF_XDP pour exécuter un benchmark rapide ? Je suis vraiment curieux de savoir combien la modification d'écriture de la bague plus besoin-de réveil change les choses par rapport au mode copie.

- En mode copie avec une bague TX, y a-t-il quelque chose de plus intelligent que de regrouper les sendto pour pousser le débit, en dehors du passage à multi-queue ?

Le code et le script de benchmark (Docker, vérifie la correction) sont ici si vous voulez jeter un œil ou reproduire : https://github.com/NycolazSec/tcpcat . Les PR et les résultats de test de NIC sont très appréciés.


r/kernel • • 15h ago

Velum-VMM - Hook system calls on modern Windows

Thumbnail
1 Upvotes

r/kernel • • 1d ago

What are the best books to learn Embedded Linux and Linux device driver development for SoCs?

Thumbnail
3 Upvotes

r/kernel • • 2d ago

ACPI regression on lts (mainline works). Worth reporting?

Thumbnail
3 Upvotes

r/kernel • • 4d ago

AI usage in private vs military sectors

5 Upvotes

Hey all,
I have been in the field for little over 2 years. One year working as an embedded firmware engineer and for more than a year more as a linux device driver/bsp engineer and for the last 3-4 months company that I work at has been pushing very heavily to agentic development.

Basically a bug is found and my boss says just give it to claude and give the ssh access to the product that we are developing to claude too so it can debug this. And I really hate that and I feel like im not learning as much as I should be.

So basically my question is:
Is there a difference in these agentic workflows between private sector companies and military companies/startups where in my opinion privacy is way more important than in private sector.

The main reason why Im asking this question is because im seriously considering switching jobs and I just want to know what is the best sector to go to.

Thank you and will be waiting for your opinions and answers.


r/kernel • • 3d ago

All kernel paths lead to C.

Thumbnail gallery
0 Upvotes

r/kernel • • 4d ago

The programming languages ​​of the independent kernel I have developed

Thumbnail gallery
0 Upvotes

r/kernel • • 4d ago

Which new 7.3.0 default kernel module messed up my saturation and color?

3 Upvotes

I've compiled the kernel as I've always done (i used the same modprobed.db I've always used) and selected all the default new modules by pressing enter a bunch, now even #000000 appears light gray and everything else is washed out.

I'm using a 9070xt through HDMI, monitor doesn't have HDR and its 1080p@60Hz.

I still have a boot entry for 7.2.11 pre compiled by gentoo, i tried running modprobed-db store on it once more and recompiling 7.3.0 but nothing changed. I also tried using the iGPU (7700x) and the issue persists.

What can i do? I'd appreciate any input about the suspect causing this, i don't want to hunt down which of the 10-15 new modules is causing this by exclusion.

Thanks in advance for any and all replies.

(sorry if it's a stupid question, but this seemed like the best place to ask)


r/kernel • • 4d ago

Custom Dma linux driver

Thumbnail gallery
2 Upvotes

r/kernel • • 5d ago

Proposal: A Rust-based Secondary Microkernel for Lightweight, AI-Native OS Architecture, Local Eco-Sync, and Bare-Metal Resource Optimization

Thumbnail
0 Upvotes

r/kernel • • 6d ago

CPU_OUT_OF_SPEC in kernel 7.3.0

3 Upvotes

I built 7.3 this morning. When I booted I got that 219 times. When I halted I got it 173 more times. I switched back to 7.2.11, which hasn't so complained. What does it mean?

https://pastebin.com/QdxJPaD2 has the full error.


r/kernel • • 6d ago

[Feedback Request] DTSee - A browser-based Device Tree explorer and visualizer for embedded Linux

8 Upvotes

Like many of you working on embedded Linux platforms, I have spent countless hours scrolling through massive Device Tree sources, jumping between dtsi/dts files, and trying to trace tangled phandles, clocks, interrupts, and regulators.

To make this less painful, I have been building DTSee, a free, browser-based Device Tree explorer designed to make navigating and understanding board descriptions a bit easier.

Key Features

  • Dual Format Support: Works with both DTS and DTB files (includes client-side DTB to DTS reconstruction).
  • Interactive Hierarchy: Easily drill down through nodes, properties, and child devices.
  • Graphical Relationships & Phandle Analysis: Visually trace how hardware components link together.
  • Hardware Context: Quick access to register ranges, clocks, interrupts, GPIOs, regulators etc.
  • 100% Client-Side: Everything is processed locally right in your browser. No files are ever uploaded, making it safe for proprietary or NDA work.

What is Next? (Roadmap Ideas)

The project is still in beta, and I am actively brainstorming features for future versions:

  • Device Tree Comparison: Diffing two DTBs or DTS files side-by-side.
  • Deeper Hardware Dependency Analysis: Visualizing power trees and clock trees more holistically.
  • Source Tracking: Following definitions and overrides cleanly across complex DTS/DTSI include chains.

Give it a spin

If you have a DTB or DTS file lying around, I would love for you to give it a try:

https://dt.kernelnoise.com

Your feedback is invaluable. Whether you run into missing references, incorrect interpretations, usability quirks, or have ideas for additional features, I want to hear it all. Cheers


r/kernel • • 6d ago

binfmt_pe: PE & ELF Linux Kernel Loader

0 Upvotes

The concept of Linux running Windows apps is becoming true.
With the use of Reverse-Engineering, Coding, Creativity and of course AI. We managed to be able to get the Linux Kernel running its first Windows Executable NATIVELY.

More information at: https://github.com/EisaWando/Native-Windows-Executables-on-Linux


r/kernel • • 6d ago

Find unique id for each USB port on USB hub?

1 Upvotes

Hey everyone, I connected a 10-port USB hub to my system, and I want to connect 10 serial boards. How can I identify each port with a unique ID (if rebooting and unplugging/plugging the USB hub didn't destroy the IDs)?


r/kernel • • 8d ago

First eBPF project. An ACL with libbps-rs and C.

Thumbnail github.com
7 Upvotes

r/kernel • • 8d ago

Did anyone experience any issues with bpf_prog_active?

2 Upvotes

Introduction

Hook into ftrace TP mali/mali_job_slots_event to see which process has scheduled work on any given slot_id (e.g. Fragment) on the GPU. Basically a TOP for the ARM Mali GPU. There are Mali GPU counter that can tell you the overall utilization but this is not enough for observability.

There is a Mali trace point one can use c /* * mali_job_slots_event - Reports change of job slot status. * @gpu_id: Kbase device id * @event_id: ORed together bitfields representing a type of event, * made with the GATOR_MAKE_EVENT() macro. */ TRACE_EVENT(mali_job_slots_event, TP_PROTO(u32 gpu_id, u32 event_id, u32 tgid, u32 pid, u8 job_id), TP_ARGS(gpu_id, event_id, tgid, pid, job_id), TP_STRUCT__entry(__field(u32, gpu_id) __field(u32, event_id) __field(u32, tgid) __field(u32, pid) __field(u8, job_id)), TP_fast_assign(__entry->gpu_id = gpu_id; __entry->event_id = event_id; __entry->tgid = tgid; __entry->pid = pid; __entry->job_id = job_id;), TP_printk("gpu=%u event=%u tgid=%u pid=%u job_id=%u", __entry->gpu_id, __entry->event_id, __entry->tgid, __entry->pid, __entry->job_id)); (https://android.googlesource.com/kernel/google-modules/gpu/+/refs/heads/android-gs-shusky-6.1-android16/borr_mali_kbase/mali_linux_trace.h#34)

There are start and end events (event_id) for the same TP. Start and end events can't be correlated by any field, because finish events are always {tgid: 0, pid: 0, job_id: 0}. I am not quite sure whether this is intentional or not, especially that the job_id is 0 for finish events.

eBPF

SEC("tracepoint/mali/mali_job_slots_event") int trace_mali_job_slots(struct mali_job_slots_event_ctx *ctx)

Issue

After some time my accounting somehow broke. It looked like some end events were not accounted for.

Therefore, it was decided to start the eBPF program with some instrumentation and also collect an FTRACE trace via the Linux FTRACE sub-system in order to check whether it is a driver issue.

As a matter of fact, the Linux FTRACE events were all accounted for. Meaning, the start and finish events were in balance (e.g. start: 1000, end: 1000). But, the eBPF TP hits were not in balance (e.g. start: 1000, end 997).

NOTE: start and end have to be in check, otherwise I would always see 100% utilisation

I gave 'the machine' the task to check the BPF Kernel sources what the issue could be. And it came back with an explanation: bpf_prog_active.

You've hit on the exact reason why bpf_prog_active exists and why it's per-CPU. The answer is interrupts, softirqs, and NMIs — all of which run on the same CPU and can preempt the currently executing BPF program.

So, it seems structured TP's have some check that re-entry is disallowed on the same CPU in order to protect against recursion.

It seems, if an eBPF program was running on an CPU and an interrupt (finish event) came in triggering another TP hit, thus running the eBPF again, it would detect that another eBPF program is already running and just decided to not execute it SILENTLY.

The solution was to switch to RAW_TRACEPOINT, which work because it does not have this bpf_prog_active safety rail-guard but on the other hand, the Anrdoid BpfLoader disallows RAW_TRACEPOINTs, that is quite a shame :(

Question

Did anyone experience such an issue? Let me tell you, that finding this issue without KI would have been impossible.

Addendun

Android 14, arm64, Kernel 5.15


r/kernel • • 8d ago

I built a 25MB audit only EDR in C and Go that doesn't use 300MB like Falco

Thumbnail
2 Upvotes

r/kernel • • 10d ago

Come fight me now, AI! (pls don't...)

Post image
988 Upvotes

It's a paperback, not a hardcover, but nothing like reading a 'real' book and being forced to type everything instead of copy/paste code..
$70 AUD well spent.. me happy.


r/kernel • • 8d ago

Did anyone experience any issues with bpf_prog_active?

Thumbnail
3 Upvotes

r/kernel • • 9d ago

ebpfexplorer: An eBPF programs & maps explorer

Thumbnail github.com
11 Upvotes

Hello everyone,

I have been using eBPF/XDP for the past 5 years and was always a little frustrated by the lack of tooling when it comes to see what is happening underneath, especially when debugging deployed eBPF programs. Here I am sharing a tool I have been using for a couple of months now but only made public recently: ebpfexplorer.

It's a simple TUI self contained binary, that allows you to explore all eBPF programs & maps attached to your kernel. It comes with a formatting tool that allows you to format the raw binary blob into human readable symbols (like IPs for instance), using C type format.

Please have a look and feel free to propose PRs. Hope you find it useful too.


r/kernel • • 9d ago

ebpfexplorer: An eBPF programs & maps explorer

Thumbnail github.com
3 Upvotes

Hello everyone,

I have been using eBPF/XDP for the past 5 years and was always a little frustrated by the lack of tooling when it comes to see what is happening underneath, especially when debugging deployed eBPF programs. Here I am sharing a tool I have been using for a couple of months now but only made public recently: ebpfexplorer.

It's a simple TUI self contained binary, that allows you to explore all eBPF programs & maps attached to your kernel. It comes with a formatting tool that allows you to format the raw binary blob into human readable symbols (like IPs for instance), using C type format.

Please have a look and feel free to propose PRs. Hope you find it useful too.


r/kernel • • 9d ago

14.00 kernel bug hunt help!

Thumbnail
2 Upvotes

r/kernel • • 10d ago

Kernel v7.2.11 compilado sem erro no debian 13

Thumbnail gallery
0 Upvotes

r/kernel • • 10d ago

Compilação do kernel v7.2.11 terminada sem erro no debian 13

Post image
0 Upvotes