Reporting issues¶

Handling non-regressions only occurring in stable or longterm kernels¶

Follow the next few steps only if the step-by-step guide sent you here. That is the case when you are (a) facing an issue in the latest release of a still supported stable or longterm series that (b) you were unable to reproduce in the current mainline and (c) is not a regression. If all of that holds true, follow these steps:

• Be aware: It is possible the issue will not be resolved, as the fix might be too big or risky to backport.

• Search Linux’s mainline Git repository or lore for the change resolving the issue. In case you have trouble locating it, consider using a bisection; alternatively, ask on the list of the affected subsystem for advice while CCing the relevant maintainers and developers.

• Check if the change is already scheduled to be backported by searching the patch description for a ‘stable tag’ (e.g., a line like ‘Cc: <stable@vger.kernel.org>’) and the patch’s title in lore.kernel.org:

  • If the change is already scheduled for backporting, it will usually be picked up within one or two weeks after being mainlined. Note, though, plans sometimes change; a comment next to the stable tag might also limit how far the fix is backported and thus exclude the series you care about. If there are good reasons for this, you are out of luck. If you can’t spot any:

    Send a reply asking the involved developers if backporting to the series is an option. Note, though, the developers might greenlight backporting, but unwilling to handle the work themselves -- in which case you or somebody else must test and submit the fix and everything it depends on, as explained in Everything you ever wanted to know about Linux -stable releases.

  • If the change is not scheduled for backporting, search lore for the review of the fix and check if backporting is planned or was rejected. If it is neither, send a reply asking the involved developers if backporting to the series is an option. Just as mentioned in the previous paragraph, you might need to handle backporting on your own.

Conclusion of the step-by-step guide¶

Did you run into trouble following the step-by-step guide not cleared up by the reference section below? Did you spot errors? Or do you have ideas on how to improve the guide?

If any of that applies, please take a moment and let the primary author of this text, Thorsten Leemhuis <linux@leemhuis.info>, know by email while ideally CCing the public Linux docs mailing list <linux-doc@vger.kernel.org>. Such feedback is vital to improve this text further, which is in everybody’s interest, as it will enable more people to master the task described here.

You likely need to compile at least one really fresh kernel¶

Be aware:You should report issues using a Linux kernel that is both really fresh and vanilla. [...] Your email address will become part of public archives [...] You might need to patch and build your own kernel [...]

You most likely will have to fiddle with your setup and install at least one fresh Linux kernel while reporting the issue or trying to resolve it. The step-by-step guide mentions a few, but the most important is:

The kernels most Linux users utilize are unsuitable for reporting bugs upstream, as the problem might have been fixed already or never happened with the upstream code in the first place. Such situations occur frequently when it comes to Linux:

1. Many developers consider all ‘longterm’ (aka ‘LTS’) kernels as too old and thus unsuitable for reporting, except for series-specific regressions (say from 6.1.13 to 6.1.15) in still-supported series (see frontpage of kernel.org). Reporting using the newest version from the latest ‘stable’ series can work -- but some developers only take a closer look at bugs reported using a fresh mainline kernel, as the bug might have been fixed already.

2. Almost all Linux-based kernels pre-installed on devices (computers, laptops, smartphones, routers, …) are therefore too old as well, but even when not, often unsuitable for a second reason:

Most vendors modify Linux’s source code (some heavily!) before building their kernels; frequently their kernels also load externally developed modules (‘out-of-tree drivers’), too. Such modifications or enhancements might be causing your issue, even when they seem tiny and unrelated. Upstream developers can do nothing about such problems.

You therefore have to report issues with such kernels to the vendor. Its developers should look into the report and, in case it turns out to be an upstream issue, report or directly fix it there. In practice that often does not work out or might not be what you want; installing your own fresh vanilla kernel while steering clear of externally developed modules is your way out here.

Note: Some developers, despite the aforementioned aspects, are willing to handle reports about issues in kernels that are somewhat older or slightly diverged. If they will highly depends on the circumstances:

• Your chances are quite good if the vendor performed only small changes to a recent mainline codebase; that, for example, often holds true for the kernels shipped by Debian GNU/Linux Sid (aka ‘unstable’) or Fedora Rawhide.

• Chances are slightly worse but still relatively good for reports about issues with the newest version from the latest Linux stable series. That is the case even when the distributor slightly modified or enhanced the kernel’s code -- this, for example, is often the case for the default kernels of Arch Linux and openSUSE Tumbleweed, as well as regular Fedora releases when using the latest stable series.

You are free to ignore this advice and report problems to the upstream Linux developers occurring with kernels that are outdated, modified, or utilizing out-of-tree drivers. But be aware that developers might react coldly or never at all. Nevertheless, it is still better than not reporting the issue: sometimes such a report directly or indirectly helps to resolve issues over time.

[back to step-by-step guide]

Issues must be reported one by one¶

You must process and report each issue separately if you deal with multiple. If there is a slim chance [...]

You will have to report issues one by one if you deal with multiple, as they likely will be handled by different developers; describing various issues in one report also makes it difficult for others to understand the situation. Hence, only combine issues in one report if they are very strongly entangled or clearly have the same cause.

[back to step-by-step guide]

Evaluate the logs¶

Skim ‘journalctl -k’ (alternatively: ‘dmesg’) for failures and warnings, as maybe there is just something wrong with your setup. [...]

Sometimes a bug you face is just a symptom of something going sideways that the kernel detected and logged -- like a missing firmware file, for example. To rule such things out, check the kernel’s log messages.

Preferably use ‘journalctl’ if your distribution supports it, as in contrast to ‘dmesg’ it always contains all messages since the kernel started.

Especially look out for messages in bold, yellow, or red, as both tools use such to set warnings and errors apart.

[back to step-by-step guide]

Check ‘taint’ flag¶

Check if the kernel was already ‘tainted’ when the issue first occurred [...]

The kernel marks itself with a ‘taint’ flag when something happens that might lead to follow-up errors looking totally unrelated. Your issue might be such an error, in which case there is nothing to report. That is why it is in your interest to check the taint status early in the reporting process. This is the main reason why this step is here in the guide, as you most likely will have to install a different kernel for reporting later -- and then need to recheck the flag, as that is when it matters.

To check the tainted flag, execute cat /proc/sys/kernel/tainted: If it returns ‘0’ everything is fine; if it contains a higher number, it is tainted.

In some situations it is impossible to check that file. That is why the kernel also mentions the taint status when it reports small (a ‘warning’ or a ‘bug’) or big (an ‘Oops’ or a ‘panic’) problems. In such cases, search for a line starting with ‘CPU:’ near the top of the error messages printed on the screen or in the log. If the kernel at that point considered itself to be fine, it will end with ‘Not tainted’; if not, you will see ‘Tainted:’ followed by a few spaces and some letters.

If your kernel is tainted, check Tainted kernels to find out why. Note: It is quite possible that the problem you ran into caused the kernel to taint itself, in which case you are free to ignore the flag. But if the kernel was tainted beforehand, you might have to eliminate the cause or rule out that it is an influence.

These are the most frequent reasons why the kernel set the flag:

1. Some kind of error (like a ‘kernel Oops’) occurred. The kernel then taints itself, as it might misbehave in unexpected ways afterwards. In that case check your kernel or system log and look for a section starting with:

   Oops: 0000 [#1] SMP
   

   That is the first Oops since boot-up, as the ‘#1’ between the brackets shows. Every later Oops and any other problem that happens afterwards might be a follow-up issue that would never have happened otherwise, even if both look totally unrelated. Rule this out by eliminating the cause for the first Oops and reproducing the issue afterwards. Sometimes simply restarting will be enough; other times a change to the configuration followed by a reboot can eliminate the Oops.

   Note: Do not invest too much time into this while you are still on an outdated or vendor kernel: The cause for the Oops might already be fixed in a newer Linux kernel version you most likely will have to install for reporting while following this guide.

2. Your system uses software that installs externally developed kernel modules, for example, kernel modules from Nvidia, OpenZFS, VirtualBox, or VMware. The kernel taints itself when it loads such ‘out-of-tree’ modules -- no matter what license they use, as such modules can cause errors in unrelated kernel areas and thus might be leading to the issue you face. You therefore have to prevent those modules from loading when you want to report an issue to the Linux kernel developers. Most of the time the easiest way to do that: temporarily uninstall such software including any modules they might have installed; afterwards reboot.

3. The kernel taints itself when it loads a module that resides in the staging section of the Linux kernel source. That is a special area for code (mostly drivers) that does not yet fulfill the normal Linux kernel quality standards. When you report an issue with such a module, it is totally okay if the kernel is tainted; just make sure the module in question is the only reason for the taint. If the issue happens in an unrelated area, it is wise to rule out that it is an influence. To do so, reboot and temporarily block the module from being loaded by specifying foo.blacklist=1 as kernel boot parameter (replace ‘foo’ with the name of the module in question).

[back to step-by-step guide]

Ensure there is nothing wrong with the kernel’s surroundings¶

Evaluate briefly if some glitch in your kernel’s environment might make it misbehave -- like a [...]

Problems that look like a kernel issue are sometimes caused by its surroundings. It is impossible to detect sometimes -- but it is wise to rule out a few common causes before wasting time on a meaningless bug report:

• When dealing with a regression (e.g., something stopped working or works worse after updating the kernel), make sure it is not something else that changed in parallel. That could be something else you updated at the same time, like the BIOS, the boot loader, Mesa, the linux-firmware package, or something else close to the kernel; but it could also be some change you performed in the BIOS setup or your Linux distribution’s configuration.

• Try to make sure the hardware is healthy, as problems with it can result in a multitude of issues that look like kernel bugs.

  Ideally try to rule out faulty RAM or a dying device causes the problem.

  Also ensure your computer components run within their design specifications; that is especially important for the main processor, the RAM, and the motherboard. Therefore, stop undervolting or overclocking when facing a potential kernel issue.

• Temporarily remove any optional kernel parameters you use, as they might enable unsupported or experimental features.

• In case of any problems related to booting, check if the initramfs was generated correctly.

• When dealing with a file system issue, check the file system in question with fsck, as it might be damaged in a way that leads to unexpected kernel behavior.

• Use proven tools when building your kernel, as bugs in the compiler or the linker can cause the resulting kernel to misbehave.

[back to step-by-step guide]

Search for existing reports and fixes¶

Search lore.kernel.org for similar reports and potential fixes; afterwards the wider internet, too. [...]

You don’t want to waste your time reporting an issue anew someone already brought forward or resolved already. So it is in your own interest to check for existing reports and fixes.

Issues receiving special treatment¶

Evaluate if the issue you are dealing with qualifies as a regression or security issue, as those [...]

Check if you face an issue that receives special treatment in the Linux development process:

• You deal with a regression, if some application or practical use case running fine with one Linux kernel version works worse or not at all with a newer version compiled using a similar configuration; the ‘no regression’ rule forbids that. The document Reporting regressions explains these and additional aspects in more detail, but everything important is covered in this document.

• What qualifies as a security issue is left to your judgment. Consider reading Security bugs before proceeding, which provides instructions on handling security issues.

[back to step-by-step guide]

Fast track for regressions¶

Do you face a regression? One still occurring in a kernel version less than two (ideally: one) weeks old? A kernel that is vanilla or close to it? If you answered [...]

This is an optional fast track that might relieve you from further work on reporting in case the issue is already known. Note: It are volunteers that answer these emails on a best-effort basis.

[back to step-by-step guide]

Check how to report your issue¶

You must consult MAINTAINERS to determine where developers of the affected driver or subsystem want bugs to be submitted to; use your best guess, if [...]

It is crucial to submit your report to the right place, as the Linux kernel is a big project and most of its developers are only familiar with a small subset of it. Quite a few programmers, for example, only care for just one driver, like one for a particular WiFi chip; its developer likely will only have small or no knowledge about the internals of near, remote, or unrelated subsystems, like the TCP stack, the PCIe/PCI subsystem, memory management, or file systems.

Problem is: The Linux kernel lacks a central bug tracker where you can simply file your issue and make it reach the developers that need to know about it. That is why you have to find the right place and way to report issues yourself. You can do that with the help of a script (see below), but it mainly targets kernel developers and experts. For everybody else, using the MAINTAINERS file is the better approach.

[user@something ~]$ lspci -k
[...]
3a:00.0 Network controller: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter (rev 32)
Subsystem: Bigfoot Networks, Inc. Device 1535
Kernel driver in use: ath10k_pci
Kernel modules: ath10k_pci
[...]

[user@something ~]$ realpath --relative-to=/sys/module/ /sys/class/net/wlp58s0/device/driver/module
ath10k_pci

QUALCOMM ATHEROS ATH10K WIRELESS DRIVER
Mail:          A. Some Human <shuman@example.com>
Mailing list:  ath10k@lists.infradead.org
Status:        Supported
Web-page:      https://wireless.wiki.kernel.org/en/users/Drivers/ath10k
SCM:           git git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git
Files:         drivers/net/wireless/ath/ath10k/

$ modinfo ath10k_pci | grep filename | sed 's!/lib/modules/.*/kernel/!!; s!filename:!!; s!\.ko\(\|\.xz\)!!'
drivers/net/wireless/ath/ath10k/ath10k_pci.ko

$ ./scripts/get_maintainer.pl --no-git -f drivers/net/wireless/ath/ath10k*
Some Human <shuman@example.com> (supporter:QUALCOMM ATHEROS ATH10K WIRELESS DRIVER)
Another S. Human <asomehuman@example.com> (maintainer:NETWORKING DRIVERS)
ath10k@lists.infradead.org (open list:QUALCOMM ATHEROS ATH10K WIRELESS DRIVER)
linux-wireless@vger.kernel.org (open list:NETWORKING DRIVERS (WIRELESS))
netdev@vger.kernel.org (open list:NETWORKING DRIVERS)
linux-kernel@vger.kernel.org (open list)

Search for existing reports in other places¶

If the developers of the affected driver or subsystem use a dedicated mailing list not archived on lore, search its archives for earlier reports and potential fixes; if the subsystem is [...]

Now that you know where they need to be reported to, search the target for existing reports again. If it is a mailing list, you will often find its archives on lore.

But some lists are hosted in different places. That, for example, is the case for the ath10k WiFi driver used as an example in the previous step. But you’ll often find the archives for these lists easily on the net. Searching for ‘archive ath10k@lists.infradead.org’, for example, will lead you to the Info page for the ath10k mailing list, which at the top links to its list archives. Sadly, this and quite a few other lists miss a way to search the archives. In those cases, use a regular internet search engine and add something like ‘site:lists.infradead.org/pipermail/ath10k/’ to your search terms, which limits the results to the archives at that URL.

It is also wise to check the internet, LKML and maybe bugzilla.kernel.org again at this point. If your report needs to be filed in a bug tracker, you may want to check the mailing list archives for the subsystem as well, as someone might have reported it only there.

For details on how to search and what to do if you find matching reports, see ‘Search for existing reports and fixes’ above.

Do not hurry with this step of the reporting process: spending 30 to 60 minutes or even more time can save everyone, including you, quite some trouble.

[back to step-by-step guide]

Prepare for emergencies¶

You might want to create a fresh backup and put both [...]

Remember, you are dealing with computers, which sometimes do unexpected things, especially if you fiddle with crucial parts like their operating system kernel. That is what you are about to do in this process. You thus better want to create a fresh backup. In case you need to install a kernel during the bug reporting process, also ensure you have all tools at hand to repair or reinstall the operating system as well as everything you need to restore the backup.

[back to step-by-step guide]

Start documenting how to reproduce issue¶

Write down coarsely how to reproduce the issue on a freshly booted system in [...]

During the reporting process, you most likely will have to test if the issue happens with other kernel versions. Therefore, it will make your work easier if you know exactly how to reproduce an issue quickly on a freshly booted system. And for the report you need a description anyway.

This obviously is impossible in case you want to report an issue that happened just once. Be aware that it might be fruitless to report such problems, as they might be caused by a bit flip due to cosmic radiation; but if you are experienced enough to tell such a one-time hardware error apart from a kernel issue that is worth reporting even without reproducing it, skip this step and the verification.

[back to step-by-step guide]

Verify the problem with a suitable kernel¶

You must report the problem using a kernel suitable for reporting -- so you [...] In case of a regression within a stable or longterm kernel, you also must check [...]

Following the instructions in this step dramatically increases the chance some developer will look into the report, as it ensures the bug is actually present in a codebase they care about; for regressions in stable or longterm series, they furthermore determine the right point of contact for the bug, which depends on whether the problem happens in mainline as well.

The step-by-step guide outlines the gist of the required tasks; if you need more detailed instructions, follow How to verify bugs and bisect regressions.

[back to step-by-step guide]

Search again¶

If you performed a bisection or learned anything new about the bug while following this guide so far, search once more [...]

During the previous step you likely have learned a thing or two about the issue you face. Use this knowledge and search again for existing reports and potential fixes.

[back to step-by-step guide]

Collect files to provide¶

Collect relevant files to supply with the report. [...]

The developers will usually ask for files with details about your system, as what is needed highly depends on the nature of the problem. But it is often wise to provide at least the kernel log and maybe a few things along with the report, as outlined in the step-by-step guide.

When collecting the kernel’s log messages with dmesg, make sure they start with a line like ‘Linux version 5.8-1 (foobar@example.com) (gcc (GCC) 10.2.1, GNU ld version 2.34) #1 SMP Mon Aug 3 14:54:37 UTC 2020’. The kernel discarded messages from the first boot phase already if it is missing. In that case, instead consider using journalctl -k; alternatively, reboot and reproduce the issue, before calling dmesg right afterwards.

In case the kernel’s log messages contain a ‘panic’, ‘Oops’, ‘warning’, or ‘BUG’, you might want to decode them as described below if that is easy for you -- but that is optional, as many bugs can be solved without this.

On many Linux distributions the tools mentioned by the guide are installed by default, except maybe alsa-info.sh, which the sound subsystem developers provide.

[user@something ~]$ sudo dmesg | ./linux-5.10.5/scripts/decode_stacktrace.sh ./linux-5.10.5/vmlinux

[user@something ~]$ sudo dmesg | ./linux-5.10.5/scripts/decode_stacktrace.sh \
/usr/lib/debug/lib/modules/5.10.10-4.1.x86_64/vmlinux /usr/src/debug/kernel-5.10.10-4.1.x86_64/

[   68.387301] RIP: 0010:test_module_init+0x5/0xffa [test_module]

[   68.387301] RIP: 0010:test_module_init (/home/user/linux-5.10.5/test-module/test-module.c:16) test_module

Prepare and optimize the report¶

Prepare and optimize the report. [...]

Most developers just take a few seconds to skim a report before deciding between taking a closer look or moving on, as they receive a ton of messages. That is why the title/subject, the first sentence, and the three or four following it are crucial.

People will also stop reading if the report’s text is long or hard to follow; the same is true if crucial information is not at hand. So be sure to describe things as short, straightforward, and simple as possible while providing everything important.

How to do that is partly explained by the three documents linked to in the reference section’s intro. The next few subsections thus will only mention a few essentials as well as things specific to the Linux kernel.

Submit the report¶

Submit your report in the appropriate way, which depends on the outcome [...]

The step-by-step guide covers all the important details already.

[back to step-by-step guide]

Things to do after the report went out¶

Wait for reactions and keep the ball rolling until you can accept the outcome [...]

If your report was good and you are lucky, some developers might immediately spot what is causing the issue. They then might provide a fix for you to test, which you should do in a timely manner; afterwards they then send it out for integration into the mainline kernel while tagging it for backporting to affected stable and longterm kernels if appropriate.

But frequently it is a little less straightforward. That is why the job often is only starting once you send a report. What you’ll have to do depends on the situation. Here are a few tips:

Check who you deal with: Most of the time a developer for the particular area of code will respond. But as issues are usually reported in public, it could be anyone -- including people that want to help but in the end send you off track. That is why it might be wise to run a quick search on lore to see who you are interacting with.

Inquiries for data: Often you will be asked to test something or provide additional details. Try to supply the requested information soon, as you have the attention of someone willing to help and risk losing it the longer you wait; that outcome is even likely if you do not provide the information within a few business days.

Always reply in public! When you submitted the issue in a bug tracker, always reply there and do not contact any of the developers privately; for mailed reports, always use the ‘Reply-all’ function when replying to any emails you receive. That includes emails with any additional data you might want to add to your report: Go to your email application’s ‘Sent’ folder and use ‘reply-all’ on your email with the report. This approach will make sure the public mailing lists and everyone else who becomes involved later can access everything; it also keeps the email thread intact, which, among others, is really important for others that later run into the same problem and check the thread for a solution.

There are just two situations where a comment in a bug tracker or a ‘Reply-all’ is unsuitable:

• Someone tells you to send something privately.

• You were told to supply something containing information that should not be exposed to the public. In that case, it is okay to send it in private to the person who asked for it. But note in the ticket or an email that you did so to keep the public record straight.

Requests for testing: When you are asked to test a diagnostic patch or a possible fix, try to test it in a timely manner, too. But do it thoroughly and do not rush it: Mixing things up can happen easily and leads to a lot of confusion. A common mistake, for example, is thinking a proposed fix was applied when building a test kernel, when in fact it was not.

Try to help yourself before asking for help: During this part of the process someone might tell you to do something that requires a skill you might not have mastered yet. For example, you might be asked to use some test tools you have never heard of yet; or you are asked to apply a patch to the Linux kernel sources to test. It usually will be fine replying asking for instructions on how to do that. But before going that route, try to find the answer on your own by searching the internet; alternatively, consider asking elsewhere for advice. For example, ask a friend or post your question to a chat room or forum you normally hang out in.

Be patient: If you are really lucky, you might receive a reply to your report within a few hours. But most of the time it will take longer, as maintainers might be in a different time zone -- one where people currently take a few days off or already enjoy their night away from the keyboard. They might also simply be busy with other work, on a trip to a conference, or simply enjoying a long holiday.

[back to step-by-step guide]

Regularly check if the bug is still present¶

Retest at least every first release candidate (e.g., -rc1) of a new mainline version released and [...]

Every time the first pre-release of a new mainline kernel version is released (the ‘rc1’), go and check if the issue is fixed there or if anything of importance has changed. Mention the outcome in the ticket or in a mailed reply to your report (make sure to CCs everyone that up to that point participated in the discussion). This will show your commitment. It also tells developers the issue persists and acts as an implicit reminder.

An occasional retest at another time (for example, with -rc4 or -rc7) is also wise, but only report your results if something relevant changed or if you are writing anyway.

[back to step-by-step guide]

What to do when nothing of substance happens¶

If things stall for more than two to three weeks, evaluate why. It can happen due to good or bad reasons, like [...]

Sometimes you will not receive any reaction from the responsible developers; or a discussion around the issue evolves but ends fruitlessly.

In these cases, wait two to three weeks before sending a friendly reminder: Maybe the right developers were just away from their keyboards when you sent your report or had something more important to take care of.

When writing the reminder, kindly ask if there was anything wrong with the report or if anything from your side is needed to get the ball rolling. If the report was submitted by email, send a reply inserting your query after quoting the intro while including a full quote of the original report below: Then all the recipients will have both the gist of the problem and the details at hand immediately in convenient order.

After sending a reminder, wait three more weeks for replies. If you still don’t receive a proper reaction, reconsider your approach. Did you maybe try to reach out to the wrong people? Was the report possibly offensive or so confusing that people decided to stay away from it?

The best way to rule out such factors: Show the report to one or two people familiar with FLOSS issue reporting and ask for their opinion. Also ask them for their advice on how to move forward. That might mean preparing a better report and making those people review it before sending it out. Such an approach is totally fine; just mention that this is the second and improved report on the issue and include a link to the first report.

If the report was proper, you can send a second reminder; in it, ask for advice on why the report did not receive any replies. An ideal moment for this is shortly after retesting with the first pre-release of a new mainline release (the ‘rc1’), as you should retest and provide a status update at that point anyway (see above).

[back to step-by-step guide]

In most cases nobody is obliged to help¶

Be aware that nobody is obliged to help you, unless it is [...]

Developers ideally should react somehow to every issue report, but sometimes do not reply or, in the end, do not address problems. This is due to reasons [Why some bugs remain unfixed and some reports are ignored] explains in more detail, which also explains why some code does not even have maintainers.

Try to help yourself in that case. You, for example, could team up with others affected to then create a better report or narrow down the root cause of a problem. With a bit of luck, someone on the team might even know a bit about programming and provide a fix.

[back to step-by-step guide]

Why some bugs remain unfixed and some report are ignored¶

When reporting a problem to the Linux developers, be aware that they are only obliged to fix regressions, security issues, and severe problems. Developers, maintainers, or, if all else fails, Linus Torvalds himself will make sure of that. They will fix a lot of other issues as well, but sometimes they can’t or won’t help -- and sometimes there isn’t even anyone to send a report to.

This situation is best explained using kernel developers that contribute to the Linux kernel in their spare time. Quite a few of the drivers in the kernel were written by such programmers; often they simply wanted to make the hardware they owned usable on their favorite operating system.

These programmers most of the time will happily fix problems other people report. But nobody can force them to do so, as they are contributing voluntarily.

There are also situations where such developers would like to fix issues, but can’t: They might lack programming documentation to do so or hardware to test. The former can happen when the publicly available docs are superficial or when a driver was written with the help of reverse engineering.

Sooner or later, spare-time developers usually stop caring for the driver. Maybe their test hardware broke, was replaced by something more fancy, or became so old that it is something you don’t find much outside of computer museums anymore. Other times developers also stop caring when something different in life becomes more important to them. Then sometimes nobody is willing to take over the job as maintainer -- and nobody else can be forced to, as contributing is voluntary. The code nevertheless often stays around, as it is useful for people; removing it would also cause a regression, which is not allowed in Linux.

The situation is not that different with developers that are paid for their work on the upstream Linux kernel. Those contribute the most changes these days. But their employers set the priorities. And those sooner or later stop caring for some code or make their employees focus on other things. Hardware vendors, for example, earn their money mainly by selling new hardware -- they thus often are not much interested in investing much time and energy in maintaining a Linux kernel driver for a chip they stopped selling years ago. Enterprise Linux distributors often care for a longer time period, but in new versions might set support for old and rare hardware aside to limit the scope, too. Often spare-time contributors take over once employed developers orphan some code, but as mentioned earlier: Sooner or later they will usually leave the code behind, too.

Priorities are another reason why some issues are not fixed, as developers quite often are forced to set those: The spare-time of volunteers or the time employers allot for upstream Linux kernel work is often limited. Sometimes developers are also flooded with good and bad reports, even if a driver is working well. To not get completely stuck, the programmers might have no other choice than to prioritize bug reports and ignore some.

But do not worry too much about all of this, a lot of drivers have active maintainers who are quite interested in fixing as many issues as possible.

Why reporting Linux kernel bugs is somewhat complicated¶

The Linux kernel’s development model differs from typical Open Source projects in a few important aspects. Four of those complicate bug reporting:

1. Developers are free to solely focus on the latest mainline codebase.

2. The ‘stable team’ maintains the stable and longterm kernel series, but is not allowed to fix many bugs just there if they happen in mainline, too.

3. There is no central bug tracker.

4. Most kernels used in Linux distributions are totally unsuitable for reporting bugs upstream.

Due to the first aspect, some of the developers ignore or react coldly to reports about bugs in, say, Linux 6.1 when 6.2-rc1 is already out.

The combination of the first and the second aspect is why some of the developers are unwilling to look into reports with stable or longterm kernels: the problem might never have happened in the code they work on, for example because the stable team did something wrong between 6.1.1 and 6.1.2.

The stable team due to those two aspects is often in a similar but opposite situation when it comes to bugs reported using a kernel like 6.1.2: If that issue already happened in 6.1 and still happens in the latest mainline kernel, then it must be fixed there first. That is among the reasons why reporters in the end always have to check if mainline is affected, as the stable team often is the wrong point of contact, unless it is a series specific regression.

There are various reasons why no central bug tracker exists. They, among others, were not a thing yet when Linux started, which is why reporting my email was the norm initially -- and still is, as quite a few developers prefer to handle all aspects of kernel development via email. Some, on the other hand, saw benefits in trackers and set up bugzilla.kernel.org, which due to the aforementioned aspect never became mandatory and has some problematic aspects; this is why it frequently does not even forward newly filed reports to the appropriate developers. Yet other developers prefer the comfort of Git forges for development and issue tracking; but subsystems use various forges, so those trackers are spread over the web.

The fourth aspect is explained in the second point of the reference section already: Old codebases, modified sources, and add-ons lead to bugs that were fixed a long time ago already or never happened upstream in the first place. These are problems for many other software packaged by Linux distributions as well. But there it is usually a smaller problem, as the modifications and extensions distributors apply to the kernel are often much bigger and more dangerous; the kernel also changes way more quickly, is a lot more complex, and naturally more fragile. Due to these aspects, many developers expect reports to be based on fresh and vanilla kernels. Furthermore, most of them receive way more bug reports than they are able to handle, which is why they prioritize the ones that look more promising.

Reporting a bug due to these and other unmentioned aspects is harder than in other Free/Libre & Open Source Software projects -- the complexity of this document proves that. But that is the state of affairs for now. The primary author of this text hopes documenting it will lay some groundwork to improve the situation over time.