[Linux] deferred probe pending

deferred probe pending…

I am documenting this as I encountered a new log message during my work.

This occurred while upgrading to kernel version 6.6 to work on the DTB for ODROID-M1, and then rebooting after the kernel installation.

Loading, please wait...
Starting version 5:245.4-4ubuntu3+202310161612~focal
[    2.766964] rockchip-vop2 fe040000.vop: Adding to iommu group 2
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
[   13.028331] platform fdc20000.syscon:io-domains: deferred probe pending
[   13.028943] platform fe010000.ethernet: deferred probe pending
[   13.029461] platform fe0a0000.hdmi: deferred probe pending
[   13.029947] platform fe2b0000.mmc: deferred probe pending
Begin: Waiting for root file system ... Begin: Running /scripts/local-block ... done.
done.
Gave up waiting for root file system device.  Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT!  /dev/disk/by-uuid/8b892d53-d29b-447f-9a2e-0f9f3d52ec73 does not exist.  Dropping to a shell!


BusyBox v1.30.1 (Ubuntu 1:1.30.1-4ubuntu6.4) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)

The issue wasn’t too complex; clearly, the platform devices shown in the log failed to probe.
Still, I looked into the details.

According to this article:

When the kernel loads a driver, some drivers have specific dependencies on others.
At build time, it's impossible to know all dependencies between different hardware.
Since there are many drivers, the kernel cannot simply add them all in the correct order.

To resolve this, if a driver cannot be probe()‘d yet, it returns -EAGAIN to defer the probe.
These drivers are then managed in a specific list for retries.

However, looking at the kernel log, the drivers ultimately failed to load, leading to a BusyBox shell.

The log shows that the MMC driver failed to probe. Since the kernel couldn’t recognize the storage device where the root filesystem resides, it failed to boot.

[   13.029947] platform fe2b0000.mmc: deferred probe pending

First, I checked the DTB. The source is available on the Hardkernel GitHub.
Files: arch/arm64/boot/dts/rockchip/rk356x.dtsi and
arch/arm64/boot/dts/rockchip/rk3566-odroid-m1s.dts.

        sdmmc0: mmc@fe2b0000 {
                compatible = "rockchip,rk3568-dw-mshc", "rockchip,rk3288-dw-mshc";
                reg = <0x0 0xfe2b0000 0x0 0x4000>;
                interrupts = <GIC_SPI 98 IRQ_TYPE_LEVEL_HIGH>;
                clocks = <&cru HCLK_SDMMC0>, <&cru CLK_SDMMC0>,
                         <&cru SCLK_SDMMC0_DRV>, <&cru SCLK_SDMMC0_SAMPLE>;
                clock-names = "biu", "ciu", "ciu-drive", "ciu-sample";
                fifo-depth = <0x100>;
                max-frequency = <150000000>;
                resets = <&cru SRST_SDMMC0>;
                reset-names = "reset";
                status = "disabled";
        };

&sdmmc0 {
        bus-width = <4>;
        cap-sd-highspeed;
        cd-gpios = <&gpio0 RK_PA4 GPIO_ACTIVE_LOW>;
        disable-wp;
        pinctrl-names = "default";
        pinctrl-0 = <&sdmmc0_bus4 &sdmmc0_clk &sdmmc0_cmd &sdmmc0_det>;
        sd-uhs-sdr104;
        vmmc-supply = <&vcc3v3_sys>;
        vqmmc-supply = <&vccio_sd>;
        status = "okay";
};

To use the MMC driver, the following are required:
gpio0, sdmmc0_bus4, sdmmc0_clk, sdmmc0_cmd, sdmmc0_det, vcc3v3_sys, and vccio_sd.

Re-examining the logs, I noticed a commonality:

[   13.028331] platform fdc20000.syscon:io-domains: deferred probe pending
[   13.028943] platform fe010000.ethernet: deferred probe pending
[   13.029461] platform fe0a0000.hdmi: deferred probe pending
[   13.029947] platform fe2b0000.mmc: deferred probe pending

Among the requirements for the MMC device, one overlapped: io-domains.
I suspected the pmic side and checked the code.

&pmu_io_domains {
        pmuio1-supply = <&vcc3v3_pmu>;
        pmuio2-supply = <&vcc3v3_pmu>;
        vccio1-supply = <&vccio_acodec>;
        vccio2-supply = <&vcc_1v8>;
        vccio3-supply = <&vccio_sd>;
        vccio4-supply = <&vcc_3v3>;
        vccio5-supply = <&vcc_3v3>;
        vccio6-supply = <&vcc_3v3>;
        vccio7-supply = <&vcc_3v3>;
        status = "okay";
};

        rk809: pmic@20 {
                compatible = "rockchip,rk809";
                reg = <0x20>;
                ...
                regulators {
                        vdd_logic: DCDC_REG1 {
                        ...
                        vcc3v3_pmu: ... {
                        ...
                        vccio_sd: ... {
                        ...
        }

I then checked if the driver was properly loaded.

$ git grep "rockchip,rk809" | grep drivers
drivers/mfd/rk8xx-i2c.c:        { .compatible = "rockchip,rk809", .data = &rk809_data },

$ cat drivers/mfd/Makefile | grep rk8xx-i2c
obj-$(CONFIG_MFD_RK8XX)         += rk8xx-core.o
obj-$(CONFIG_MFD_RK8XX_I2C)     += rk8xx-i2c.o
obj-$(CONFIG_MFD_RK8XX_SPI)     += rk8xx-spi.o

As expected, CONFIG_MFD_RK8XX_I2C and CONFIG_MFD_RK8XX were missing in the kernel config.

I had used the config from kernel 6.1, but the config options changed in version 6.6.

In 6.1, it was used like this:

$ git checkout odroid-6.1.y
$ git grep "rockchip,rk809" | grep drivers
drivers/mfd/rk808.c:    { .compatible = "rockchip,rk809" },
drivers/mfd/rk808.c:        of_device_is_compatible(np, "rockchip,rk809")) {

$ cat drivers/mfd/Makefile | grep rk808
obj-$(CONFIG_MFD_RK808)         += rk808.o

This was the difference.

I added CONFIG_MFD_RK8XX_I2C=y and CONFIG_MFD_RK8XX=y to the config, rebuilt the kernel, and rebooted.

It now boots correctly.

+ Note that kernel 6.6 no longer uses .scmversion, which was used in 6.1.