Breaking free from Bazel in the Pixel kernel pipeline
When I first got my Pixel 7 in March of this year, I didn't know what I'd be signing up for in the future.
I used to just use the prebuilt kernel trees provided by Google for building my ROMs because I found it convenient. Nonetheless, cyberknight777, a kernel developer, and a good friend of mine, got me to finally start building a custom kernel for the device.
And when I finally started on around June of this year, we stumbled upon tons of unexpected design choices. This blog covers our journey of building the gs201 kernel with just Make and a build script, and throwing Google's bazel in the bin.
The very start
We didn't do much, so we just cloned the specific tag of the ACK (Android Common Kernel) for
the device (android-gs-pantah-6.1) from Google's source, and started the build with the default build scripts that Google
gives its users - build_pantah.sh
What I noticed was that Google makes it looks like an AOSP build, it compresses verbose messages like you see while building the Kernel in pure make, and just acts as a script that does some magic in the background and prints some messages.
We instantly found this to be holding us back from our freedom. So I ran the following commands manually:
1. ARCH=arm64 make LLVM=/home/sidharthify/kernel/prebuilts/clang/host/linux-x86/clang-r487747c/bin/ O=out gki_defconfig
2. ARCH=arm64 make -j$(nproc) LLVM=/home/sidharthify/kernel/prebuilts/clang/host/linux-x86/clang-r487747c/bin/ O=out Image.lz4
And there we have it, we started the build with make. And no, this isn't the end, not even close.
I used this clang version because it corresponds to the version used in stock, we can see this by expanding the kernel info inside about phone.
After this, we just expanded on the script to build the modules as well. Modules, modules, oh modules... This will be the most important talking point of this whole blog.
Either way, the next thing we needed to do was build the modules as well. Because of course, this is a modern GKI (Generic Kernel Image) Android kernel. Heavy words, eh?
Generic Kernel Image
GKI does not mean that the kernel is smaller, simpler, or “minimal” in the usual sense. It means that the kernel is no longer fragmented.
Before GKI, the Android kernel stack looked roughly like this: Linux Stable → Android Common Kernel → Vendor Kernel → OEM Kernel. Each layer carried its own patches, its own ABI expectations, and its own long-term maintenance burden.
GKI collapses that stack.
With GKI, there is a single, Google-maintained core kernel image that is shared across devices. This kernel is generic in the literal sense: it contains support for a wide range of hardware and use cases, rather than being tightly tailored to a specific device.
Device-specific functionality is no longer implemented by modifying the core kernel. Instead, it is delivered through loadable kernel modules.
Storage drivers, WiFi, Bluetooth, NFC, display drivers, fingerprint drivers, and large parts of SoC-specific logic all live outside the core image as modules.
This is the key tradeoff of GKI. The core kernel is unified, but the system as a whole becomes more modular, more policy-driven, and more sensitive to how modules are built, installed, signed, and loaded.
Building the kernel image with make was only half the job.
The real work began when we had to replicate how Google builds, installs,
signs, and validates kernel modules across
system_dlkm, vendor_dlkm, and
vendor_kernel_boot.
From this point on, most failures were no longer obvious boot failures. They manifested as silent breakages, missing functionality, modules refusing to load, and VINTF rejecting the build outright.
Building the modules
We started by declaring the directories of the modules we want: (for now, this logic was heavily modified later on)
EXT_MODULES="private/google-modules/amplifiers/snd_soc_wm_adsp \
private/google-modules/amplifiers/cs35l41 \
private/google-modules/amplifiers/cs35l45 \
private/google-modules/amplifiers/cs40l25 \
private/google-modules/amplifiers/cs40l26 \
private/google-modules/amplifiers/drv2624 \
private/google-modules/amplifiers/tas256x \
private/google-modules/amplifiers/tas25xx \
private/google-modules/amplifiers/audiometrics \
private/google-modules/aoc \
private/google-modules/aoc/alsa \
private/google-modules/aoc/usb \
private/google-modules/bluetooth/broadcom \
private/google-modules/gps/broadcom/bcm47765 \
private/google-modules/gpu/mali_kbase \
private/google-modules/gpu/borr_mali_kbase \
private/google-modules/soc/gs/drivers/soc/google/s2mpu \
private/google-modules/soc/gs/drivers/soc/google/pkvm-s2mpu/common/hyp \
private/google-modules/soc/gs/drivers/soc/google/pkvm-s2mpu/pkvm-s2mpu \
private/google-modules/soc/gs/drivers/soc/google/pkvm-s2mpu/pkvm-s2mpu-v9 \
private/google-modules/soc/gs/drivers/soc/google/pt \
private/google-modules/soc/gs/drivers/soc/google/smra \
private/google-modules/soc/gs/drivers/soc/google/vh/kernel \
private/google-modules/soc/gs/drivers/soc/google/vh/kernel/cgroup \
private/google-modules/soc/gs/drivers/soc/google/vh/kernel/fs \
private/google-modules/soc/gs/drivers/soc/google/vh/kernel/metrics \
private/google-modules/soc/gs/drivers/soc/google/vh/kernel/mm \
private/google-modules/soc/gs/drivers/soc/google/vh/kernel/pixel_em \
private/google-modules/soc/gs/drivers/soc/google/vh/kernel/sched \
private/google-modules/soc/gs/drivers/soc/google/vh/kernel/thermal \
private/google-modules/soc/gs/drivers/spi \
private/google-modules/soc/gs/drivers/spmi \
private/google-modules/soc/gs/drivers/thermal \
private/google-modules/soc/gs/drivers/thermal/google \
private/google-modules/soc/gs/drivers/thermal/samsung \
private/google-modules/soc/gs/drivers/tty/serial \
private/google-modules/soc/gs/drivers/ufs \
private/google-modules/soc/gs/drivers/usb \
private/google-modules/soc/gs/drivers/usb/dwc3 \
private/google-modules/soc/gs/drivers/usb/gadget \
private/google-modules/soc/gs/drivers/usb/gadget/function \
private/google-modules/soc/gs/drivers/usb/host \
private/google-modules/soc/gs/drivers/usb/typec \
private/google-modules/soc/gs/drivers/usb/typec/tcpm \
private/google-modules/soc/gs/drivers/usb/typec/tcpm/google \
private/google-modules/soc/gs/drivers/video/backlight \
private/google-modules/soc/gs/drivers/watchdog \
private/google-modules/trusty \
private/google-modules/trusty/drivers/trusty \
private/google-modules/uwb/qorvo/dw3000/kernel \
private/google-modules/uwb/qorvo/qm35/qm35s \
private/google-modules/video/gchips \
private/google-modules/wlan/bcm4383 \
private/google-modules/wlan/bcm4389 \
private/google-modules/wlan/bcm4390 \
private/google-modules/wlan/bcm4398 \
private/google-modules/wlan/dhd43752p \
private/google-modules/wlan/wcn6740/cnss2 \
private/google-modules/wlan/wcn6740/wlan/qcacld-3.0 \
private/google-modules/wlan/wcn6740/wlan/qca-wifi-host-cmn/iot_sim \
private/google-modules/wlan/wcn6740/wlan/qca-wifi-host-cmn/qdf \
private/google-modules/wlan/wcn6740/wlan/qca-wifi-host-cmn/spectral \
private/google-modules/wlan/wlan_ptracker \
private/google-modules/sensors/hall_sensor \
private/google-modules/touch/synaptics/syna_c10 \
private/google-modules/touch/synaptics/syna_gtd \
private/google-modules/touch/focaltech/ft3658 \
private/google-modules/touch/focaltech/ft3683u \
private/google-modules/touch/fts/fst2 \
private/google-modules/touch/fts/ftm5_legacy \
private/google-modules/touch/fts/ftm5 \
private/google-modules/touch/goodix \
private/google-modules/touch/novatek/nt36xxx \
private/google-modules/touch/common \
private/google-modules/touch/common/usi \
private/google-modules/touch/sec \
private/google-modules/power/reset \
private/google-modules/power/mitigation \
private/google-modules/misc/sscoredump"
Yeah, long list. But either way, these were pulled directly from android.googlesource.com , by cloning the directories into a seperate folder outside the kernel.
After defining these, we build the modules roughly like so:
for EXT_MOD in ${EXT_MODULES}; do
ABS_EXT_MOD="${ROOT_DIR}/${EXT_MOD}"
EXT_MOD_REL=$(rel_path "${ABS_EXT_MOD}" "${KERNEL_DIR}")
mkdir -p "${OUT_DIR}/${EXT_MOD_REL}"
set -x
make -C "${ABS_EXT_MOD}" M="${EXT_MOD_REL}" KERNEL_SRC="${KERNEL_DIR}" \
O="${OUT_DIR}" "${TOOL_ARGS[@]}" "${MAKE_ARGS[@]}" modules_install
make -C "${ABS_EXT_MOD}" M="${EXT_MOD_REL}" KERNEL_SRC="${KERNEL_DIR}" \
O="${OUT_DIR}" "${TOOL_ARGS[@]}" ${MODULE_STRIP_FLAG} \
INSTALL_MOD_PATH="${MODULES_STAGING_DIR}" \
INSTALL_HDR_PATH="${KERNEL_UAPI_HEADERS_DIR}/usr" \
"${MAKE_ARGS[@]}" modules_install
set +x
doneThis... well, I later realized that this wasn't the best method to do this. But for now, I was getting my kernel to be built with the modules, so I was happy enough.
Why this approach was flawed
At first glance, this felt like the correct way to do things.
Google keeps most Pixel specific code under
private/google-modules/, outside the core kernel directories.
Bazel treats these as separate units. The natural assumption, then, was
that these modules needed to be built separately as well.
That assumption was wrong.
From Kbuild’s point of view, there is no meaningful distinction between
“kernel code” and “Google modules”. Anything that lives inside the kernel
source tree and contains proper Kbuild files is simply part of
the kernel build graph.
By treating these directories as external modules, I was effectively fighting Kbuild instead of using it.
The long EXT_MODULES list, the manual loop, the relative-path
juggling, all of that existed to maintain an artificial boundary that the
kernel build system does not recognize.
This worked, but only in the sense that forcing things to work often does.
What actually needed to happen
Once I understood this, the solution became obvious.
Instead of treating Google’s modules as out-of-tree components, they should be built the same way every other kernel module is built: as part of a single, unified kernel tree.
In practice, this meant removing the explicit
EXT_MODULES list entirely and letting Kbuild discover modules
through its normal obj-m and obj-y relationships.
Instead of invoking make modules separately for each directory,
the entire module set could be built in a single pass, with all modules
staged through one unified modules_install step.
In other words, stop telling Kbuild what to build, and let it do what it was designed to do.
The refactor
The script no longer loops over module directories or calls
make -C on individual paths. Instead, it builds the kernel and
all associated modules in one pass:
ROOT_DIR="/home/sidharthify/kernel"
KERNEL_DIR="${ROOT_DIR}/aosp"
LLVM_DIR="${ROOT_DIR}/prebuilts/clang/host/linux-x86/clang-r487747c/bin/"
OUT_DIR="${KERNEL_DIR}/out"
MODULES_STAGING_DIR="${OUT_DIR}/modules_staging"
TOOL_ARGS=(LLVM="${LLVM_DIR}")
# enter kernel tree
cd "${KERNEL_DIR}"
# build kernel image
make -j96 "${TOOL_ARGS[@]}" gs201_defconfig
make -j96 "${TOOL_ARGS[@]}" Image.lz4
# build in-kernel modules
make -j96 "${TOOL_ARGS[@]}" modules
make -j96 "${TOOL_ARGS[@]}" INSTALL_MOD_PATH="${MODULES_STAGING_DIR}" modules_installThis dramatically simplified the script, removing more than a hundred lines of fragile glue logic, and aligning the build flow with how upstream Kbuild expects to operate.
More importantly, it removed an entire class of subtle failures.
And?
This pattern repeated itself... a lot of times.
Google’s tooling makes the Pixel kernel look more complex than it actually is. Bazel enforces boundaries that are convenient for Google’s internal build infrastructure, but those boundaries do not exist at the kernel level.
Once the tooling is stripped away, what remains is a conventional (android) Linux kernel tree with conventional rules.
Finishing the script up
Up to this point, the kernel technically worked. It built, it booted, and the modules existed. But this was still not a real kernel in the Android sense.
What Bazel had been quietly handling all along was not just compilation,
but integration. Modules were not meant to simply exist as .ko
files copied into a directory. They needed to be sorted, stripped, grouped
into the correct partitions, indexed properly, and finally packaged into
filesystem images that Android actually understands.
Cleaning Up the Old Assumptions
The first visible change was simple: instead of trying to
update individual .ko files in an existing prebuilt tree,
the script now deletes all previously staged modules and copies over
everything that was just built.
This matters more than it sounds. Kernel modules are tightly coupled to the exact kernel image they are built against. Allowing old modules to linger creates “zombie” mismatches that are extremely hard to debug. Starting from a clean slate every time makes the result deterministic.
Introducing Proper DLKM Partitioning
#####
#####
# pack images
#####
#####
echo "packing images..."
# erofs for dlkms
mkfs.erofs -z lz4hc "${OUT_DIR}/vendor_dlkm.img" "${DLKM_STAGING}/vendor_dlkm"
mkfs.erofs -z lz4hc "${OUT_DIR}/system_dlkm.img" "${DLKM_STAGING}/system_dlkm"
# combine dtbs
DTB_PATHS=("${OUT_DIR}/arch/arm64/boot/dts/google/gs201" "${OUT_DIR}/google-devices/gs201/dts" "${OUT_DIR}/arch/arm64/boot/dts/google")
DTB_SOURCE=""
for path in "${DTB_PATHS[@]}"; do
if [ -d "$path" ] && compgen -G "$path/*.dtb" > /dev/null; then DTB_SOURCE="$path"; break; fi
done
cat "${DTB_SOURCE}"/*.dtb > "${OUT_DIR}/combined.dtb"
# generate dtbo
if [ -f "${MKDTBOIMG}" ]; then
DTBO_FILES=$(find "${DTB_SOURCE}" -name "*.dtbo")
if [ -n "${DTBO_FILES}" ]; then python3 "${MKDTBOIMG}" create "${OUT_DIR}/dtbo.img" ${DTBO_FILES}; fi
fi
# vendor ramdisk and boot image
( cd "${DLKM_STAGING}/vendor_kernel_boot" && find . | cpio -H newc -o 2>/dev/null | lz4 -l -12 --favor-decSpeed > "${OUT_DIR}/vendor_ramdisk.cpio.lz4" )
python3 "${MKBOOTIMG}" --vendor_ramdisk "${OUT_DIR}/vendor_ramdisk.cpio.lz4" --dtb "${OUT_DIR}/combined.dtb" --header_version 4 --vendor_boot "${OUT_DIR}/vendor_kernel_boot.img"
Modern Android kernels do not load all modules from a single location. Instead, modules are split across three partitions, each with different expectations and lifetimes:
vendor_kernel_boot contains modules that must be available
extremely early in the boot process. These are modules that need to be
loadable in recovery and during first-stage init, before the full userspace
is up.
On Pixel devices, modules that previously lived in
vendor_boot were moved into
vendor_kernel_boot.
Google does not document the exact reason for this change. It appears to be a Pixel-specific design decision related to boot flow and tooling, rather than a general Android requirement.
What makes this evolution clearer is how it unfolded across Pixel generations.
On the Pixel 6 series, Google shipped multiple
vendor_ramdisk fragments: a default fragment and a
dlkm fragment, with kernel modules stored in the latter.
This design allowed kernel modules to be separated from the rest of
vendor_boot.
However, vendor ramdisk fragments are treated as independently replaceable by flashing tools, which made it easy to accidentally remove required fragments and break early boot.
Starting with Pixel 7, Google stopped placing modules in
vendor_ramdisk entirely and introduced a dedicated
vendor_kernel_boot partition. This isolated early-boot modules
from ramdisk fragment handling and removed an entire category of failure
modes.
vendor_dlkm contains vendor-owned hardware modules. These are
SoC and device-specific drivers supplied by the vendor and loaded
dynamically at runtime, outside of the early boot path.
system_dlkm contains core GKI modules. These modules are treated
as an extension of the kernel itself under the GKI model. They are expected
to remain untouched by OEMs and are governed by Google’s kernel and KMI
policy, rather than vendor customization.
Bazel encodes this logic implicitly. Without it, the rules have to be made explicit.
The partition assignment is driven by an explicit priority order.
Modules listed for vendor_kernel_boot are placed there
unconditionally. If a module is not listed there, it is then evaluated
against vendor_dlkm. Any remaining modules fall back to
system_dlkm.
Stripping, Staging, and Dependency Resolution
Every module is stripped of debug symbols before being staged. This is not strictly required by Android, but it is the standard practice used by Google and most downstream kernel builders.
Unstripped modules are significantly larger, often ballooning to hundreds of megabytes, which is undesirable for production images. Stripping keeps module size reasonable and avoids unnecessary bloat in DLKM partitions.
Once modules are placed into their respective partition trees, the script
runs depmod separately for each partition root. This step is
critical.
Its job is to generate the metadata required for the kernel to understand the dependency structure of kernel modules at runtime.
This metadata is what allows the kernel and userspace tooling to determine load order, resolve symbol requirements, and handle inter-module relationships correctly.
Running it per-partition ensures that dependency information is generated against the exact set of modules visible within that partition root. This allows the kernel to correctly resolve inter-module dependencies at runtime, including cross-partition references, without leaking assumptions from unrelated module trees.
To make this work, the script temporarily provides depmod
with the same metadata files Bazel would normally supply:
modules.builtin, modules.order, and
System.map.
Once dependency generation is complete, all temporary artifacts are removed, leaving behind a clean, self-contained module tree.
Finally, each DLKM tree is packaged into an EROFS image using
mkfs.erofs, and the vendor_kernel_boot
is packaged with mkbootimg.
Building the kernel binary into AOSP
Now that the prebuilt kernel directory was built, I had to finally start up a build and get my phone booted, right? Well, yes, but here's where it gets good.
The kernel built. The images were picked up correctly. The build made it much further than before.
And then it failed.
Not with a compiler error, not with a missing module. But with a hard stop from VINTF.
VINTF, or the Vendor Interface, is Android’s way of enforcing compatibility between the kernel, vendor components, and userspace. It does not care whether your kernel boots. It cares whether your kernel is allowed to boot.
The error was exactly like so:
2025-12-12 07:53:36 - check_target_files_vintf.py - INFO : stderr: ERROR: files are incompatible: Runtime info and framework compatibility matrix are incompatible: No kernel entry found for kernel version 6.1 at kernel FCM version 6. The following kernel requirements are checked:
Minimum LTS: 4.19.191, kernel FCM version: 6
Minimum LTS: 4.19.191, kernel FCM version: 6, with conditionals
Minimum LTS: 4.19.191, kernel FCM version: 6, with conditionals
Minimum LTS: 4.19.191, kernel FCM version: 6, with conditionals
Minimum LTS: 4.19.191, kernel FCM version: 6, with conditionals
Minimum LTS: 4.19.191, kernel FCM version: 6, with conditionals
Minimum LTS: 4.19.191, kernel FCM version: 6, with conditionals
Minimum LTS: 4.19.191, kernel FCM version: 6, with conditionals
Minimum LTS: 5.4.86, kernel FCM version: 6
Minimum LTS: 5.4.86, kernel FCM version: 6, with conditionals
Minimum LTS: 5.4.86, kernel FCM version: 6, with conditionals
Minimum LTS: 5.4.86, kernel FCM version: 6, with conditionals
Minimum LTS: 5.4.86, kernel FCM version: 6, with conditionals
Minimum LTS: 5.4.86, kernel FCM version: 6, with conditionals
Minimum LTS: 5.4.86, kernel FCM version: 6, with conditionals
Minimum LTS: 5.4.86, kernel FCM version: 6, with conditionals
Minimum LTS: 5.4.86, kernel FCM version: 6, with conditionals
Minimum LTS: 5.10.43, kernel FCM version: 6
Minimum LTS: 5.10.43, kernel FCM version: 6, with conditionals
Minimum LTS: 5.10.43, kernel FCM version: 6, with conditionals
Minimum LTS: 5.10.43, kernel FCM version: 6, with conditionals
Minimum LTS: 5.10.43, kernel FCM version: 6, with conditionals
Minimum LTS: 5.10.43, kernel FCM version: 6, with conditionals
Minimum LTS: 5.10.43, kernel FCM version: 6, with conditionals
Minimum LTS: 5.10.43, kernel FCM version: 6, with conditionals
Minimum LTS: 5.10.43, kernel FCM version: 6, with conditionals
Minimum LTS: 5.10.107, kernel FCM version: 7
Minimum LTS: 5.10.107, kernel FCM version: 7, with conditionals
Minimum LTS: 5.10.107, kernel FCM version: 7, with conditionals
Minimum LTS: 5.10.107, kernel FCM version: 7, with conditionals
Minimum LTS: 5.10.107, kernel FCM version: 7, with conditionals
Minimum LTS: 5.10.107, kernel FCM version: 7, with conditionals
Minimum LTS: 5.10.107, kernel FCM version: 7, with conditionals
Minimum LTS: 5.10.107, kernel FCM version: 7, with conditionals
Minimum LTS: 5.10.107, kernel FCM version: 7, with conditionals
Minimum LTS: 5.10.107, kernel FCM version: 7, with conditionals
Minimum LTS: 5.10.107, kernel FCM version: 7, with conditionals
Minimum LTS: 5.15.41, kernel FCM version: 7
Minimum LTS: 5.15.41, kernel FCM version: 7, with conditionals
Minimum LTS: 5.15.41, kernel FCM version: 7, with conditionals
Minimum LTS: 5.15.41, kernel FCM version: 7, with conditionals
Minimum LTS: 5.15.41, kernel FCM version: 7, with conditionals
Minimum LTS: 5.15.41, kernel FCM version: 7, with conditionals
Minimum LTS: 5.15.41, kernel FCM version: 7, with conditionals
Minimum LTS: 5.15.41, kernel FCM version: 7, with conditionals
Minimum LTS: 5.15.41, kernel FCM version: 7, with conditionals
Minimum LTS: 5.15.41, kernel FCM version: 7, with conditionals
Minimum LTS: 5.15.41, kernel FCM version: 7, with conditionals
Minimum LTS: 5.15.0, kernel FCM version: 8
Minimum LTS: 5.15.0, kernel FCM version: 8, with conditionals
Minimum LTS: 5.15.0, kernel FCM version: 8, with conditionals
Minimum LTS: 5.15.0, kernel FCM version: 8, with conditionals
Minimum LTS: 5.15.0, kernel FCM version: 8, with conditionals
Minimum LTS: 5.15.0, kernel FCM version: 8, with conditionals
Minimum LTS: 5.15.0, kernel FCM version: 8, with conditionals
Minimum LTS: 5.15.0, kernel FCM version: 8, with conditionals
Minimum LTS: 5.15.0, kernel FCM version: 8, with conditionals
Minimum LTS: 5.15.0, kernel FCM version: 8, with conditionals
Minimum LTS: 5.15.0, kernel FCM version: 8, with conditionals
Minimum LTS: 6.1.0, kernel FCM version: 8
Minimum LTS: 6.1.0, kernel FCM version: 8, with conditionals
Minimum LTS: 6.1.0, kernel FCM version: 8, with conditionals
Minimum LTS: 6.1.0, kernel FCM version: 8, with conditionals
Minimum LTS: 6.1.0, kernel FCM version: 8, with conditionals
Minimum LTS: 6.1.0, kernel FCM version: 8, with conditionals
Minimum LTS: 6.1.0, kernel FCM version: 8, with conditionals
Minimum LTS: 6.1.0, kernel FCM version: 8, with conditionals
Minimum LTS: 6.1.0, kernel FCM version: 8, with conditionals
Minimum LTS: 6.1.0, kernel FCM version: 8, with conditionals
Minimum LTS: 6.1.0, kernel FCM version: 8, with conditionals
Minimum LTS: 6.1.0, kernel FCM version: 202404
Minimum LTS: 6.1.0, kernel FCM version: 202404, with conditionals
Minimum LTS: 6.1.0, kernel FCM version: 202404, with conditionals
Minimum LTS: 6.1.0, kernel FCM version: 202404, with conditionals
Minimum LTS: 6.1.0, kernel FCM version: 202404, with conditionals
Minimum LTS: 6.1.0, kernel FCM version: 202404, with conditionals
Minimum LTS: 6.1.0, kernel FCM version: 202404, with conditionals
Minimum LTS: 6.1.0, kernel FCM version: 202404, with conditionals
Minimum LTS: 6.1.0, kernel FCM version: 202404, with conditionals
Minimum LTS: 6.1.0, kernel FCM version: 202404, with conditionals
Minimum LTS: 6.1.0, kernel FCM version: 202404, with conditionals
Minimum LTS: 6.6.0, kernel FCM version: 202404
Minimum LTS: 6.6.0, kernel FCM version: 202404, with conditionals
Minimum LTS: 6.6.0, kernel FCM version: 202404, with conditionals
Minimum LTS: 6.6.0, kernel FCM version: 202404, with conditionals
Minimum LTS: 6.6.0, kernel FCM version: 202404, with conditionals
Minimum LTS: 6.6.0, kernel FCM version: 202404, with conditionals
Minimum LTS: 6.6.0, kernel FCM version: 202404, with conditionals
Minimum LTS: 6.6.0, kernel FCM version: 202404, with conditionals
Minimum LTS: 6.6.0, kernel FCM version: 202404, with conditionals
Minimum LTS: 6.6.0, kernel FCM version: 202404, with conditionals
Minimum LTS: 6.6.0, kernel FCM version: 202404, with conditionals
Minimum LTS: 6.12.0, kernel FCM version: 202504
Minimum LTS: 6.12.0, kernel FCM version: 202504, with conditionals
Minimum LTS: 6.12.0, kernel FCM version: 202504, with conditionals
Minimum LTS: 6.12.0, kernel FCM version: 202504, with conditionals
Minimum LTS: 6.12.0, kernel FCM version: 202504, with conditionals
Minimum LTS: 6.12.0, kernel FCM version: 202504, with conditionals
Minimum LTS: 6.12.0, kernel FCM version: 202504, with conditionals
Minimum LTS: 6.12.0, kernel FCM version: 202504, with conditionals
Minimum LTS: 6.12.0, kernel FCM version: 202504, with conditionals
Minimum LTS: 6.12.0, kernel FCM version: 202504, with conditionals
Minimum LTS: 6.12.0, kernel FCM version: 202504, with conditionals: Success
The Apparent Contradiction
On paper, the failure made sense. The Pixel 7 ships with API level 33. API level 33 maps to FCM level 7. FCM level 7 officially supports kernels up to 5.15.
But the kernel I was building was 6.1.141.
From VINTF’s perspective, this should never work. Except it clearly does.
Pixel 7 runs 6.1 kernels on the exact same API and FCM levels. There was no experimental configuration here. This was not undefined behavior. The stock kernel itself was 6.1.124.
Which meant one thing: the rules we were looking at were incomplete.
What was actually happening
At this point, we stopped trying to "fix" the error and started to trace it instead.
We dug through the VINTF validation logic and the code paths responsible for determining the effective kernel compatibility level. We even edited the code of the file that checks for the kernel requirements with VINTF and added more debugging. What we found was not a configuration flag, a manifest override, or a device-specific exception.
It was a heuristic.
Google’s build logic does not rely solely on the numeric kernel version.
It also inspects the kernel release string. When a specific prefix (android14-11) is
detected, the FCM kernel level is silently adjusted during
validation.
// To support devices without GKI, RuntimeInfo::fetchAllInformation does not report errors
// if kernel level cannot be retrieved. If so, fetch kernel FCM version from device HAL
// manifest and store it in RuntimeInfo too.
if (flags & RuntimeInfo::FetchFlag::KERNEL_FCM) {
Level deviceManifestKernelLevel = Level::UNSPECIFIED;
auto manifest = getDeviceHalManifest();
if (manifest) {
deviceManifestKernelLevel = manifest->inferredKernelLevel();
}
if (deviceManifestKernelLevel != Level::UNSPECIFIED) {
Level kernelLevel = mDeviceRuntimeInfo.object->kernelLevel();
if (kernelLevel == Level::UNSPECIFIED) {
mDeviceRuntimeInfo.object->setKernelLevel(deviceManifestKernelLevel);
} else if (kernelLevel != deviceManifestKernelLevel) {
LOG(WARNING) << "uname() reports kernel level " << kernelLevel
<< " but device manifest sets kernel level "
<< deviceManifestKernelLevel << ". Using kernel level " << kernelLevel;
}
}
}Snippet from VintfObject.cpp#671
In other words, compatibility was being inferred from metadata, not declared explicitly.
My custom kernel was functionally identical to Google’s, but it did not carry the expected release prefix. Because of that, VINTF did not recognize it as a Google-uprevved kernel.
Once that heuristic failed to trigger, VINTF treated the kernel like it would on any other device: it validated it strictly against the FCM level declared by the device configuration, without applying any Google-specific exceptions.
On Pixel devices, those exceptions exist for a reason. The Pixel 6 and 7 families were ported from older kernels to 6.1 late in their lifecycle. To make that work, Google relaxed the usual FCM checks when a known release signature is detected.
My kernel did not match that signature, so the exception was never applied. As a result, VINTF enforced the standard FCM rules, and the build failed.
Why this was so hard to see
This behavior is not documented.
Nothing in the manifests explains it. Nothing in the device configuration hints at it. If you approach the problem assuming the system is purely declarative, you will never find the answer.
The only way to uncover it is to read the code and observe how the system behaves when the metadata changes.
Once the prefix was restored and the kernel release string matched what Pixel devices expect, the VINTF error disappeared entirely.
Getting the kernel to fully boot
Now, everything was nearly done. I wasn't relying on bazel at all, and it was building normally. All I needed to check now, was whether it actually boots or not.
Well, not everything can go well, so no, it didn't. For a very specific reason. This actually completely hardbricked my phone too, and I had to format it and downgrade to Android 16 QPR0.
The reason turned out to be subtle, deeply tied to how GKI works, and completely invisible until runtime.
GKI, DLKM, and Why Modules Are Special
Under the GKI model, the kernel image and device-specific functionality are deliberately split.
The kernel image itself is generic and shared across devices, while most
hardware and feature-specific logic is delivered through loadable kernel
modules. These modules live on separate dynamic partitions inside the super
image, most notably system_dlkm and vendor_dlkm.
This split is the core idea behind GKI: allow the kernel image to evolve independently from vendor-specific components, without forcing every OEM to ship a fully custom kernel.
It is important to distinguish between different kinds of modules.
GKI modules are the modules placed in
system_dlkm. These are considered part of the GKI surface
itself and are treated as an extension of the kernel image. They are built
and signed by Google and are expected to follow the GKI Kernel Module
Interface (KMI) guarantees for a given release.
Because of this, the kernel image and its corresponding GKI modules are tightly coupled. While GKI exists to provide a stable interface, that stability is scoped and versioned. A kernel image expects a matching set of GKI modules built against the same KMI generation.
vendor_dlkm modules are different. These are vendor-owned,
device-specific drivers. They are not part of the GKI surface and do not
benefit from the same compatibility guarantees. In practice, they must be
built against the exact kernel they are intended to run on.
This is where many assumptions break down.
GKI allows the kernel image and GKI modules to be updated as a coherent, Google-controlled unit, while vendor modules remain tightly bound to the kernel they were built against. Android may store these pieces on different partitions, but at runtime the dependencies are very real.
Cross-Partition dependencies
Another important detail is that modules are not isolated by partition.
Modules in vendor_dlkm are allowed to depend on modules in
system_dlkm. These dependencies are recorded in
modules.dep during the build.
At runtime, when a vendor module is loaded, the kernel automatically
resolves these dependencies and loads the required GKI modules from
system_dlkm.
This only works if dependency metadata is correct and the modules can actually be loaded by the kernel.
Early module load errors
The first sign of trouble appeared very early during boot from pstore:
[ 0.408049][ T1] init: Failed to load kernel modulesDigging deeper revealed a specific failure:
[ 0.382399][ T122] init: Failed to insmod '/lib/modules/ufs-pixel-fips140.ko'
with args 'fips_first_lba=86406 fips_last_lba=86917 fips_lu=0 use_hw_keys=true':
Exec format error
This module is extremely sensitive and tightly coupled to the kernel
configuration. Disabling it via
CONFIG_SCSI_UFS_PIXEL_FIPS140=n allowed it to get
further in the boot process.
The device now booted, but major subsystems were missing.
It Booted, But Nothing Worked
WiFi, Bluetooth, NFC, UWB, and USB networking were all broken.
A quick lsmod showed that many critical modules were simply
not loaded, even though they had been built correctly.
Attempting to load them manually resulted in a flood of errors like:
bcmdhd4389: Unknown symbol cfg80211_register_netdevice (err -2)
bcmdhd4389: Unknown symbol rfkill_alloc (err -2)
bluetooth: Unknown symbol rfkill_register (err -2)
These errors are extremely specific. They do not indicate missing code. They indicate that the kernel refuses to link the module at load time.
Even more telling was this:
insmod rfkill.ko
insmod: failed to load rfkill.ko: Permission denied
Module signing
At this point, it became clear to cyberknight777.
The modules were present. Their dependencies were correct. The symbols existed in the kernel. But the kernel was refusing to load them.
On Android, core GKI kernel modules (system_dlkm) are not just built. They are signed.
The kernel enforces module signature verification. Any module that is not signed with the expected key is rejected at load time. This failure surfaces indirectly as “unknown symbol” errors or permission failures.
Bazel normally handles this silently.
By replacing Bazel, I had also removed the step that signs every module with the kernel’s build-time key pair.
Re-Signing the modules
The fix was straightforward once we understood the mechanisms
After staging the modules into their respective DLKM trees, every
.ko needed to be signed using the kernel’s own signing tools
and keys.
# re-sign modules
SIGN_FILE="${OUT_DIR}/scripts/sign-file"
SIGN_KEY="${OUT_DIR}/certs/signing_key.pem"
SIGN_CERT="${OUT_DIR}/certs/signing_key.x509"
if [ -f "${SIGN_FILE}" ] && [ -f "${SIGN_KEY}" ] && [ -f "${SIGN_CERT}" ]; then
find "${DLKM_STAGING}" -type f -name "*.ko" | while read -r module; do
"${SIGN_FILE}" sha1 "${SIGN_KEY}" "${SIGN_CERT}" "${module}"
done
fi
This was taken from kleaf's own build_system_dlkm function:
build_utils.sh#340
:
# Re-sign the stripped modules using kernel build time key
# If SYSTEM_DLKM_RE_SIGN=0, this is a trick in Kleaf for building
# device-specific system_dlkm image, where keys are not available but the
# signed and stripped modules are in MODULES_STAGING_DIR.
if [[ ${SYSTEM_DLKM_RE_SIGN:-1} == "1" ]]; then
for module in $(find ${SYSTEM_DLKM_STAGING_DIR} -type f -name "*.ko"); do
${OUT_DIR}/scripts/sign-file sha1 \
${OUT_DIR}/certs/signing_key.pem \
${OUT_DIR}/certs/signing_key.x509 "${module}"
done
fi
This step restores the trust relationship between the kernel and its modules.
Once the modules were properly signed, all subsystems loaded correctly. WiFi, Bluetooth, NFC, and USB came up without further changes.
Why this was so easy to miss
None of this is obvious if you approach the problem purely from a build perspective.
The kernel built. The modules built. The images flashed. The failure only appeared at runtime, and the error messages pointed in misleading directions.
This is another example of an implicit contract enforced by the Android kernel pipeline. Bazel does not just build code. It encodes policy.
Once that final piece was in place, the kernel booted cleanly, without disabling the signing and the modules fully loaded.
We did it!
This was never about proving that Bazel is bad, or that Google’s tooling is wrong. Bazel exists for real reasons, at real scale.
This was about understanding what that tooling actually does.
Once Bazel was removed, nothing magically became simpler. In fact, things became harder. Responsibilities that were previously implicit had to be made explicit. Policies had to be discovered, not assumed. Contracts had to be honored manually.
But in doing so, the system stopped being opaque.
What emerged was not a fragile hack, but a conventional Linux kernel build pipeline that behaves exactly as Android expects, as long as its rules are followed. The difficulty was never in building the kernel. It was in learning which parts of the process were technical, and which parts were policy.
A special thanks goes to cyberknight777, who helped me do all this, did a lot of the tracing, and stayed with me during the bringup.
The full build script developed during this process is available here: https://github.com/sidharthify/auto
The kernel source itself, including all related changes, lives here: https://github.com/sidharthify/android_kernel_google_gs201
The kernel is named syd. The name is not meant as branding. It is a reference to Syd Barrett, and more specifically to the idea of creation that is raw, unstable, and unconcerned with polish or approval.
This kernel exists because the system did not make sense until it was understood end to end. The name reflects that process more than the result.
If there is a takeaway from all of this, it is that most complexity in modern systems does not come from necessity. It comes from abstraction layers quietly encoding decisions on your behalf.
This was my longest blog yet, and I am glad it is. Thank you for reading!