How Kernel Prelinking Works on macOS Catalina (or not)

ChristianChristian on Mac, Little Snitch, Developer —

Update on October 30, 2019: This issue is fixed in macOS 10.15.1.

In this article we’d like to outline some technical details about how the installation of a kernel extension works on macOS Catalina, about potential pitfalls in this process, what can go wrong, and what currently unfortunately does go wrong.

It’s for those of you with some technical background, who want to know how things work. It explains the intricate paths that the code in third party kernel extensions takes until it finally ends up in the kernel. Almost all of the information presented here was reverse-engineered.

When your Mac starts up, the very first thing to happen is the loading of the macOS kernel. To make this work, there are a couple of related technologies that must play nicely together:

  1. Kernel prelinking. The kernel is the first component of the operating system to start. It has no other tools available. In particular there is no way to check code signatures, and all file system access is very hard at this point. Apple therefore decided to prelink the bare kernel with all kernel extensions every time the kernel or one of the extensions is updated, and to start only that prelinked kernel at boot time.
  2. Read-only system volume. Apple decided to store the operating system on a read-only volume in order to prevent tampering by malware. The prelinked kernel is also stored on this read-only system volume.
  3. Updates of the prelinked kernel. Since the prelinked kernel is on a read-only volume, it cannot be updated directly. Apple had to conceive a new mechanism for updates.

Prelinking the Kernel

Prelinked kernels are built by /usr/sbin/kextcache. This tool links the kernel at /System/Library/Kernels/kernel with kernel extensions from /System/Library/Extensions/ and /Library/Extensions/, checking code signatures and other prerequisites. The resulting prelinked kernel is written to /Library/Apple/System/Library/PrelinkedKernels/prelinkedkernel, a path which is on a writable volume and which is under System Integrity Protection (SIP) to prevent tampering.

Every time the directory /Library/Extensions/ is touched, the kextd daemon starts kextcache to build a new kernel.

However, the boot procedure does not use this new kernel. It uses the kernel at /System/Library/PrelinkedKernels/prelinkedkernel, which is on the read-only system volume. The kernel must be somehow copied to the read-only volume.

Copying to the System Volume

In addition to building the new prelinked kernel, kextcache installs a shell script in /var/install/shove_kernels. This script contains a call to /usr/sbin/kcditto, a tool which copies the kernel to its final destination at /System/Library/PrelinkedKernels/prelinkedkernel.

But the original problem still exists: The final destination is a read-only volume and SIP disallows remounting it in read/write mode. So when should the system run shove_kernels?

The best time is immediately before system shutdown. When you reboot or shut down your machine, launchd stops all processes. Then it remounts the system volume in read/write mode. This is possible because launchd has the entitlement com.apple.private.apfs.mount-root-writeable-at-shutdown. Then it runs /var/install/shove_kernels to copy the new kernel. All should be fine now.

Where does it fail?

The procedure outlined above fails on Catalina Beta 6 and newer, at least up to the public 10.15.0 release. In the last step, the kernel is not copied. It’s hard to debug the problem because the copying happens at a time when all system services have been shut down and log messages are no longer written to disk.

We have configured our test machine for verbose logging, and even then it’s very hard to check the logs. First, verbose logging does not work reliably. Sometimes it just does not switch to verbose mode or the screen turns black before log messages are written. And even if there is a verbose log, the font is tiny and a screen full of text appears for fractions of a second. We captured it with a camera and found the following messages:

...
bash: /private/var/install/shutdown_installer_tasks: No such file or directory
Thu Oct 10 11:00:31 2019 Catalina-Mac com.apple.xpc.launchd[1] <Notice>: shutdown_installer_taksks: optional boot task not present
bash: /private/var/install/deferred_install: No such file or directory
apfs_vfsop_mount:1125: Updating mount from /dev/disk1s7 to read/write mode is not allowed
Thu Oct 10 11:00:31 2019 Catalina-Mac com.apple.xpc.launchd[1] <Notice>: deferred_install: optional boot task not present
Thu Oct 10 11:00:31 2019 Catalina-Mac com.apple.xpc.launchd[1] <Error>: prepare_deferred_kcinstall: mounting root as read-write failed: 1 - Operation not permitted
System is shutting down. allowing script at path: /private/var/install/shove_kernels
System is shutting down. (SIP is enabled) allowing process at path: /usr/sbin/kcditto
Copying kernel: /Library/Apple/System/Library/PrelinkedKernels/prelinkedkernel -> /System/Library/PrelinkedKernels
/BuildRoot/Library/Caches/com.apple.xbs/Sources/kext_tools/kext_tools-623.11.5/rosp_staging.m.39: BOMCopier file error: 30 at /System/Library/PrelinkedKernels/prelinkedkernel
/BuildRoot/Library/Caches/com.apple.xbs/Sources/kext_tools/kext_tools-623.11.5/rosp_staging.m.248: Error moving prelinked kernel: srcPath = /Library/Apple/System/Library/PrelinkedKernels/prelinkedkernel, dstPath = /System/Library/PrelinkedKernels, error 1
/BuildRoot/Library/Caches/com.apple.xbs/Sources/kext_tools/kext_tools-623.11.5/rosp_staging.m.152: Error copying kernels...
/BuildRoot/Library/Caches/com.apple.xbs/Sources/kext_tools/kext_tools-623.11.5/kcditto_main.m.45: Error copying kernels (standalone)...
Thu Oct 10 11:00:31 2019 Catalina-Mac com.apple.xpc.launchd[1] (com.apple.xpc.launchd.domain.system) <Notice>: Userspace teardown took 5857 ms
...

We can see that remounting the system volume in read/write mode fails, in spite of the entitlement. The problem does not occur if SIP is disabled. This is an obvious bug, either in launchd or in the entitlements subsystem.

What can we do?

Not upgrade kernel extensions until Apple fixes the issue

This is the easiest workaround. Just wait until Apple has fixed the issue and then upgrade. We have changed our software update feed for Little Snitch to hide updates for Catalina users until Apple has fixed the problem. All those who have already upgraded Little Snitch and the kernel extension version is out-of-sync, please downgrade to the same version as linked into the current kernel. See this FAQ article for details.

Update the kernel in macOS Recovery

Since SIP does not apply in macOS Recovery, you can boot into this mode and trigger a kernel update there:

  1. Restart your system in macOS Recovery. Learn more
  2. If you have enabled FileVault to encrypt the contents of your system volume, you first have to mount that volume: Open “Disk Utility”, select your system volume in the sidebar and click the “Mount” button in the toolbar. Please be patient – mounting FileVault volumes may take quite a while. Once the volume is mounted, quit the “Disk Utility” application.
  3. Open “Terminal” from the Utilities menu in the menu bar.
  4. Enter the following command: touch -c "/Volumes/Macintosh HD/System/Library/Extensions" Important Note: If your system volume has a different name than “Macintosh HD”, replace this name with the actual name of the volume on which macOS Catalina is installed.
  5. Wait about 10 seconds. Then choose “Restart” from the Apple menu in the menu bar to restart your computer. Shutting down can take up to a few minutes because the system is rebuilding the boot cache in the background. Note that during this time no progress indication is shown.

Disable SIP

This is not recommended. But since the error occurs when launchd tries to remount the system volume in read/write mode and this limitation is a part of SIP, the update succeeds when SIP is disabled. There are no step-by-step instructions from Apple, but the Internet is full of instructions for how to disable SIP.