Kernels security

kernels downloads and loads remote code, so you need clear trust boundaries. We aim to address the following attack vectors:

• A kernel developer ships malicious code inside a kernel.
• The Hub credentials of a kernel developer are compromised and an attacker uses them to add malicious code to a kernel.

We do not aim to address the following attack vectors:

• Local Hub cache manipulation if the manipulation is done by the attacker on the system that loads the kernels. The reasoning here is that the attacker could do anything at this point, including tampering with the software that loads the kernel.

Improving kernel security is an ongoing effort, and we welcome feedback and contributions.

Malicious kernel developers

As a kernels user, ask two trust questions before you load a kernel:

• Do I trust the developer?
• Do I trust the developer to properly secure their infrastructure and credentials?

kernels also blocks accidental loads from publishers you have not opted into, such as from trusted publishers, the kernels-community, and kernel provenance.

Trusted publishers

A trusted publisher is a kernel developer who is deemed reputable by the Hugging Face kernels team. The trusted publisher status is queried from the Hub when downloading a kernel.

The set of trusted publishers is intentionally very small and generally only contains organizations that have years of experience in developing kernels and maintaining secure build infrastructure.

By default, a call such as

activation = get_kernel("someorg/activation", version=1)

will raise an exception unless someorg is a trusted publisher.

Pass trust_remote_code=True to allow any publisher, or pass a list of repo IDs to allow only those repos (plus trusted publishers). The default is False to always block unknown publishers.

activation = get_kernel(
    "someorg/activation",
    version=1,
    trust_remote_code=["someorg/activation"]
)

If your library loads kernels on behalf of its users, expose a flag so they can control untrusted publishers. For example:

TRUST_REMOTE_KERNELS = os.environ.get("MYLIB_TRUST_REMOTE_KERNELS", "0") == "1"

activation = get_kernel(
    "someorg/activation",
    version=1,
    trust_remote_code=["someorg/activation"] if TRUST_REMOTE_KERNELS else False
)

kernels-community

The kernels-community organization provides a large number of kernels that are maintained by Hugging Face. Kernels from this organization are publicly maintained through GitHub and are built using ephemeral build containers. PRs that are merged into kernels-community go through an automated security audit and can only be merged by a very small group of maintainers.

The kernels-community organization is a trusted publisher, so it provides good kernel coverage out of the box.

Kernel provenance

You can also check that a build came from the claimed source. Every kernel built with kernel-builder records provenance in the build variant’s metadata.json, including the Git commits of:

• The revision of kernel-builder that the kernel was built with.
• The revision of the kernel source itself.

This information is stored in the build variant’s metadata.json. For example:

"provenance": {
  "kernel-builder": {
    "version": "0.17.0-dev0",
    "commit": "a7f0afdb29a6a3372b1d47180cc0c182454c5e3b",
    "dirty": false
  },
  "kernel": {
    "commit": "a137a8498a30a98631a3deedfa44ea01bf1cef7c",
    "dirty": false
  }
}

For a full example, see the flash-attn3 kernel.

This provenance information can be used to rebuild the kernel at the exact same revisions of the kernel and the builder. A binary diff between the remote build and your local build can help reveal any tampering.

Kernel builds are largely reproducible because kernels are built using Nix inside a sandbox. kernel-builder pins the full toolchain (compiler, C library, etc.) in flake.lock. You can find these pins in the kernels repository.

Kernel compromises

Even if you trust a kernel developer, their credentials might be compromised. An attacker could use the credentials to upload a malicious version of a kernel. kernels offers two protections against these attack vectors:

• Kernel locking
• Code signing

Kernel locking

Kernel locking records the Git commit hash of a kernel in a kernels.lock file. After the kernel is locked, load it with get_locked_kernel (downloads if needed) or load_kernel (requires a pre-downloaded kernel and errors if it is missing). Either path only loads the locked commit. If an attacker compromises a kernel Hub repository and pushes a new, malicious version, it will not be used. See Lock kernel versions for how to lock the kernels of a project.

An attacker can circumvent the lock by trying to craft a commit that collides with the SHA-1 hash. However, this is currently hard, since the Hugging Face Hub platform uses a Git implementation with SHA-1 collision detection (sha1dc).

Code signing

kernels can verify kernels with cosign.

On load, kernels checks that the files match signed digests in metadata.json. Signing uses cosign with short-lived keys, and the signature is recorded in a ledger. That combination makes leaked CI signing keys much harder to reuse.

The builder computes the SHA-256 digest of each file in the kernel and stores it in metadata.json:

"digest": {
  "algorithm": "sha256",
  "files": {
    "__init__.py": "iY6XPtdZUgS8HizpndYTF+U/5kUmeLsADwCNM0UXMME=",
    "_ops.py": "5ecEPtZkkJLggvzM/luMYUmHIpCZXiBMEwXG0GmIMBU=",
    "_rmsnorm_xpu_89d4054.abi3.so": "yb8gSiUShkgjnrdDNTPJ0prhLPQAvTIFWwbllgqK5po=",
    "layers.py": "8KMmxy30Olm/16XW7vvHAwWNVWcnA8js0nNuoOSGEO8="
  }
}

Aside from the main signature, cosign also records information about how the signature was made, such as the OIDC issuer, the source repository, and the workflow path/branch.

Signature verification performs the following steps:

• Verify the signature against the given policy. The default policy only accepts kernels signed by workflows in the huggingface/kernels-community GitHub repository.
• Verify the authenticity of metadata.json using the signature.
• Use the digests in metadata.json to verify the kernel files.

At this time, a signature verification error will only result in a warning. Moreover, signature verification is only performed when the sigstore Python package is installed. However, we will make signature verification mandatory in the future.

The same steps can be performed on demand with the kernels verify-signature command.

Signature verification receipts

To avoid the high cost of signature verification, a kernel is only verified in full once. The first time a kernel is loaded, we perform all the steps above. Upon successful verification, we write a receipt file to the kernels cache.

When a receipt is found on a later load, the signature and digest checks are skipped. The signing certificate is still checked against the policy, since the receipt could have been written by a verification with a different policy.

Receipts are stored by kernel identity. The name of a receipt file is a hash of:

• The repo ID
• The Git commit hash
• The build variant

This means that downloading a different revision or build variant of the same kernel will not collide with the existing receipt and will be verified as expected.