Skip to content

Backend Key Provisioning Failed

URI: https://docs.ankatech.co/errors/backend-kek-provisioning-failed
HTTP Status: 422 Unprocessable Entity

When you see this

The per-tenant key-encryption-key phase did not complete, and waiting will not change the answer. It is raised by the activation of the deployment key-protection backend; by the eager key step of a tenant create and a key create (POST /api/v3/admin/tenants/{tenantId}/keys) when the tenant has no key-encryption key yet; and — for a tenant on an ANKA-managed Cloud KMS tier — by the tenant create, the key create and the Edition assign when the tier's backend answers durably, or no client for it could be built (then the detail says nothing was contacted). The declared backend is unchanged.

A backend that gave no usable answer in time, or asked the platform to slow down (UNREACHABLE, TIMED_OUT, THROTTLED), is not this type: on the deployment plane it is Backend Key Provisioning Unavailable (503, with Retry-After), and on a managed tier it is the 503 key-protection-backend-unavailable. Which is which is decided by the same classification every self-test verb reads.

One type, more than one situation — and the sentence in detail is not what tells them apart. extensions.reason is. It carries the closed SelfTestReason vocabulary, and the member is what a client and an operator branch on:

  • The backend was addressed, and the key phase failed there. REFUSED — it answered and declined the operation for a credential it accepted, on authorization or key state. AUTHENTICATION_REFUSED — it did not accept the credential. KEY_ABSENT — it answered that the declared key does not exist. ROUND_TRIP_MISMATCH — the unwrap did not return the bytes the wrap consumed. These are the members on which a create-key capability was exercised, and the ones the remedies below are written for.
  • Nothing was contacted. ROUTING_UNAVAILABLE — the platform could not build a protector for a reason on its own side of the boundary, so no round trip ran and no capability was exercised; it stays this 422 because admin-api builds that protector itself and there is no upstream to name. DESCRIPTOR_INVALID — on a PKCS#11 token, the module library could not be loaded into admin-api at all. KEK_NOT_CONFIGURED — the supplier named no key-encryption key, so no wrap ran and nothing is asserted about the backend at all. On these the backend neither proved nor declined anything, and there is nothing to widen on a cloud account or a token.

admin-api creates these keys, not the environment host. Bootstrap and per-tenant key-encryption keys are provisioned by admin-api's own provisioning service against the declared backend; init-hsm.sh's activation helpers generate no KEK, and create-env.sh creates no tenant KEK on any path. So a failure here is a refusal by the backend admin-api addresses — a cloud account or an HSM token — and is never fixed by re-running a host script to create the key by hand.

Adoption is by provenance, not presence: before generating, the platform resolves the deterministic KEK label against the backend and adopts an object already present under that label, reaching generation only when none exists. Lookups on that path enumerate with room for more than one result and refuse on ambiguity rather than picking one.

Adoption also checks what it adopts. On a PKCS#11 token an object found under the label is adopted only if the token confirms it to be a sensitive, non-extractable AES-256 key — the one shape this platform creates. An object of any other shape arrives here as REFUSED with a fixed detail that asserts no capability class and no round trip, because the object is examined before anything is generated. On a deployment that declares no descriptor the same condition is the typed 409 unfit-key-material-object, whose page carries the cause and the remedy.

On a PKCS#11 token the label lookup is only half of it. The bootstrap tenant UUIDs are fixed platform-wide, so those labels are byte-identical in every environment created against the same security domain; adoption therefore also requires the token to carry this environment's own ANKASECURE_ENV_CLAIM marker. A token carrying another environment's claim, or no claim beside pre-existing bootstrap-labelled keys, is not this problem type at all — it is 409 foreign-environment-claim, which refuses before any key is created. The five Cloud KMS families hold no such marker and the gate does not apply to them.

It is reachable on every declarable backend family. The deployment plane admits the four PKCS#11 HSM tokens (softhsm, nshield, luna, cloudhsm) as well as the five Cloud KMS backends, so the same refusal can name an HSM token that answered a round trip and then refused to create an object.

Common Causes

  • The credential is granted encrypt and decrypt but not key creation, so it passes the self-test and fails this phase.
  • The backend refused the creation or the credential (REFUSED, AUTHENTICATION_REFUSED).

  • The backend answered and reported that the declared key does not exist (KEY_ABSENT) — a wrong coordinate, not a transport fault. Reachable on every Cloud KMS backend, each from the provider's own not-found code (AWS KMS NotFoundException, Google Cloud KMS NOT_FOUND, an Azure HTTP 404), and on an external PKCS#11 token whose session authenticated but carries no object with the KEK label. Which verdict applies follows from the KEK supplier, not from the plane: where the supplier names no KEK at all the answer is KEK_NOT_CONFIGURED / NOT_APPLICABLE, because no wrap ran and nothing is asserted about the backend. On the deployment plane the platform provisions the KEK itself, so that is the normal answer there. The provider answers were observed live — a direct call on 2026-09-29 to each of AWS KMS, Google Cloud KMS and Azure Key Vault naming a key that does not exist returned exactly those codes — but no run through a platform plane has yet produced the KEY_ABSENT verdict; the mapping from answer to verdict rests on unit and contract tests.

    The provisioning path this page covers has been exercised live: on 2026-09-02 an Azure Managed HSM created one oct-HSM KEK per tenant, each with key_ops of exactly wrapKey and unwrapKey, each followed by a wrap/unwrap self-test, with one key version per tenant and no re-creation. A failure here is therefore a real refusal by the cloud account — most often the missing key-creation permission named above — and not an untested code path. - A key-ring or resource-policy constraint on the cloud account blocked the new key. A request-rate limit is THROTTLED and arrives as the 503, not here. - On a PKCS#11 token: the PIN identity mounted at /run/secrets/pkcs11_pin can wrap and unwrap but is not permitted to create objects on the partition, the partition is full, or the deterministic KEK label resolves to more than one object and the lookup refuses on the ambiguity rather than choosing.

Response Example

{
  "type": "https://docs.ankatech.co/errors/backend-kek-provisioning-failed",
  "title": "Backend Key Provisioning Failed",
  "status": 422,
  "detail": "The deployment key-protection backend answered a wrap/unwrap round trip but did not complete per-tenant key-encryption-key provisioning. The failing capability class is create-key. This deployment's declared backend is unchanged.",
  "instance": "/api/v3/admin/platform/setup/backend/activate",
  "timestamp": 1730000000,
  "extensions": {
    "reason": "REFUSED",
    "requestId": "1c4b8e60-7f2a-4d31-95ab-2e60f4d7c018"
  }
}

Content-Type: application/problem+json — the response follows RFC 9457 Problem Details.

How to Resolve

  1. Read extensions.reason first. It is the discriminator, not the sentence: it separates a permission refusal from a transport condition, from a timeout, from a declared key that does not exist — and it separates all of those from the case where nothing was contacted at all, where every step below is the wrong investigation.
  2. On ROUTING_UNAVAILABLE, DESCRIPTOR_INVALID or KEK_NOT_CONFIGURED, look at this deployment rather than at the backend: no protector could be built, the module could not be loaded, or no key-encryption key was named, so no request ever left the platform. Check that the declared backend's descriptor resolves and that its vendor module or client library is present and loadable in admin-api.
  3. On the members where the backend was addressed, grant the credential the key-creation permission on the target key ring or vault — or, on a PKCS#11 token, grant the PIN identity object-creation rights on the partition. There the round trip did prove wrap and unwrap, so widening the encrypt/decrypt policy is the wrong change.
  4. Re-run POST /api/v3/admin/platform/setup/backend/activate. Keys the failed attempt already created are adopted on provenance rather than re-created, and they are named on the failure audit row, so a partial run leaves nothing orphaned.

For full schema definitions, examples, and interactive testing, see the Developer Hub Reference.