Bootstrap Key-Encryption Key Belongs To A Different Backend

URI: https://docs.ankatech.co/errors/deployment-backend-kek-provider-mismatch
HTTP Status: 422 Unprocessable Entity

When you see this

POST /api/v3/admin/platform/setup/backend/activate found a bootstrap tenant that already holds an active key-encryption key, and that key belongs to a different backend from the one the deployment now declares. The activation is refused before the mismatched key is adopted.

Activation provisions one key-encryption key per bootstrap tenant, and a re-run after a partial failure adopts the keys the earlier attempt already created rather than contacting the backend for them again — that is what makes a partial run recoverable instead of leaving billable keys nobody names. Adoption is admissible only when the recorded key belongs to the declared backend. A key recorded against Google Cloud KMS cannot stand in for a key on AWS KMS: adopting it would leave the deployment declaring one backend while one of its tenants was wrapped under another, and that split is not detectable afterwards — once every bootstrap tenant holds a key the deployment reads as bound, permanently.

So the platform refuses instead, and the refusal is a statement about recorded state, not about your cloud account. Nothing was contacted for the mismatched key, and the response carries no reason member, because no backend answered anything.

Common Causes

• A first activation ran far enough to provision one bootstrap tenant's key and then failed — a quota, a key-ring policy or a permission — and the deployment was afterwards re-pointed at a different cloud provider.
• A backend was withdrawn and re-declared while a key-encryption-key row from the earlier backend was still standing, because the withdrawal left it standing on purpose (see below).
• The environment carries the bootstrap key-encryption-key rows seeded for a PKCS#11 environment and a Cloud KMS backend was declared over them. Those rows are real recorded custody, not residue.
• Two activations ran concurrently and one of them committed a key-encryption-key row on a different backend while the other was still provisioning. This is the only cause that reaches the second form below, and the only one under which the refusal leaves keys behind.

Response Example

The refusal has two forms, and they differ in one sentence: whether the run that refused had already created any key. The first is the normal one.

Refused on the pre-scan — nothing was provisioned

{
  "type": "https://docs.ankatech.co/errors/deployment-backend-kek-provider-mismatch",
  "title": "Bootstrap key-encryption key belongs to a different backend",
  "status": 422,
  "detail": "Bootstrap tenant 00000000-0000-0000-0000-000000000001 already holds an active key-encryption key recorded against provider 'gcpkms', while this deployment declares provider 'awskms'. That key was created by a different backend and cannot be adopted, so nothing was adopted and the declared backend is unchanged. No key-encryption key was provisioned. Withdraw the declaration (DELETE /api/v3/admin/platform/setup/backend/declare) to clear the rows it owns, then activate.",
  "instance": "/api/v3/admin/platform/setup/backend/activate",
  "timestamp": 1730000000,
  "extensions": {
    "requestId": "5f0b2d84-91ae-4c37-b6d1-08e3a5c9f742"
  }
}

Refused on the race backstop — keys WERE created first, and the detail says so

{
  "type": "https://docs.ankatech.co/errors/deployment-backend-kek-provider-mismatch",
  "title": "Bootstrap key-encryption key belongs to a different backend",
  "status": 422,
  "detail": "Bootstrap tenant 00000000-0000-0000-0000-000000000002 already holds an active key-encryption key recorded against provider 'gcpkms', while this deployment declares provider 'awskms'. That key was created by a different backend and cannot be adopted, so nothing was adopted and the declared backend is unchanged. 1 key-encryption key was created on the declared backend before this refusal and is named on the platform audit trail for this activation. Withdraw the declaration (DELETE /api/v3/admin/platform/setup/backend/declare) to clear the rows it owns, then activate.",
  "instance": "/api/v3/admin/platform/setup/backend/activate",
  "timestamp": 1730000000,
  "extensions": {
    "requestId": "9e1c47b0-2d63-4a15-8f7e-c30b6a92d541"
  }
}

Content-Type: application/problem+json — the response follows RFC 9457 Problem Details.

The detail names the tenant and both provider labels and no coordinate: a provider label is closed registry vocabulary, while a key ARN, a Cloud KMS resource path or an Azure account URL is deployment topology.

Nothing was adopted and no declaration was committed — on either form. The provenance of every bootstrap tenant is established BEFORE anything is provisioned, so on the pre-scan the refusal genuinely leaves nothing behind and the detail says so.

The second form is the exception, and it is why the sentence is composed rather than asserted. The provisioning phase holds no lock, and each tenant's key-encryption-key row is committed in its own transaction as it is created. A foreign row that a concurrent activation commits after this run scanned and while it was provisioning is caught by a backstop inside the loop — by which point this run has already created and committed real, billable keys. Telling you nothing was provisioned there would be worse than saying nothing, because it says there is nothing to reconcile.

The detail names the COUNT and never the keys. A kekRef is an AWS CMK ARN, a GCP CryptoKey path or an Azure key identifier — your own cloud account, not platform vocabulary — so the references ride the access-controlled REFUSED audit row for the activation instead, which names each key by tenant, name and reference. That is the same asymmetry backend-kek-provisioning-failed applies to the keys a failed provisioning run leaves behind.

How to Resolve

1. Read the deployment's current declaration with GET /api/v3/admin/platform/setup/backend/configuration and confirm which backend you intend to run.
2. If the declared backend is the one you want, the mismatched key is residue from an earlier attempt. Withdraw the declaration — DELETE /api/v3/admin/platform/setup/backend/declare — which clears the bootstrap key-encryption-key rows that belong to the backend being withdrawn and names on its audit row every key reference it stopped tracking, so you can delete those keys on your own cloud account. Then declare, save the coordinates, seal the credential and activate again.
3. If the detail says keys were created before the refusal, read the REFUSED audit row for that activation before doing anything else: it names every key the run created, by tenant, key name and key reference. The withdrawal in step 2 clears the rows the withdrawn backend owns and names what it stopped tracking, but the keys themselves live in your cloud account and only you can delete them.
4. If the mismatched key is the one you want, declare that backend instead, and activate on it.
5. If the refusal names a key recorded against pkcs11, the deployment already holds bootstrap custody on a hardware or software HSM. That is not residue and the withdrawal deliberately leaves it standing; a Cloud KMS backend is not available on that environment without recreating it.

For full schema definitions, examples, and interactive testing, see the Developer Hub Reference.