Skip to content

Backend Self-Test Failed

URI: https://docs.ankatech.co/errors/backend-selftest-failed
HTTP Status: 422 Unprocessable Entity

When you see this

A deployment key-protection backend command was attempted, the backend answered, and its answer to the wrap/unwrap round trip under the credential in effect is durable: waiting will not change it. The command has a postcondition that was not reached, so it refuses instead of returning a verdict inside a 200 body. extensions.reason carries the closed self-test vocabulary, which can never contain an ARN, resource path, Azure account URL, region, account id or vendor message.

A backend that was contacted and gave no answer in time, or answered by asking the caller to slow down (UNREACHABLE, TIMED_OUT, THROTTLED), is not this type: it is Backend Self-Test Unavailable (503, with Retry-After). A round trip for which no client could be built at all (ROUTING_UNAVAILABLE) is this type, with a detail stating that nothing was contacted: on this plane admin-api builds the backend client itself, so the fault is a platform-side configuration one with no upstream to name — it never answers 502 here. The classes are the ones the BYOK bind and credential rotation use; only the routing case differs, because there core-api builds the client and the same reason is a 502 naming it by role.

It is reachable on every declarable backend family. The deployment plane admits the four PKCS#11 HSM tokens (softhsm, nshield, luna, cloudhsm) as well as the five Cloud KMS backends, so the same refusal covers an HSM token that answered and did not complete the round trip. On that family the credential in effect is not a sealed deployment credential — it is the token PIN the host mounts at /run/secrets/pkcs11_pin — so the remedy is the token, its coordinates or that PIN, never a credential call.

Common Causes

  • The backend did not accept the supplied credential (AUTHENTICATION_REFUSED — on a Cloud KMS an access key, secret, token, service-account key or client secret the provider rejects), or accepted it and declined the operation (REFUSED).
  • The backend answered and reported that the declared key does not exist (KEY_ABSENT). This is a distinct reason and not an unreachable endpoint: the host resolved, authenticated and replied, and the coordinate names nothing. The remedy is the declared key identifier, never the network path. It is reachable on every Cloud KMS backend, each from the provider's own not-found code (AWS KMS NotFoundException, Google Cloud KMS NOT_FOUND, an Azure HTTP 404). It is also reachable through the wrap/unwrap round trip on an external PKCS#11 token (luna, nshield, cloudhsm): the session authenticated and no object carries the KEK label. Note also which verdict applies where, and it follows from the KEK supplier rather than from the plane: where the supplier names no KEK at all, no wrap runs and the answer is KEK_NOT_CONFIGURED / NOT_APPLICABLE, which asserts nothing about the backend; KEY_ABSENT requires that a KEK was named, the wrap reached the backend, and the backend answered that the key does not exist. On the deployment plane the platform provisions the KEK itself, so an unconfigured key normally resolves to the first case. The provider answers were observed live — a direct call on 2026-09-29 to each of AWS KMS, Google Cloud KMS and Azure Key Vault naming a key that does not exist returned exactly those codes — but no run through a platform plane has yet produced the KEY_ABSENT verdict (the live Managed HSM runs of 2026-09-01 and 2026-09-02 both had a key that existed), so the mapping from answer to verdict rests on unit and contract tests.
  • A rotation or restore candidate reaches a different key-encryption key than the credential in effect, so the cross-credential round trip did not reproduce what it wrapped (ROUND_TRIP_MISMATCH).
  • The recorded descriptor cannot support a round trip (DESCRIPTOR_INVALID) — on a PKCS#11 token this includes a module library the JVM cannot load at all (a missing file, a wrong architecture, an unresolved native dependency).
  • The platform could not build a protector for the declared backend (ROUTING_UNAVAILABLE): a credential that will not unseal, a credential whose provider family does not match the declared token, a vendor client that will not construct, or — on a PKCS#11 token — a deployment topology the platform cannot read. Nothing was contacted.

  • On a PKCS#11 token, the activation self-test creates a throwaway key on the declared token, round-trips it and destroys it, so four of its outcomes are told apart by reason alone:

    • SECURITY_DOMAIN_ABSENT — the module loaded and answered, and the declared security domain (the token or slot this deployment binds to) is not present on it. Not UNREACHABLE, which says the token was not reached or did not answer, and not KEY_ABSENT, which says a key is missing from a domain that is present. The verdict never names the domain.
    • AUTHENTICATION_REFUSED — the security domain is present and refused the PIN; no operation was attempted. Not REFUSED, whose contract is that the backend answered and declined the operation. It requires a C_Login that failed with one of the six declared PIN return codes (CKR_PIN_INCORRECT, CKR_PIN_INVALID, CKR_PIN_LEN_RANGE, CKR_PIN_EXPIRED, CKR_PIN_LOCKED, CKR_USER_PIN_NOT_INITIALIZED) — a failed login is not by itself a refused credential.
    • UNREACHABLE — the module loaded and a session to the token could not be opened for a reason the classification does not type, or a C_Login failed with a return code outside that set, such as CKR_DEVICE_ERROR. This one is answered as the 503 Backend Self-Test Unavailable, not as this 422.
    • REFUSED — the token answered and declined to create the probe key, or its non-extractability could not be confirmed.

    The first two, and the UNREACHABLE case, are classified from typed facts — the exception type, the PKCS#11 operation name and the CKR_* return code — by one shared entry point that the activation self-test and the wrap/unwrap round trip both call, so the same fault reaches the same reason whichever produced it. No message text is matched.

    TIMED_OUT is not produced by this arm: the probe is deliberately not time-bounded.

Response Example

{
  "type": "https://docs.ankatech.co/errors/backend-selftest-failed",
  "title": "Backend Self-Test Failed",
  "status": 422,
  "detail": "The deployment key-protection backend did not complete a wrap/unwrap round trip under the supplied credential. The credential in effect is unchanged and nothing was activated.",
  "instance": "/api/v3/admin/platform/setup/backend/credential",
  "timestamp": 1730000000,
  "extensions": {
    "reason": "ROUND_TRIP_MISMATCH",
    "requestId": "6b3f9a12-84c7-4d0e-b5a3-71f2c8d94e60"
  }
}

Content-Type: application/problem+json — the response follows RFC 9457 Problem Details.

How to Resolve

  1. Read extensions.reason — it separates a rejected credential from a refused operation, from a declared key that does not exist (KEY_ABSENT), and from a candidate that reaches the wrong key. Those remedies are different, and KEY_ABSENT is the one that is resolved entirely in the declaration. On softhsm the missing-key case is not separated out.

  2. On a PKCS#11 activation, reason names the plane. (UNREACHABLE arrives as the 503, below.)

    • SECURITY_DOMAIN_ABSENT — create or attach the token or slot on the host, then re-run the activation. There is nothing to wait for: the module is answering and the domain is not there.
    • AUTHENTICATION_REFUSED — correct the token PIN, which on this family is the host's Docker Swarm secret at /run/secrets/pkcs11_pin and never a credential call. Stop retrying first — a repeated wrong PIN locks the token's user PIN on most appliances, and recovering needs the SO-PIN.
    • UNREACHABLE (the 503 Backend Self-Test Unavailable) — check the module path and the vendor library install. If the module answered and the login itself failed, the fault is at the device or the appliance link rather than in the credential; the reason does not assert one was refused.
    • REFUSED — check the token's policy for key generation and for a sensitive, non-extractable AES key.

    🔴 On a SoftHSM token bound from the console this check is the FIRST proof of the token — and it is no longer the only one afterwards. The host command init-hsm.sh <env> --activate runs a token cross-check and a host self-test of its own, but a console activation never runs that command, however the environment was created. The host command also skips both checks on a SoftHSM environment created without --hsm-provider, even though the environment creation already made that token. So those host checks have run on a SoftHSM token only when it was bound with that command after a create-env.sh --hsm-provider softhsm create; on every other SoftHSM bind nothing had proved the token usable before this activation ran. A failure is a finding about the token, not a regression in the activation. What follows the bind is the same on every route: the platform chooses how to probe the token from the backend you declared, not from an environment setting, so the admin service repeats this wrap/unwrap round trip each time it restarts and the verdict GET /api/v3/admin/platform/setup/status publishes is re-proved rather than reported as NOT_APPLICABLE.

    A refusal can also leave a probe key behind: the self-test creates a token object and a failure between creating it and destroying it cannot remove it. This 422 does not name it. The label rides undeletedSelfTestObjectLabel on the activation result, and a refusing self-test throws before any result is constructed — so on this path there is no response body to carry it. It is on the independently committed audit row instead, written before this refusal precisely so the refusal cannot roll it back, which is what makes the durable record total on both outcomes where the response is not. Read the label there and remove the object out of band. Every probe label carries the prefix ANKASECURE_SELFTEST_PROBE_, lives outside the KEK_TENANT_ namespace and can never collide with a key-encryption key. 3. Correct the credential or the backend coordinates and re-submit. The live sealed credential is never written before the round trip returns, so the credential in effect is unchanged and the transient candidate has been deleted. On a PKCS#11 token there is no credential to re-submit: correct the coordinates in the console under Platform Settings → Cryptography & Trust → HSM / Cloud KMS, where they are hot-reloaded and reach the running services without a redeploy, or correct the PIN secret on the host. 4. Rehearse coordinates with POST /api/v3/admin/platform/setup/backend/validate before committing them again. The rehearsal presents no credential and dials no security domain, so it can never answer SECURITY_DOMAIN_ABSENT or AUTHENTICATION_REFUSED.

For full schema definitions, examples, and interactive testing, see the Developer Hub Reference.