Detached Signatures & Streaming Explained

Target Audience: Developers, Integration Engineers

Reading Time: 12 minutes

Prerequisites: Basic understanding of digital signatures, JSON

Executive Summary

Detached signatures separate the signature from the signed content, enabling efficient streaming operations for large files. ANKASecure implements detached JWS (JSON Web Signature) and detached JWE (JSON Web Encryption) formats with O(1) constant memory usage, supporting unlimited file sizes.

Why detached? Traditional embedded signatures require loading entire files into memory. Detached mode processes data in small chunks, making it practical for multi-GB files.

Table of Contents

1. Problem: Embedded Signatures Don't Scale
2. Solution: Detached Content Mode
3. Detached JWS (Digital Signatures)
4. Detached JWE (Encryption)
5. Streaming Architecture
6. Use Cases
7. Implementation Examples
8. Performance Comparison

Problem: Embedded Signatures Don't Scale

Traditional JWS (Compact Mode)

Structure:

base64url(header) . base64url(payload) . base64url(signature)

Example:

eyJhbGciOiJQUzI1NiIsImtpZCI6Im15LWtleSJ9.eyJkYXRhIjoidGhpcyBpcyBteSBwYXlsb2FkIn0.dBjftJeZ4CVP...

Problem with large files:

1. Base64 overhead: 33% size increase (binary → base64url encoding)
2. Memory usage: Entire file must fit in memory
3. HTTP limitations: Tokens >8KB cannot be sent in HTTP headers
4. Processing delay: Cannot start verification until full token received

Example: 1GB file

Practical limit: ~5 MB for compact JWS tokens (ANKASecure enforced limit).

Solution: Detached Content Mode

Detached Signature Concept

Key idea: Signature and payload are transmitted/stored separately.

Structure:

• Signature object (small, ~200 bytes JSON)
• Payload (any size the path to the API accepts, streamed separately; a proxy in front of the deployment may cap uploads; see Behind a buffering proxy)

Benefits:

"No API limit" is the API's own bound; a proxy in front of the deployment may cap uploads; see Behind a buffering proxy.

How It Works

Signing process:

1. Read payload in chunks (8 KiB buffer)
2. Update signature algorithm with each chunk
3. Finalize signature → Output signature JSON
4. Payload streamed separately (not modified)

Verification process:

1. Parse signature JSON (get algorithm, kid)
2. Stream payload in chunks (8 KiB buffer)
3. Update verification algorithm with each chunk
4. Compare computed signature with provided signature
5. Result: Valid or Invalid

Key point: Payload never fully loaded into memory.

Detached JWS (Digital Signatures)

Format Specification

Signature object (JSON) — the flattened {"protected","signature"} body, captured from a real POST /api/v3/crypto/stream/sign round-trip with the RSA key my-rsa-3072-key (the signature value is cut after 64 characters, …):

{
  "protected": "eyJhbGciOiJQUzI1NiIsImI2NCI6ZmFsc2UsImNyaXQiOlsiYjY0Il0sImtpZCI6Im15LXJzYS0zMDcyLWtleSIsInZlciI6IjIuMCIsImFua19rdiI6MSwidGlkIjoiN2QzYTljMWUtNGIyZi00ZThhLTljNzMtMjc1MDAwMDAwMDExIn0",
  "signature": "p8dT_3fDTGUGf4ppiRwVkeTOi2sLjPw9r-hgbJn1VvLCYvvv9OPaUYTn_bLrL9V9…"
}

Decoded protected header:

{
  "alg": "PS256",
  "b64": false,
  "crit": ["b64"],
  "kid": "my-rsa-3072-key",
  "ver": "2.0",
  "ank_kv": 1,
  "tid": "7d3a9c1e-4b2f-4e8a-9c73-275000000011"
}

Every key signs this shape: SIMPLE and composite detached signatures alike are this flattened body and carry b64: false and crit: ["b64"] (RFC 7797 §3, §6). The protected headers are otherwise not alike. Both carry kid, ank_kv and, when the local clock stamps it (the default), iat, plus exp when the request set validityDays; a composite header names the pairing in alg and carries nothing else, while a SIMPLE header also carries ver and tid. A composite detached JWS carries one signature, the composite signature. RSA keys sign PS256: RSASSA-PSS with SHA-256, MGF1 with SHA-256 and a 32-octet salt (RFC 7518 §3.5).

Payload: Transmitted separately (file, stream, HTTP multipart, etc.)

Signature Computation

Signing input (RFC 7797 §3, b64: false):

ASCII(BASE64URL(UTF8(JWS Protected Header)) || '.') || JWS Payload

The payload octets enter the signing input as they are, not base64url-encoded, so the stream never has to be encoded.

For detached mode:

1. Feed the encoded protected header, ., then the payload octets as they stream into the signature algorithm (for a composite key, into its pre-hash)
2. Sign with the private key
3. Base64url-encode the signature
4. Output JSON with protected header and signature

Verification Process

Steps:

1. Parse signature JSON
2. Base64url-decode protected header
3. Extract algorithm (alg) and key identifier (kid)
4. Retrieve public key for kid
5. Stream payload in chunks:
   • Read chunk (8 KiB)
   • Update hash algorithm
   • Repeat until end of stream
6. Finalize hash
7. Verify signature against hash using public key
8. Return validation result

ANKASecure Endpoint

Sign (detached):

POST /api/v3/crypto/stream/sign

Headers:
  Content-Type: multipart/form-data
  Authorization: Bearer {jwt}

Request body (multipart) - IN THIS ORDER:
  1. metadata (application/json): {"kid": "my-signing-key"}
  2. file     (application/octet-stream): [binary stream]

Response (captured, `signature` cut after 64 characters):
{
  "protected": "eyJhbGciOiJQUzI1NiIsImI2NCI6ZmFsc2UsImNyaXQiOlsiYjY0Il0sImtpZCI6Im15LXJzYS0zMDcyLWtleSIsInZlciI6IjIuMCIsImFua19rdiI6MSwidGlkIjoiN2QzYTljMWUtNGIyZi00ZThhLTljNzMtMjc1MDAwMDAwMDExIn0",
  "signature": "p8dT_3fDTGUGf4ppiRwVkeTOi2sLjPw9r-hgbJn1VvLCYvvv9OPaUYTn_bLrL9V9…"
}

Verify (detached):

POST /api/v3/crypto/stream/verify

Headers:
  Content-Type: multipart/form-data
  Authorization: Bearer {jwt}

Request body (multipart) - IN THIS ORDER:
  1. metadata (application/json): {"detachedJws": "{\"protected\":\"eyJhbGciOiJQUzI1NiIsImI2NCI6ZmFsc2Us…\",\"signature\":\"p8dT_3fDTGUGf4ppiRwVkeTOi2sLjPw9r-hgbJn1VvLCYvvv9OPaUYTn_bLrL9V9…\"}"}
  2. file     (application/octet-stream): [binary stream]

Response:
{
  "valid": true,
  "keyRequested": "my-signing-key",
  "algorithmUsed": "ML-DSA-65",
  "signatureResults": [
    { "kid": "my-signing-key", "algorithm": "ML-DSA-65", "verified": true }
  ]
}

Example Use Case: Signing a 5GB Video File

Compact mode (impossible):

• ❌ Cannot base64-encode 5 GB into memory
• ❌ Would create ~6.65 GB token
• ❌ No streaming possible

Detached mode (streaming):

• ✅ Sign 5 GB file with 8 KiB memory buffer
• ✅ Signature output: ~200 bytes JSON
• ✅ Original file transmitted unchanged
• ✅ Verification also streams with 8 KiB buffer

Detached JWE (Encryption)

Format Specification

JWET (JWE Token) Header (JSON):

{
  "protected": "eyJhbGciOiJSU0EtT0FFUC01MTIiLCJlbmMiOiJBMjU2R0NNIiwia2lkIjoibXktZW5jcnlwdGlvbi1rZXkifQ",
  "encrypted_key": "OKOawDo13gRp2ojaHV7LFpZcgV7T6DVZKTyKOMTYUmKo...",
  "iv": "48V1_ALb6US04U3b",
  "tag": "XFBoMYUZodetZdvTiFvSkQ"
}

Decoded protected header:

{
  "alg": "RSA-OAEP-512",
  "enc": "A256GCM",
  "kid": "my-encryption-key"
}

Ciphertext: Transmitted separately as binary stream (not base64-encoded).

Encryption Process

Steps:

1. Generate random symmetric key (CEK - Content Encryption Key)
2. Generate random IV (Initialization Vector)
3. Encrypt CEK with recipient public key → encrypted_key
4. Stream plaintext in chunks:
   • Read chunk (8 KiB)
   • Encrypt chunk with AES-GCM using CEK
   • Write encrypted chunk to output stream
   • Repeat until end of plaintext
5. Finalize AES-GCM → authentication tag
6. Output JWET header (JSON) + ciphertext stream (binary)

Decryption Process

Steps:

1. Parse JWET header JSON
2. Base64url-decode protected header
3. Extract algorithm (alg, enc) and key identifier (kid)
4. Retrieve private key for kid
5. Decrypt encrypted_key using private key → CEK
6. Stream ciphertext in chunks:
   • Read encrypted chunk (8 KiB)
   • Decrypt chunk with AES-GCM using CEK and IV
   • Write plaintext chunk to output stream
   • Repeat until end of ciphertext
7. Verify authentication tag
8. Return plaintext stream

ANKASecure Endpoint

Encrypt (detached):

POST /api/v3/crypto/stream/encrypt

Headers:
  Content-Type: application/octet-stream
  Authorization: Bearer {jwt}
  X-Key-Id: my-encryption-key

Request body:
  [binary plaintext stream]

Response (multipart):
  Part 1 (application/json):
    {
      "protected": "eyJhbGciOiJNTC1LRU0tNzY4IiwiZW5jIjoiQTI1NkdDTSIsImtpZCI6Im15LWtleSJ9",
      "encrypted_key": "T0tPYXdEbzEz...",
      "iv": "NDhWMV9BTGI2VVMwNFUzYg",
      "tag": "WEZCb01ZVVpvZGV0WmR2VGlGdlNrUQ"
    }
  Part 2 (application/octet-stream):
    [binary ciphertext stream]

Decrypt (detached):

POST /api/v3/crypto/stream/decrypt

Headers:
  Content-Type: multipart/form-data
  Authorization: Bearer {jwt}

Request body (multipart) - IN THIS ORDER:
  1. header (application/json): {"protected": "...", "encrypted_key": "...", "iv": "...", "tag": "..."}
     ^^^^^^ NOT `metadata` - decrypt/reencrypt/decrypt-verify name their small part `header`
  2. file   (application/octet-stream): [binary ciphertext stream]

Response:
  [binary plaintext stream]

Example Use Case: Encrypting a 10GB Database Backup

Compact mode (impossible):

• ❌ Cannot load 10 GB into memory
• ❌ Would create ~13.3 GB base64-encoded ciphertext
• ❌ No streaming possible

Detached mode (streaming):

• ✅ Encrypt 10 GB backup with 8 KiB memory buffer
• ✅ JWET header: ~400 bytes JSON
• ✅ Ciphertext: 10 GB (no size overhead)
• ✅ Decryption also streams with 8 KiB buffer

Streaming Architecture

Memory Model

Single-pass streaming:

Input Stream (file/network)
    ↓
Buffer (8 KiB)
    ↓
Cryptographic Engine (Bouncy Castle)
    ↓
Output Stream (file/network)

Memory usage: O(1) constant - only 8 KiB buffer in memory at any time.

Throughput: Limited by I/O speed, not memory.

Dual-Thread Pipeline (Advanced)

For re-encryption/re-signing:

Thread 1: Decrypt/Verify
    Input Stream → Buffer (2 MiB) → Crypto Engine → Plaintext Queue
                                                            ↓
Thread 2: Encrypt/Sign                                      ↓
    Plaintext Queue → Buffer (2 MiB) → Crypto Engine → Output Stream

Memory usage: O(1) constant - 2 MiB per thread (4 MiB total).

Benefit: Parallelizes decryption and encryption for better CPU utilization.

See: Streaming Architecture Overview^*

Supported Operations

Use Cases

1. Large File Signing (Video, Backups, Logs)

Scenario: Sign 100 GB video file before archiving.

Detached signature workflow:

1. Stream video file through signing API
2. Receive signature JSON (~200 bytes)
3. Store signature separately (database, metadata file)
4. Original video file unchanged (no re-encoding)

Verification:

1. Retrieve signature JSON from metadata
2. Stream video file through verification API
3. Receive validation result (valid/invalid)

Benefit: No memory limit, no file size increase.

2. Encrypted Database Backups

Scenario: Nightly backup of 500 GB production database.

Detached encryption workflow:

1. pg_dump outputs to stream
2. Stream through encryption API
3. Receive JWET header (~400 bytes)
4. Store JWET header in backup metadata
5. Store ciphertext stream to S3/Azure Blob

Restoration:

1. Retrieve JWET header from metadata
2. Stream ciphertext from S3/Azure Blob through decryption API
3. Pipe plaintext stream to psql for restoration

Benefit: No intermediate files, constant memory usage.

3. Confidential Document Distribution

Scenario: Distribute 10 MB confidential report to 100 recipients.

Workflow:

1. Sign report once (detached JWS)
2. For each recipient:
   • Encrypt report with recipient's public key (detached JWE)
   • Send JWET header + ciphertext + signature JSON
3. Recipient:
   • Decrypt ciphertext (verify authenticity via tag)
   • Verify signature (verify non-repudiation)

Benefit: Single signature, per-recipient encryption, small metadata overhead.

4. Multi-GB Log File Archival

Scenario: Archive 50 GB application logs daily.

Workflow:

1. Stream logs through signature API (detached JWS)
2. Stream through encryption API (detached JWE)
3. Store JWET + signature JSON in database
4. Store ciphertext in cold storage (S3 Glacier)

Compliance:

• ✅ Integrity: Signature proves logs unmodified
• ✅ Confidentiality: Encryption protects sensitive data
• ✅ Non-repudiation: Signature proves origin
• ✅ Efficient: No memory overhead, fast streaming

5. Microservices Communication

Scenario: Microservice A sends 200 MB dataset to Microservice B.

Workflow:

1. Microservice A signs dataset (detached JWS)
2. Microservice A encrypts dataset (detached JWE)
3. HTTP POST: Multipart (JWET + JWS + ciphertext stream)
4. Microservice B:
   • Decrypts ciphertext stream
   • Verifies signature stream
   • Processes data in real-time (no buffering)

Benefit: No intermediate storage, minimal latency, secure end-to-end.

Implementation Examples

Java SDK Example (Detached JWS)

import co.ankatech.sdk.AnkaSecureClient;
import co.ankatech.sdk.crypto.StreamingCryptoService;
import java.io.FileInputStream;
import java.io.InputStream;

AnkaSecureClient client = new AnkaSecureClient(config);
StreamingCryptoService crypto = client.streamingCrypto();

// Sign large file (detached)
try (InputStream input = new FileInputStream("large-file.bin")) {
    DetachedJWS signature = crypto.signDetached("my-signing-key", input);

    System.out.println("Protected: " + signature.getProtected());
    System.out.println("Signature: " + signature.getSignature());

    // Store signature JSON separately
    Files.writeString(Path.of("large-file.jws"), signature.toJson());
}

// Verify later
try (InputStream input = new FileInputStream("large-file.bin")) {
    String signatureJson = Files.readString(Path.of("large-file.jws"));
    DetachedJWS signature = DetachedJWS.fromJson(signatureJson);

    boolean valid = crypto.verifyDetached(signature, input);
    System.out.println("Valid: " + valid);
}

See: Flow 2 - Detached-JWS Sign/Verify (Streaming)

cURL Example (Detached JWE)

Encrypt:

# Encrypt large file
curl -X POST https://api.ankatech.co/api/v3/crypto/stream/encrypt \
  -H "Authorization: Bearer $JWT" \
  -H "X-Key-Id: my-encryption-key" \
  -H "Content-Type: application/octet-stream" \
  --data-binary @large-file.bin \
  -o encrypted-output.multipart

# Extract JWET header (first part) and ciphertext (second part)
# (Manual multipart parsing required)

Decrypt:

# Decrypt large file
curl -X POST https://api.ankatech.co/api/v3/crypto/stream/decrypt \
  -H "Authorization: Bearer $JWT" \
  -F "[email protected]" \
  -F "[email protected]" \
  -o decrypted-output.bin

Python Example (Using requests library)

import requests
import json

# Sign detached
with open('large-file.bin', 'rb') as f:
    response = requests.post(
        'https://api.ankatech.co/api/v3/crypto/stream/sign',
        headers={'Authorization': f'Bearer {jwt}'},
        files={'payload': f},
        data={'kid': 'my-signing-key'}
    )
    signature_json = response.json()

    with open('large-file.jws', 'w') as sig_file:
        json.dump(signature_json, sig_file)

# Verify detached
with open('large-file.bin', 'rb') as f, open('large-file.jws', 'r') as sig_file:
    signature_json = json.load(sig_file)
    response = requests.post(
        'https://api.ankatech.co/api/v3/crypto/stream/verify',
        headers={'Authorization': f'Bearer {jwt}'},
        files={'payload': f},
        data={'jwsJson': json.dumps(signature_json)}
    )
    result = response.json()
    print(f"Valid: {result['valid']}")

Performance Comparison

Compact vs Detached Mode

Test scenario: Sign/encrypt files of varying sizes

OOM = Out of Memory (>5 MB limit enforced)

Memory Usage Comparison

Throughput Benchmarks

Hardware: 4-core CPU, NVMe SSD

Bottleneck: Usually I/O (disk/network), not crypto engine.

See: Streaming Performance Benchmarks

Quick Reference

When to Use Detached Mode

Detached vs Embedded Decision Tree

Is file size > 1 MB?
  ├── YES → Use detached mode
  │
Is streaming required?
  ├── YES → Use detached mode
  │
Is memory limited?
  ├── YES → Use detached mode
  │
Is it a JWT token (authentication)?
  ├── YES → Use compact mode (JWTs must be self-contained)
  │
DEFAULT → Use detached mode (safer, more efficient)

Related Documentation

Foundational Concepts

• JOSE Technical Reference
• Interoperability & Standards

Implementation

• Streaming Architecture Overview^*
• Streaming Endpoints Reference^*
• Streaming Output Formats^*
• Memory & Performance^*

SDK Examples

• Flow 1 - Asymmetric Encrypt/Decrypt (Streaming)
• Flow 2 - Detached-JWS Sign/Verify (Streaming)
• Flow 3 - Symmetric Encrypt/Decrypt (Streaming)
• Flow 4 - Asymmetric Re-encrypt (Streaming)
• Flow 12 - Stream Re-sign
• Flow 16 - Detached-JWS Stream Verification

Performance

• Streaming Encryption Performance
• Streaming Signature Performance

Document Status: ✅ ACTIVE

Last Updated: January 2025

Document Location: 02-concepts/detached-signatures-explained.md

Target Audience: Developers, Integration Engineers

Feedback: For corrections or additions, contact the documentation team.