Skip to content

Large File Encryption (Streaming Mode)

This document describes how large files can be encrypted in a single pass using the streaming capabilities of the Anka Secure API. This approach is highly efficient, as it avoids loading the entire file into memory, and is especially useful when dealing with files measured in gigabytes or larger.

Process Overview

  1. Key Generation/Preparation
  2. Ensure that the required encryption key (e.g., AES, RSA, ML-KEM) is already present in the keystore.

  3. If necessary, create or import a key with the appropriate encryption algorithm.

  4. Streaming Encryption

  5. The client sends the file in chunks via multipart/form-data.
  6. The server encrypts each chunk on the fly and streams the encrypted data back (or to storage), never buffering the entire file at once.

Why Streaming Encryption Matters

  • Memory Efficiency: The entire file does not need to be read into memory.
  • Scalability: Handles very large files without requiring additional resources.
  • Security: Data is never written to disk unencrypted; encryption happens as each block arrives.
  • Performance: The pipeline-like design reduces overall processing time.

High-Level Flow

Below is a simplified Mermaid diagram representing the streaming encryption pipeline from the client's perspective:

flowchart TD
    A[Client] -->|multipart/form-data| B(API /encrypt Stream)
    B -->|read chunk| C[Encrypt chunk in memory]
    C -->|write encrypted chunk| D[Encrypted Output Stream]
    D --> B
    B --> E((Done))
1. Client: Initiates the request by uploading the file in parts.

  1. API /encrypt Stream: Receives each chunk, processes it, and forwards it for encryption.

  2. Encrypt chunk in memory: The chunk is encrypted using the specified key (e.g., AES-256, ML-KEM) immediately.

  3. Encrypted Output Stream: The encrypted chunk is written out to the response or to a storage destination, depending on the API usage.

  4. The process repeats for each chunk until the file ends.

Relevant API Endpoints

1. Generate or Retrieve an Encryption Key

  1. Endpoint (Generate Key): POST /api/key-management/keys 
    {
    "kid": "mySymKey",
    "kty": "oct",
    "alg": "AES-256",
    "keyOps": ["encrypt","decrypt"],
    "exportable": false
}
    Response: 201 Created if successfully generated.
  2. Endpoint (Get Key): GET /api/key-management/keys/{kid}
    • Ensures the key (e.g., mySymKey) exists and is active.

2. Stream Encrypt

Endpoint: POST /api/crypto/stream/encrypt

  • Description: Encrypts a file in streaming mode using the public key (for asymmetric encryption) or the symmetric key (kty=oct) if so configured.

  • Multipart Form Fields:

    1. metadata (JSON) -- Must contain: 
      {
    "kid": "mySymKey"
}
    2. file (binary) -- The plaintext file.

Example (multipart/form-data):

Sample Request (pseudo raw HTTP)

POST /api/crypto/stream/encrypt HTTP/1.1
Host: demo.ankatech.co
Authorization: Bearer <token>
Content-Type: multipart/form-data; boundary=----Boundary

------Boundary
Content-Disposition: form-data; name="metadata"
Content-Type: application/json

{
  "kid": "mySymKey"
}
------Boundary
Content-Disposition: form-data; name="file"; filename="largeFile.bin"
Content-Type: application/octet-stream

<RAW_FILE_CONTENTS>
------Boundary--`

Response

  • Streams back the encrypted data chunk by chunk (application/octet-stream).
  • The client can read these chunks and write them to a file as they arrive.

Example Workflow

  1. Check if Key Exists

curl -X GET https://demo.ankatech.co/api/key-management/keys/mySymKey\
    -H "Authorization: Bearer <TOKEN>"
- If the key isn't found, generate it.

  1. Stream Encryption

curl -X POST "https://demo.ankatech.co/api/crypto/stream/encrypt"\
    -H "Authorization: Bearer <TOKEN>"\
    -F "metadata={\"kid\":\"mySymKey\"};type=application/json"\
    -F "[email protected];type=application/octet-stream"\
    --output largeFile.enc
- Note: --output largeFile.enc writes the streamed encrypted data to a file on the client side.

Key Advantages

  • No Large Memory Footprint: The server reads and encrypts each chunk on the fly.
  • Security: The plaintext data is never stored unencrypted. If using an asymmetric key, the private key remains secure on the server side.
  • Performance: Efficient for large data sets (e.g., multi-gigabyte files), as encryption happens in constant memory time.
  • Easy Client Integration: Standard multipart/form-data can be sent from virtually any programming language or environment.

Conclusion

Large File Encryption (Streaming Mode) is a core feature of the Anka Secure API, enabling highly scalable, memory-efficient, and secure encryption for massive datasets. By leveraging these streaming endpoints, you can encrypt files of any size in a single pass without compromising on performance or security.