Pluggable Encryption: How We Built a Provider-Agnostic Crypto Service

01Why Pluggable Encryption

Most encryption code is tightly coupled to a specific provider. You import the AWS KMS SDK, call kms.encrypt(), and move on. It works. But it creates three problems.

First, testing is painful. KMS calls require AWS credentials, network access, and a real KMS key. Unit tests become integration tests. Mocking the KMS SDK is fragile and breaks when the SDK updates.

Second, provider switching is expensive. If you need to move from KMS to Vault — for compliance, for multi-cloud, for cost — you are rewriting every file that calls kms.encrypt(). In a platform with 34 services, that is hundreds of call sites.

Third, different environments need different providers. Production uses KMS with automatic key rotation. Development uses a local provider with a static key. Staging might use Vault. If the provider is hardcoded, you need conditional logic everywhere.

The pluggable encryption service solves all three. Business logic calls encrypt() and decrypt() on a provider-agnostic interface. The factory creates the right provider based on configuration. Tests use the local provider. Production uses KMS. Vault is available when compliance requires it. The switch is a configuration change, not a code change.

🔌

Business logic calls encrypt() and decrypt() on a provider-agnostic interface. The factory creates the right provider. Switching from KMS to Vault is a config change, not a code change.

02The EncryptionProvider Interface

The EncryptionProvider interface defines the contract that every provider must implement. It has seven methods.

encrypt(data, context) takes a string and an optional encryption context (key-value pairs that bind the ciphertext to a specific purpose) and returns an encrypted string. decrypt(encryptedData, context) reverses the operation.

generateHash(data, algorithm) produces a cryptographic hash — used for integrity checks, deduplication, and password-adjacent operations. verifyHash(data, hash, algorithm) checks if a hash matches.

encryptLargeData(data) handles objects that exceed the provider's size limit. KMS, for example, has a 4KB limit on direct encryption. Large data uses envelope encryption — encrypt the data with a data key, encrypt the data key with the master key, return both. decryptLargeData reverses this.

The interface is deliberately minimal. Every provider must support encrypt and decrypt. The other methods are optional — providers that do not support hashing or large data encryption can omit them, and the PluggableEncryptionService will throw a clear error if a caller tries to use an unsupported method.

📐

Seven interface methods: encrypt, decrypt, generateHash, verifyHash, encryptLargeData, decryptLargeData, plus encryption context binding. Minimal contract, maximum flexibility.

03The Providers

Four providers implement the EncryptionProvider interface.

The AWS KMS Provider uses AWS Key Management Service for encryption. It calls kms:Encrypt and kms:Decrypt with a customer-managed key (CMK). Key rotation is automatic — AWS rotates the backing key annually, and old ciphertexts remain decryptable. The provider supports encryption context, which binds ciphertext to a specific purpose (pagination, session, PII) and prevents context-stripping attacks.

The HashiCorp Vault Provider uses Vault's Transit secrets engine. It authenticates via token or AppRole, then calls the transit/encrypt and transit/decrypt endpoints. Vault handles key management, rotation, and access control independently of AWS. This provider is for organizations that use Vault as their central secrets management platform or need encryption that is not tied to a single cloud provider.

The KMS Encryption Provider adds envelope encryption on top of KMS. Instead of encrypting data directly with the CMK (which has a 4KB limit), it generates a data encryption key (DEK), encrypts the data with the DEK using AES-256-GCM, encrypts the DEK with the CMK, and returns both. This supports arbitrarily large data while keeping the CMK secure. Data key caching reduces KMS API calls for repeated operations.

The Local Provider uses Node.js crypto with AES-256-GCM and a static key from environment variables. It is for development and testing only — never for production. It provides the same interface as the other providers, so code that works with the local provider works identically with KMS or Vault.

04The PluggableEncryptionService

The PluggableEncryptionService is the class that callers interact with. It wraps an EncryptionProvider and adds higher-level operations.

encryptObject takes an object and a list of field names, and encrypts only those fields — leaving the rest untouched. This is how we encrypt PII fields (email, phone, address) in a DynamoDB item without encrypting the partition key or sort key. decryptObject reverses it.

encryptLargeData serializes an object to JSON, compresses it, and encrypts the result using the provider's large data support. This is used for encrypting file metadata, large configuration objects, and bulk export data.

The service supports runtime provider switching via setProvider(). This enables zero-downtime migration between providers. You can start encrypting new data with Vault while still decrypting old data with KMS. Once all old data has been re-encrypted, you remove the KMS provider entirely.

The fromEnvironment() factory method reads the ENCRYPTION_PROVIDER environment variable and creates the appropriate provider. Set it to kms for AWS KMS, vault for HashiCorp Vault, kms-encryption for envelope encryption, or local for development. The same code, different providers, controlled by a single environment variable.

⚙️

Set ENCRYPTION_PROVIDER=kms for production, vault for compliance, local for development. Same code, different providers, one environment variable.

05The Factory and Connection Pooling

The EncryptionServiceFactory creates providers with proper configuration, caching, and connection pooling.

Provider instances are cached by configuration key. If two services request a KMS provider with the same key ARN, they get the same instance. This prevents connection proliferation in Lambda functions that handle multiple encryption operations per invocation.

Connection pooling is configurable: maximum connections, idle timeout, and connection reuse strategy. For KMS, this controls the HTTP connection pool to the KMS API. For Vault, it controls the HTTP connection pool to the Vault server. The defaults are tuned for Lambda — small pool sizes, short idle timeouts, aggressive connection reuse.

The factory also handles Vault authentication. The Vault provider needs a token to call the Transit engine. The factory supports token-based auth (for development) and AppRole auth (for production). AppRole credentials are stored in AWS Secrets Manager and resolved at runtime.

The clearCache() method resets all cached providers — useful for testing and for Lambda functions that need to pick up configuration changes without a cold start.

06Where We Use It

The encryption service is used across the platform for different purposes.

Pagination tokens: The PaginationEncryption utility encrypts DynamoDB lastEvaluatedKey objects into tamper-proof tokens. Clients receive an opaque string. They cannot see the DynamoDB key structure, cannot modify it, and cannot forge a token for a different page. The encryption context binds the token to the pagination purpose, preventing it from being used as a session token or API key.

Session data: Refresh tokens and session metadata are encrypted before storage. Even if the DynamoDB table is compromised, the session data is unreadable without the encryption key.

User PII: Email addresses, phone numbers, and other personally identifiable information are encrypted at the field level using encryptObject. The DynamoDB item stores encrypted values — a database dump reveals nothing.

API keys: Service-to-service API keys are encrypted before storage and decrypted at validation time. The encryption context includes the service name, preventing a key issued for one service from being used with another.

File encryption: Files uploaded to S3 are encrypted using the large data provider before storage. The encrypted file and the encrypted data key are stored together. Decryption requires both the file and access to the master key.

🔐

Pagination tokens, session data, user PII, API keys, file uploads — all encrypted through the same pluggable service. One interface, consistent protection, across every service.

07Lessons Learned

Building a pluggable encryption service taught us several lessons.

First, encryption context is not optional. Without context binding, an encrypted pagination token could be replayed as a session token. Context ties ciphertext to its purpose and prevents cross-purpose attacks.

Second, the local provider is essential for developer experience. If every unit test requires KMS credentials, developers will skip encryption in tests. The local provider makes encryption testable without infrastructure.

Third, envelope encryption is necessary for any data larger than 4KB. Direct KMS encryption has a size limit. Envelope encryption removes that limit while keeping the master key secure in KMS.

Fourth, provider switching should be a configuration change, not a migration. The pluggable interface means you can run KMS and Vault side by side during a transition. New data uses the new provider. Old data is decrypted with the old provider and re-encrypted with the new one on access. Zero downtime, zero data loss.

Fifth, cache provider instances aggressively. Creating a new KMS client or Vault connection for every encryption call is expensive. The factory's caching ensures one instance per configuration, reused across all operations in a Lambda invocation.