OpenSSL API: Core Structures and Pitfalls

The EVP (Envelope) API is OpenSSL's high-level interface for cryptographic operations and is the correct API to use in application code. It abstracts algorithm-specific details behind a consistent interface, supports engine and provider hardware acceleration automatically, and handles padding and mode details correctly. OpenSSL's lower-level algorithm-specific APIs are deprecated in modern versions and should be avoided.

Symmetric encryption in OpenSSL uses the EVP_CIPHER_CTX context object. The operation sequence is: EVP_CIPHER_CTX_new to allocate the context, EVP_EncryptInit_ex to initialize with cipher and key, EVP_EncryptUpdate repeatedly to process data, EVP_EncryptFinal_ex to flush the final block and retrieve remaining output, and EVP_CIPHER_CTX_free to release memory. Skipping EncryptFinal_ex is a common mistake that produces truncated output.

Message digesting uses EVP_MD_CTX similarly: EVP_MD_CTX_new, EVP_DigestInit_ex with the algorithm (EVP_sha256(), EVP_sha3_256(), etc.), EVP_DigestUpdate with data chunks, and EVP_DigestFinal_ex to retrieve the hash. HMAC operations use a separate HMAC_CTX or the newer EVP_MAC API. The EVP abstraction allows swapping hash algorithms with a single function call change.

EVP_PKEY is the unified asymmetric key object in OpenSSL, supporting RSA, ECDSA, Ed25519, DH, and other algorithms under a single interface. Keys are created from PEM or DER files with PEM_read_PrivateKey and PEM_read_PUBKEY, or generated with EVP_PKEY_keygen. EVP_PKEY_sign and EVP_PKEY_verify perform signing and verification for any key type using the same function calls.

BIO (Basic I/O) is OpenSSL's I/O abstraction layer that allows reading and writing TLS-protected or raw data through a chain of BIO objects. A BIO chain might be: BIO_new_ssl (TLS layer) chained to BIO_new_connect (TCP connection). Writing to the SSL BIO automatically encrypts and sends; reading decrypts received data. BIO objects support files, sockets, memory buffers, and filters (base64, buffering).

TLS connections in OpenSSL use two objects: SSL_CTX (context) stores the shared configuration: certificate, private key, CA bundle, protocol version limits, and cipher list. SSL represents an individual connection derived from the context. Create one SSL_CTX per application, then create SSL objects from it for each connection. SSL_read and SSL_write transfer plaintext; OpenSSL handles all TLS framing and encryption internally.

The most dangerous OpenSSL mistake is ignoring return values. EVP_EncryptFinal_ex, SSL_read, and SSL_write all return negative values or zero to indicate errors. Calling SSL_read in a loop without checking SSL_get_error can cause infinite loops or miss errors. Every OpenSSL function that can fail must have its return value checked; failures often indicate security-relevant conditions like handshake failures or certificate validation errors.

SSL_read and SSL_write can return SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE, indicating that the operation should be retried after waiting for I/O. These are not true errors but non-blocking I/O notifications. Code that does not call SSL_get_error and branch correctly on want-read/want-write will either busy-loop or incorrectly treat these conditions as errors. Always use SSL_get_error to interpret negative return values from TLS I/O operations.

OpenSSL contains many deprecated low-level functions that should not appear in new code: direct calls to DES_encrypt1, direct use of SHA1 or MD5 hash functions without the EVP layer, RSA_private_encrypt instead of EVP_PKEY_sign, and AES_encrypt (AES ECB without authentication). These APIs bypass security checks, use deprecated algorithms, or are simply less safe than their EVP equivalents. Compiler warnings often flag deprecated API usage.

OpenSSL uses explicit memory management; every allocated object must be explicitly freed. Common memory leaks: forgetting EVP_CIPHER_CTX_free after encryption, not calling SSL_free after closing a connection, or not calling X509_free after certificate verification. In long-running servers, these leaks cause gradual memory exhaustion. Use tools like valgrind or AddressSanitizer to detect OpenSSL memory leaks during development.

OpenSSL 3.0 introduced the provider model, replacing the older ENGINE API for hardware acceleration. Providers are loadable modules that implement cryptographic algorithms: the default provider (software), legacy provider (older algorithms), FIPS provider (FIPS 140-2 validated), and third-party providers for HSMs. The FIPS provider enables compliant operation by restricting available algorithms to those approved by FIPS 140-2 without changing application code.

Why is the OpenSSL EVP API preferred over algorithm-specific low-level APIs?

OpenSSL API recap: always use the EVP high-level API, EVP_CIPHER_CTX for symmetric operations, EVP_MD_CTX for hashing, EVP_PKEY for asymmetric keys, BIO chains for I/O abstraction, SSL_CTX and SSL for TLS connections, always check return values and call SSL_get_error, free all allocated objects to prevent leaks, avoid deprecated low-level functions, and use OpenSSL 3.0 providers for FIPS compliance or hardware acceleration.