Hashing and Data Integrity

A cryptographic hash function is a mathematical algorithm that takes an input of any size and produces a fixed-length output called a hash, digest, or fingerprint. Hash functions are one-directional: given a hash, it should be computationally infeasible to determine the original input. They are deterministic: the same input always produces the same hash. Hash functions are the foundation of data integrity verification, digital signatures, password storage, and many other security mechanisms.

A secure cryptographic hash function must have four key properties. Pre-image resistance: given a hash output, it's computationally infeasible to find any input that produces it. Second pre-image resistance: given an input, it's infeasible to find a different input with the same hash. Collision resistance: it's infeasible to find any two different inputs that produce the same hash output. Avalanche effect: changing even a single bit of input causes a completely different hash output, making tampering detectable.

MD5 (Message Digest 5) produces a 128-bit hash and was once widely used. However, researchers demonstrated collision attacks against MD5 in 2004 — meaning two different inputs that produce the same hash can be found. This is catastrophic for integrity verification: an attacker could substitute a malicious file that has the same MD5 hash as the original. MD5 is completely broken for cryptographic security and must not be used for integrity verification, digital signatures, or password hashing. It is still sometimes seen for non-security checksums (legacy file verification), which is acceptable for detecting accidental corruption only.

The Secure Hash Algorithm (SHA) family is the most important set of hash functions for Security+. SHA-1 (160-bit) was the standard for years but was broken by a practical collision attack in 2017 (Google's SHAttered project) — now deprecated. SHA-256 (part of SHA-2 family, 256-bit output) is the current industry standard for digital signatures, TLS certificates, and file integrity verification. SHA-3 (Keccak) is NIST's alternative algorithm offering resistance to attacks that target SHA-2's structure, though SHA-256 remains the dominant choice for most applications.

Storing passwords as plain text is never acceptable. When a user creates a password, a hash is stored in the database instead of the password itself. When the user logs in, their entered password is hashed and compared against the stored hash. However, standard SHA-256 is too fast for password storage — attackers use GPUs to test billions of hashes per second against stolen password databases. Purpose-built password hashing functions like bcrypt, PBKDF2, and Argon2 add computational cost (work factors) and memory requirements that make bulk cracking impractical.

A rainbow table is a precomputed lookup table mapping common passwords to their hashes — allowing an attacker to instantly look up a hash and find the original password. To defeat rainbow tables, password hashes must be salted. A salt is a random value added to the password before hashing, making each user's hash unique even if two users have the same password. The salt is stored alongside the hash (it doesn't need to be secret). Even if an attacker has your entire password database, precomputed rainbow tables are useless because each hash used a different salt.

HMAC (Hash-based Message Authentication Code) is a construction that uses a hash function with a secret key to produce an authentication code. Unlike a plain hash, an HMAC can only be computed and verified by parties that know the secret key. This provides both integrity (detecting changes) and authentication (proving the message came from someone with the key). HMAC-SHA256 is widely used in API authentication (signing API requests), JWT tokens, and TLS MAC operations. It prevents an attacker who intercepts a message from replacing it with a crafted alternative.

Hash functions play a central role in digital signatures. When signing a large document, you don't encrypt the entire document with the private key (which would be extremely slow). Instead, you: (1) compute the hash of the document (fast, produces a small fixed-length digest), then (2) sign the small hash with the private key. The signature is just an encrypted hash. Verification follows the same pattern: decrypt the signature to get the hash, independently hash the document, and compare. SHA-256 is mandatory in all modern PKI certificates; SHA-1 is forbidden in new certificates.

File Integrity Monitoring (FIM) uses hash functions to detect unauthorized changes to critical system files. At baseline, the FIM tool hashes all monitored files and stores the hashes securely. Periodically (or continuously), it rehashes the files and compares against the baseline. Any mismatch indicates a file was modified — potentially by malware or an unauthorized administrator action. FIM tools like Tripwire, AIDE (Linux), and Windows' own integrity checking use SHA-256 hashes. FIM is required by compliance frameworks like PCI-DSS for protecting cardholder data environments.

It's important to distinguish checksums from cryptographic hashes. Checksums like CRC32 are designed to detect accidental data corruption during transmission — they are fast and simple but can be easily forged by an attacker. Cryptographic hashes (SHA-256) are designed to be collision-resistant and tamper-evident even against adversarial manipulation. Use checksums when you only need to verify that a download wasn't corrupted in transit; use cryptographic hashes when you need to verify a file has not been intentionally tampered with by an adversary.

Hash collision attacks have real-world impact. In 2017, researchers created two different PDF files with the same SHA-1 hash — demonstrating that SHA-1 certificates could be forged. In 2008, researchers used MD5 collisions to create a fake CA certificate that appeared legitimate. These attacks are why browsers reject MD5 and SHA-1 certificates and why certificate authorities must use SHA-256 minimum. Understanding collision attacks helps you explain why algorithm deprecation decisions are made and why 'but it still produces a hash' is not a sufficient argument for using broken algorithms.

Test your understanding of CompTIA Security+ (SY0-701) concepts from this lesson.

In this lesson you learned: hash functions produce fixed-length digests for integrity verification; MD5 and SHA-1 are broken and deprecated; SHA-256 is the current standard; salting defeats rainbow table attacks; and HMAC adds authentication to hashing using a shared secret key. Next up we explore Key Exchange and Hybrid Encryption.