0Pricing
Assembly Language & x86 Low-Level Systems Programming · Lesson

Floating-Point Precision, Rounding, and Exceptions

Understand IEEE 754 representation, rounding modes, and how the FPU and SSE units signal exceptions like overflow, underflow, and invalid operations.

Floating-Point Precision, Rounding, and Exceptions is a free Assembly Language & x86 Low-Level Systems Programming lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Assembly Language & x86 Low-Level Systems Programming learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

IEEE 754 Format

x86 floating-point follows the IEEE 754 standard. A number is stored as sign, exponent, and mantissa:

  • Single (32-bit): 1 + 8 + 23
  • Double (64-bit): 1 + 11 + 52
  • Extended (80-bit): used internally by the x87 FPU

Why Precision Matters

Most decimal fractions cannot be represented exactly in binary. For example 0.1 + 0.2 does not equal exactly 0.3. Accumulated rounding error is a core hazard in numerical code.

The Four Rounding Modes

IEEE 754 defines four rounding modes:

  • Round to nearest, ties to even (default)
  • Round toward negative infinity (floor)
  • Round toward positive infinity (ceil)
  • Round toward zero (truncate)

Controlling Rounding on x87

The x87 control word holds the rounding-control (RC) bits. You read it with fstcw, modify, then load with fldcw.

sub esp, 4
fstcw [esp]          ; store control word
or word [esp], 0x0C00 ; RC = 11 -> round toward zero
fldcw [esp]          ; load it back
add esp, 4

SSE MXCSR Register

SSE has its own control/status register, MXCSR. It holds rounding-control bits plus exception masks and flags. Manage it with ldmxcsr and stmxcsr.

sub rsp, 4
stmxcsr [rsp]        ; save MXCSR
ldmxcsr [rsp]        ; restore MXCSR
add rsp, 4

Floating-Point Exceptions

The standard defines six exceptions:

  • Invalid (e.g. 0/0, sqrt of negative)
  • Denormal operand
  • Divide by zero
  • Overflow
  • Underflow
  • Inexact (rounding occurred)

Masked vs Unmasked

Each exception can be masked. A masked exception produces a default result (like infinity or NaN) and sets a flag. An unmasked exception raises a CPU trap so your handler can intervene.

Special Values: NaN and Infinity

IEEE 754 reserves bit patterns for special values:

  • +Inf / -Inf from overflow or divide-by-zero
  • NaN (Not a Number) from invalid operations

Any comparison with NaN is false — even NaN == NaN.

Detecting NaN

Because NaN is unordered, you detect it with an unordered compare. In SSE, ucomiss sets the parity flag when an operand is NaN.

ucomiss xmm0, xmm0   ; compare value with itself
jp is_nan            ; PF set => NaN detected

A Runnable C Demonstration

This self-contained C program shows that floating-point addition is not exact and detects a NaN.

#include <stdio.h>
#include <math.h>
int main(void) {
    double a = 0.1 + 0.2;
    printf("0.1+0.2 = %.17f\n", a);
    printf("equals 0.3? %d\n", a == 0.3);
    double nan_val = 0.0 / 0.0;
    printf("isnan? %d\n", isnan(nan_val));
    return 0;
}

Practical Advice

To write robust floating-point code:

  • Never compare floats with ==; use an epsilon tolerance
  • Be aware of accumulation order in sums
  • Keep the default round-to-nearest unless you have a reason
  • Check status flags after risky operations

Quick Check

Test your floating-point knowledge.

Recap

You explored floating-point behavior:

  • IEEE 754 defines single, double, and 80-bit extended formats
  • Four rounding modes are configured via the x87 control word or MXCSR
  • Six exceptions can be masked (default result + flag) or unmasked (trap)
  • NaN and infinity are special values; NaN compares false to everything

Frequently asked questions

Is the “Floating-Point Precision, Rounding, and Exceptions” lesson free?

Yes — the full text of “Floating-Point Precision, Rounding, and Exceptions” is free to read here on the web, and the Assembly Language & x86 Low-Level Systems Programming course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Assembly Language & x86 Low-Level Systems Programming course, upgrade to CoddyKit PRO.

What will I learn in “Floating-Point Precision, Rounding, and Exceptions”?

Understand IEEE 754 representation, rounding modes, and how the FPU and SSE units signal exceptions like overflow, underflow, and invalid operations. You practise Assembly Language & x86 Low-Level Systems Programming with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Assembly Language & x86 Low-Level Systems Programming?

No prior experience is required. Assembly Language & x86 Low-Level Systems Programming on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Floating-Point Precision, Rounding, and Exceptions” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Assembly Language & x86 Low-Level Systems Programming lesson?

Yes. Every Assembly Language & x86 Low-Level Systems Programming lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. x87 FPU Programming Basics
  2. SSE/AVX Instruction Sets Introduction
  3. Vectorizing Code with SIMD
  4. Floating-Point Precision, Rounding, and Exceptions
← Back to Assembly Language & x86 Low-Level Systems Programming