Variational Autoencoders (VAE)
ELBO loss, reparameterization trick, latent space exploration, image generation with VAE.
Variational Autoencoders (VAE) is a free Learn AI with Python lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
From Autoencoder to VAE
A plain autoencoder learns scattered points, so sampling new ones produces garbage. A Variational Autoencoder (VAE) learns a smooth, continuous latent space you can sample from, turning it into a true generative model.
Encoding to a Distribution
Instead of one point, a VAE encoder outputs a distribution per input: a mean mu and a log-variance log_var. Each input maps to a small fuzzy region of latent space, not a single dot.
class Encoder(nn.Module):
def __init__(self):
super().__init__()
self.fc = nn.Linear(784, 256)
self.fc_mu = nn.Linear(256, 64)
self.fc_logvar = nn.Linear(256, 64)Why mu and log_var?
We predict log_var instead of variance directly because it can be any real number (no positivity constraint) and is numerically stable. Exponentiating recovers the variance when needed.
def forward(self, x):
h = torch.relu(self.fc(x))
return self.fc_mu(h), self.fc_logvar(h)Sampling Latent z
To decode, we must sample z from the predicted distribution. Naively this random draw is not differentiable, so we cannot backpropagate through it. The reparameterization trick fixes that.
The Reparameterization Trick
Rewrite the sample as z = mu + eps * exp(0.5 * log_var) where eps is random noise. Now the randomness sits in eps (no gradient needed) and gradients flow through mu and log_var.
def reparameterize(mu, log_var):
std = torch.exp(0.5 * log_var)
eps = torch.randn_like(std)
return mu + eps * stdThe Decoder
The decoder takes the sampled z and reconstructs the image, just like a normal autoencoder decoder, expanding 64 dims back to 784 pixels with a Sigmoid output.
decoder = nn.Sequential(
nn.Linear(64, 256),
nn.ReLU(),
nn.Linear(256, 784),
nn.Sigmoid()
)Reconstruction Loss Term
The first half of the VAE loss is reconstruction: how well the decoded image matches the input. Binary cross-entropy (or MSE) is used over the pixels.
recon_loss = nn.functional.binary_cross_entropy(
recon, x, reduction="sum"
)The KL Divergence Term
The second half is the KL divergence, which pulls each encoded distribution toward a standard normal. This regularizes the latent space so it is smooth and continuous, enabling generation.
kl = -0.5 * torch.sum(1 + log_var - mu.pow(2) - log_var.exp())The ELBO Objective
VAEs maximize the ELBO (Evidence Lower Bound), equivalent to minimizing reconstruction + KL. Reconstruction makes outputs faithful; KL keeps the space well-shaped. Balancing the two is the whole game.
loss = recon_loss + kl
loss.backward()Generating New Images
Because the latent space matches a standard normal, you can generate brand-new images by sampling z from N(0,1) and decoding, no input image required.
with torch.no_grad():
z = torch.randn(16, 64)
new_images = decoder(z) # 16 generated samplesWhy VAEs Matter
VAEs give a principled, smooth latent space you can interpolate and sample. They underpin many generative techniques and teach the core idea of learning distributions, not just points.
Quick Check
Test your VAE understanding.
Recap: Variational Autoencoders
You learned the VAE: an encoder outputs mu and log_var, the reparameterization trick z = mu + eps * exp(0.5 * log_var) keeps sampling differentiable, and the loss is ELBO = reconstruction + KL. Sampling z from N(0,1) and decoding generates new images.
Frequently asked questions
Is the “Variational Autoencoders (VAE)” lesson free?
Yes — the full text of “Variational Autoencoders (VAE)” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.
What will I learn in “Variational Autoencoders (VAE)”?
ELBO loss, reparameterization trick, latent space exploration, image generation with VAE. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Learn AI with Python?
No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Variational Autoencoders (VAE)” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Learn AI with Python lesson?
Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Autoencoders for Representation Learning
- Variational Autoencoders (VAE)
- GANs: Generator and Discriminator
- Conditional GANs and Style Transfer