DR Testing Without Impact

A disaster recovery plan that has never been tested is only a hypothesis. Real-world experience shows that DR plans frequently reveal gaps — configuration drift, missing automation, outdated runbooks, or longer-than-expected startup times — that only surface under test conditions. Regular DR testing is the only way to have confidence that your plan will work when you need it most.

Test Failover is a built-in Azure Site Recovery feature that lets you simulate a failover to the secondary region without interrupting production. During a test failover, ASR creates copies of the replicated VMs in an isolated virtual network in the secondary region. Production VMs continue running normally in the primary region, so there is no risk to live users.

The isolated test VNet must not have connectivity to production systems. This prevents test VMs from accidentally writing to the production database, sending emails to real customers, or triggering payment transactions. Create a dedicated test failover VNet with no peering to production VNets and no internet access, then use it exclusively for DR drills.

A DR test should validate a specific set of criteria:

• Boot time — do all VMs start within the expected time window?
• Application startup — does the application initialise correctly when connected to the recovered database?
• Data integrity — is the data at the recovery point consistent and complete?
• Actual RTO — measure the total elapsed time from failover trigger to application serving requests
• Runbook execution — did all automation scripts complete successfully?

During the test, start a timer the moment you trigger the test failover. Stop the timer when the application is confirmed healthy (load balancer health probe returns 200 OK). This is your actual RTO. Compare it to your target RTO. If the actual exceeds the target, identify the bottlenecks — slow VM startup, long database initialisation, DNS propagation delay — and address them.

After the test failover completes, connect to the recovered database and verify the data. Check that transactions committed before the replication cutoff are present, and that partially committed transactions are handled correctly (rolled back or completed). For databases with point-in-time restore (PITR), test restoring to a specific timestamp and verify the expected data state.

When the test is complete, you must clean up test failover resources — the test VMs, their disks, and network interfaces in the secondary region. Azure Site Recovery provides a 'Cleanup test failover' action in the portal that removes all test resources automatically. Forgetting to clean up wastes money and clutters the secondary region with stale resources.

After each DR test, write a test report that includes: the date and scope of the test, the actual RTO and RPO achieved, a checklist of validation items with pass/fail status, any gaps or failures observed, and the corrective actions planned. This report is valuable for compliance audits (ISO 27001, SOC 2, HIPAA) and for tracking improvement in DR maturity over time.

Industry best practices and compliance frameworks typically require DR tests at minimum annually, but many organisations test quarterly or even monthly for Tier 1 workloads. More frequent testing catches configuration drift earlier and builds team confidence and muscle memory. Automate as much of the test setup and verification as possible to reduce the effort of frequent testing.

Azure Chaos Studio is a managed chaos engineering service that lets you inject controlled failures into Azure resources to test application resilience. You can shut down VMs, fail zones, throttle CPU, or inject network latency to observe how your application behaves. Unlike a standard DR drill, chaos engineering tests whether your application degrades gracefully under partial failure conditions.

DR testing is most valuable as part of a continuous improvement cycle: Plan → Execute → Measure → Remediate → Repeat. After each test, address the gaps found, update runbooks and documentation, and then test again. Over time, the gap between your stated RTO/RPO targets and your actual achieved values should narrow until you consistently pass every test within tolerance.

Test your understanding of Microsoft Azure Fundamentals (AZ-900) concepts from this lesson.

In this lesson you learned: test failover lets you simulate a DR event without interrupting production by creating copies of VMs in an isolated VNet; you must measure actual RTO and verify data integrity during the test; and you should clean up test resources and document results after each drill. Next up we explore disaster recovery specifically for PaaS services like Azure SQL Database.