Chaos engineering deliberately introduces controlled failures to verify your systems genuinely handle them gracefully — surfacing weaknesses before they cause real incidents. This guide covers a practical introduction on VPS infrastructure.
The Core Chaos Engineering Principle
Rather than assuming your systems handle failure gracefully (untested resilience is just a hypothesis), chaos engineering deliberately tests this in a controlled way — similar philosophy to How to Test a Full Disaster Recovery Scenario (Fire Drill), but focused on smaller-scale, more frequent resilience testing.
Starting Small and Building Confidence
Begin with low-risk experiments in a genuinely non-production environment before considering anything closer to production — chaos engineering's value comes from controlled, deliberate experimentation, not reckless production disruption.
Simple Manual Chaos Experiments
sudo systemctl stop redisThe simplest chaos experiment: deliberately stop a dependency and observe how your application actually behaves — does it degrade gracefully (see general resilience patterns) or fail catastrophically?
Simulating Network Latency
sudo tc qdisc add dev eth0 root netem delay 200msLinux's traffic control (tc) can simulate network latency, packet loss, and other network degradation — useful for testing how your application handles slower-than-normal dependency responses.
Simulating Packet Loss
sudo tc qdisc add dev eth0 root netem loss 5%Removing the Simulated Condition
sudo tc qdisc del dev eth0 root netemAlways have a clear way to remove the injected chaos condition, and verify it's actually removed after your experiment concludes.
Testing Resource Exhaustion
stress --cpu 4 --timeout 60sThe stress tool can simulate CPU/memory pressure — useful for verifying your application's behavior (and your alerting) under genuine resource constraint conditions rather than just theoretical assumption.
Testing Database Connection Exhaustion
Deliberately exhaust your database connection pool (many concurrent slow queries) and observe application behavior — does it queue gracefully, fail with clear errors, or hang indefinitely? Understanding actual behavior beats assumption.
Using a Dedicated Chaos Engineering Tool (For More Systematic Testing)
Several open-source chaos engineering tools provide more systematic experiment orchestration (scheduled experiments, automatic rollback, blast radius control) than manual ad-hoc commands — worth considering as your chaos engineering practice matures beyond initial manual experimentation.
Documenting Findings and Following Through
Chaos experiments that reveal weaknesses provide value only if you actually address the findings — treat discovered issues as genuine action items (similar to How to Write an Effective Incident Postmortem's follow-through principle), not just interesting observations.
Building Toward Production Chaos Testing (With Genuine Care)
Production chaos experiments (when your organization is genuinely ready) require careful blast radius limitation, monitoring, and rollback capability — this is a mature practice to grow into gradually, not a starting point for teams new to chaos engineering.
Common Errors
Chaos experiment causes unexpected broader impact — always start with the smallest possible blast radius, have monitoring actively watching during the experiment, and a clear, tested rollback/stop mechanism ready before beginning any experiment.
Continue Reading
- How to Test a Full Disaster Recovery Scenario (Fire Drill)
- How to Diagnose Network Packet Loss on a VPS
- How to Write an Effective Incident Postmortem
Browse more articles in Advanced Observability & Incident Management.
