What the signals actually promise
There are two, they mean different things, and conflating them is the most common source of bad spot architecture.
The rebalance recommendation is a risk signal. It tells you an instance is at elevated risk of interruption. It can arrive well before the interruption notice, or it can arrive at the same moment, and AWS is explicit that it is emitted on a best-effort basis. It is not a deadline. Treating it as one causes teams to thrash their fleets, replacing instances that were never going to be reclaimed.
The interruption notice is the actual warning. AWS issues it two minutes before it stops or terminates the instance. This is your bounded shutdown window.
Two important qualifications on that two minutes.
First, it is best effort. AWS states plainly that an instance can be interrupted before the warning is made available. Any design that assumes the two minutes will always arrive is a design with an unhandled case in it.
Second, it does not apply to hibernation. If you configure hibernation as the interruption behavior, you receive a notice but no two-minute window, because the hibernation process begins immediately.
The practical consequence is that two minutes is your planning budget and zero is your worst case. Build for the second and use the first.
Restart cost is the metric that matters
Replace the stateless-versus-stateful question with a measurable one.
Restart cost is how long a workload takes to become useful again after it is killed, plus whatever happens to work in flight during that window. It is a property you can measure rather than a category you argue about.
Four bands, and the placement decision follows directly.
Low restart cost. Seconds to become useful, no in-flight work lost. These are also usually the workloads whose resource requests are most obviously padded. Stateless API replicas behind a load balancer, most worker pods pulling from a durable queue, CI runners. Spot is straightforwardly correct here and the discount is nearly free.
Medium restart cost. Tens of seconds to a few minutes. Services with substantial warm-up, JVM applications with slow class loading, anything that populates a large in-memory cache before it can serve. Spot works with enough replicas, enough diversification, and a genuine over-provision buffer so the fleet can absorb a loss without dipping below capacity.
High restart cost. Long-running work that loses progress. Training jobs without checkpointing, long batch computations, data processing that has to restart from the beginning. Spot is viable only with checkpointing, and without checkpointing it is a bad trade regardless of the discount, because the expected cost of lost work exceeds the saving.
Critical restart cost. Anything where restarting is itself the incident. Stateful databases with local storage, singleton controllers, workloads holding long-lived connections that cannot be reestablished transparently. Keep these on on-demand capacity, and put them on a dedicated pool so a bin-packing decision cannot quietly move them. This is precisely the case for changes that are reviewed and reversible rather than autonomous.
Notice that this classification cuts across the architectural one. Plenty of nominally stateless services have high restart cost because of warm-up. Plenty of nominally stateful workloads have low restart cost because their state is durable and external.
Diversification is the whole game
If you change one thing about how you run spot, change this.
Interruptions happen when a specific capacity pool runs short, where a pool is roughly an instance type in an availability zone. Concentrate your fleet into one pool and you have made your interruption rate a function of a single market. Spread it across many pools and a shortage in one becomes a partial event you absorb rather than an outage you survive.
Most teams end up concentrated by accident rather than by choice. A node pool gets defined with one instance type because that was the type someone benchmarked, and it never gets revisited. The fix is to define pools around resource shape, roughly this much CPU and memory in this architecture, and let many instance families satisfy it.
Two practical notes. Older and less fashionable instance families are frequently both cheaper and less contended, because everyone else is competing for the current generation. And track your interruption rate per pool over time, then remove the pools that are chronically reclaimed. That data is available and almost nobody looks at it.
What has to be true before anything goes on spot
None of this is exotic and all of it is skipped.
Pod disruption budgets that reflect reality. These are what stop a drain from taking out more replicas than the service can survive. Set them deliberately rather than accepting whatever a chart shipped with, and size the workloads themselves from observed behavior rather than defaults. A disruption budget so strict that nothing can ever be evicted is its own failure mode, because it blocks routine maintenance too.
Health checks that mean something. If readiness returns true before the workload can actually serve, replacement capacity will be added to the load balancer while it is still warming up, and you will conclude that spot caused an outage that your health check caused.
A drain path that has been tested under load. Not reasoned about. Tested. AWS provides fault injection tooling that will interrupt a Spot Instance on demand for exactly this purpose. Run it during a busy period with the handler active and measure whether in-flight requests actually complete.
Something watching for the signal. If nothing polls instance metadata and no event rule exists, the two minutes are spent doing nothing at all. AWS recommends checking for interruption notices every five seconds.
A floor of on-demand capacity. Decide the minimum capacity that must exist regardless of spot availability, and hold it on on-demand. Alarm when the fleet approaches that floor, because that is the signal that your diversification has run out.
The state problem, and what actually moves it
Everything above is standard practice. It is also why the high and critical restart bands stay off spot: within a two-minute window you can drain a stateless replica gracefully, and you cannot relocate a running process without losing what is in its memory.
That constraint is a property of the tooling rather than a law of nature. Checkpoint and restore techniques, which capture the full state of a running process including memory and open connections and reconstruct it elsewhere, change what is possible inside a two-minute window. The Linux checkpoint/restore work that underpins this is mature and open source, and it is the mechanism behind live migration of running containers between nodes.
Where it applies, the calculation changes. A workload that could not previously tolerate interruption becomes one that can be moved rather than killed, which moves it down a restart-cost band and into spot eligibility.
We build this, so treat the framing with appropriate skepticism and test it against your own workloads rather than taking our word for it. The honest position is that it widens the set of workloads that can use spot capacity. It does not make everything safe to interrupt, and anything holding a strong durability guarantee should still stay on dedicated capacity.
A sane target
There is no correct spot ratio, but there is a sane way to arrive at one.
Start with the low restart cost workloads only. Get the drain path, the signal handling and the diversification right on those, where the cost of being wrong is small. Measure your actual interruption rate for a quarter rather than reasoning about it.
Then move up one band at a time, and review the ratio against interruption history rather than against anxiety. Most estates find that a substantial share of compute could sit on spot and does not, purely because nobody did the first step.
One interaction worth flagging. Moving workloads onto spot reduces your on-demand consumption, which reduces the baseline your commitments were sized against. If you are inside a commitment term, a large spot migration can push you below the committed floor and produce no saving at all. Sequence the two together, or you will do excellent work for no financial result.
The underlying point
Spot is usually framed as a risk decision, and it is really a measurement decision.
The teams that use it well are not braver. They have measured restart cost per workload, diversified their capacity pools, tested the drain path under load, and know their actual interruption rate rather than their imagined one. Everything after that is arithmetic.
The teams that avoid it entirely are usually not protecting themselves from interruption. They are protecting themselves from not knowing what would happen, which is a different problem with a much cheaper solution.
About OptOps. OptOps is the optimization layer for enterprise infrastructure, covering Kubernetes in the cloud and HPC and GPU compute on premise. It classifies workloads by restart cost, keeps high-cost workloads off interruptible capacity, and supports state-preserving relocation for the ones in between. Read-only by default, so it deploys where write-access tools cannot.
References
- Amazon Web Services. Best practices for Amazon EC2 Spot.
- Amazon Web Services. Spot Instance interruption notices.
- Amazon Web Services. EC2 instance rebalance recommendations.
- Amazon Web Services. Prepare for Spot Instance interruptions.
- Amazon Web Services. Initiate a Spot Instance interruption for testing.