Production incidents are often associated with complex race conditions, memory leaks, or failed deployments. In our case, the trigger was a command that did not change the contents of a file at all:
touch application.bundle
The command did not update the code, replace the artifact, or restart a systemd service. It changed only the file’s timestamp.
That was enough for the application platform to conclude that a new version had arrived and start a rolling replacement. The new process began an expensive initialization sequence, created contention for memory and disk I/O, and temporarily made production unavailable.
This story is not really about the touch command. It is about a dangerous hidden relationship between filesystem metadata and the production control plane.
The Initial Symptoms
The production backend began responding slowly. HTTP 500, 502, and 504 errors followed.
At first, there were many plausible explanations:
- an external traffic spike;
- a new deployment;
- a load balancer problem;
- a virtual machine failure;
- an OOM kill;
- an application service restart;
- RDS degradation;
- a storage problem.
However, the standard indicators did not support any of these obvious hypotheses.
The load balancer and virtual machine health checks passed. The main application service had not restarted. There was no clear OOM kill. Nobody had delivered a new artifact. There were also no relevant remote commands or cloud control-plane events during the incident window.
At the same time, the application logs showed a new process starting as if a normal rolling deployment were in progress.
The Hidden Deployment
The server had a cron job intended to perform regular soft restarts of the application:
0 3 * * 2,4,7 /user/cron/app-restart.sh
To prevent multiple servers from restarting at exactly the same time, the script introduced a random delay:
sleep $((RANDOM % 600))
It then located the active application bundle and ran:
touch application.bundle
The idea was simple: updating the modification time would make the application platform restart the application gracefully.
In practice, the cron job was not an ordinary maintenance task. It was an alternative deployment mechanism.
On the day of the incident, the job started on schedule, waited for its random delay, and updated the bundle timestamp. At almost the same moment, the platform detected the change and began creating a replacement process.
The file itself remained unchanged:
- its inode did not change;
- its creation time remained the same;
- the application content was not updated;
- only its metadata timestamps changed.
This distinction allowed us to separate a real artifact deployment from a metadata-triggered restart.
How a Soft Restart Became an Outage
The application platform used side-by-side replacement:
- The old process continues serving production traffic.
- A new process starts beside it on a different port.
- The platform waits for the new process to become healthy.
- Traffic moves to the new process.
- The old process terminates.
This model is generally safer than a stop-and-start restart. However, it relies on an important assumption: the server must have enough resources to run both processes at the same time.
In our case, the new process began loading hundreds of large objects from RDS before opening its health port.
It processed a substantial part of the dataset successfully, but then stopped making progress. No application-level exception appeared, so the logs showed only the last object loaded successfully.
At that point, the same server was running all of the following:
- the active production process;
- the replacement process;
- intensive data loading from RDS;
- the filesystem cache;
- normal system services.
Their combined memory consumption rose sharply.
When a Memory Problem Became a Storage Problem
As available memory became scarce, Linux entered active direct reclaim.
Direct reclaim means that a process requesting memory must spend time freeing pages itself. File-backed pages can be evicted from the page cache and then read from disk again when the application accesses them later.
The incident developed through the following chain:
scheduled cron
-> touch an unchanged bundle
-> side-by-side replacement
-> large preload from RDS
-> two processes compete for memory
-> Linux enters direct reclaim
-> file-backed pages are read from disk again
-> EBS reaches its throughput limit
-> replacement startup stalls
-> the active production process slows down
-> users receive errors and timeouts
This is an important detail: the incident looked like both a memory problem and a storage problem.
Memory was the initial constraint, but repeated EBS reads amplified the user impact. The increase in disk throughput was a consequence of memory reclaim, not an independent storage failure.
The old production process did not crash. It became a victim of resource contention with the process intended to replace it.
Why the Replacement Never Completed
The platform gave the new process a limited amount of time to start listening on its health port.
The new process spent that time preloading data and never became reachable. When the startup timeout expired, the platform rejected the replacement. The old process remained active.
After the replacement process exited, resource pressure fell and production gradually recovered.
This is why ordinary operational metrics could be misleading:
- the application service did not restart;
- the main process was not killed by the OOM killer;
- the load balancer target ultimately remained healthy;
- the virtual machine did not experience an infrastructure failure;
- no deployment delivered new code.
Users still experienced a real outage.
How We Identified the Cause
Finding the cron job was not enough. To establish causality, we had to build a consistent timeline from independent data sources.
We correlated:
- the cron journal;
- the application bundle timestamp;
- the file’s inode and creation time;
- application platform logs;
- startup logs from the new process;
- preload progress;
- systemd service state;
- memory pressure;
- disk throughput;
- load balancer responses;
- virtual machine health checks;
- RDS activity;
- remote command history;
- cloud audit events.
The strongest evidence was the timing correlation:
cron job completed
~=
bundle received a new timestamp
~=
application platform started replacement
We also confirmed that the bundle had not been physically replaced. That ruled out a normal deployment and established the metadata change as the trigger.
Why the Obvious Fixes Do Not Solve the Problem
After an incident like this, adding resources is an obvious response:
- increase the memory limit;
- raise EBS throughput;
- extend the startup timeout;
- add swap;
- restart the application more frequently.
These actions may reduce the likelihood of recurrence, but they do not remove the architectural cause.
More Memory
Additional memory creates headroom, but an unbounded preload can eventually consume that headroom as the dataset grows.
More EBS Throughput
Higher throughput may reduce the impact of repeated reads, but it does not stop memory reclaim or resource contention between the two processes.
A Longer Startup Timeout
If the replacement is placing pressure on the live process, a longer timeout may simply prolong the outage.
More Frequent Restarts
A restart may temporarily clear accumulated state, but scheduled maintenance should not create more risk than the condition it is intended to mitigate.
What a Proper Fix Looks Like
A reliable solution requires changes to the startup and replacement model itself.
1. Do Not Use a Timestamp as an Implicit Command
If touch starts a deployment, that behavior should be explicitly documented and controlled like any other production change.
A dedicated restart command or API would be safer if it provided:
- an audit trail;
- preflight checks;
- locking;
- timeout control;
- a clear result;
- the ability to decline the operation safely.
2. Check Resource Headroom Before Replacement
Before starting a second process, the platform should evaluate:
- available memory;
- current memory pressure;
- PSI metrics;
- disk throughput headroom;
- RDS activity;
- current request volume;
- whether another replacement is already running.
If sufficient headroom is unavailable, the restart should be postponed.
3. Bound the Preload
Startup should not load the entire dataset without limits before the process becomes healthy.
Possible approaches include:
- lazy loading;
- bounded concurrency;
- a memory budget;
- batched loading;
- prioritizing essential data;
- moving preload work to a separate process;
- opening the readiness endpoint after reaching a minimum usable state.
4. Separate Liveness from Readiness
A process can be alive without being ready to serve traffic.
It is useful to expose separate signals:
liveness = the process is running and has not stalled
readiness = the process has loaded the minimum required state
progress = startup is continuing to move forward
Without progress telemetry, it is difficult to distinguish slow initialization from a stalled process.
5. Monitor More Than OOM Events
The absence of an OOM kill does not mean there was no memory incident.
Useful signals include:
memory.current;memory.events;memory.high;- reclaim activity;
- memory and I/O PSI;
- major page faults;
- EBS throughput percentage;
- startup duration;
- preload progress;
- failed replacement attempts.
In this case, production degraded long before the system could reach an obvious terminal event.
Key Lessons
Metadata Can Be a Control Plane
When a system reacts to modification time, the touch command is functionally equivalent to a deployment or restart command.
These operations require the same level of control, observability, and change management.
Zero-Downtime Deployment Requires Double-Process Headroom
Side-by-side replacement is safe only when the infrastructure can support both the old and new processes under production load.
Without that headroom, a rolling restart can be more dangerous than a short, controlled restart.
Startup Is Part of the Production Workload
Preload, cache warming, schema discovery, and initialization queries are not free background operations.
During replacement, they run on the same server and compete with user requests.
Storage Saturation Can Begin with Memory Pressure
High disk throughput does not always mean that the application suddenly began processing more data. It may be caused by eviction and repeated reads of file-backed pages during memory reclaim.
A Timeout Is Not Resource Control
A startup timeout determines only how long the system tolerates a failed launch. It does not limit memory usage, I/O consumption, or the damage caused to the active process.
Conclusion
The touch command was not the root cause by itself. The incident required several unsafe assumptions to align:
- a timestamp change was treated as a deployment signal;
- scheduled maintenance could start a production replacement without preflight checks;
- rolling replacement required two large processes to run simultaneously;
- startup performed an unbounded preload;
- the active and replacement processes shared memory and storage;
- observability exposed the timeout but not startup progress or resource pressure.
The most important lesson is simple:
A command’s safety is not determined by what it changes in the filesystem. It is determined by how the rest of the platform reacts to that change.
In a system where timestamps trigger deployments, touch is no longer a metadata operation. It is a production control-plane command.