RemingtonCross5246A support hostname cutover has a nasty ambiguity: a DNS record was deleted, and nobody knows which...
A support hostname cutover has a nasty ambiguity: a DNS record was deleted, and nobody knows which old value belongs back. The live zone can tell you what is missing, but not what used to be there. Speed matters less than restoring the right owner, type, value, and TTL.
Short answer: If nobody knows which DNS record was deleted, debug your own successful change logs for its before-image, verify it against a versioned zone snapshot and the intended cutover, then recover the complete record set only if the old endpoint still works. If the evidence is absent, don't guess from today's zone.
Imagine a customer-support app moving help.example.test to a new endpoint. The team wants a fast cutover and an equally clear path back. A deleted record discovered during that change might be the old address, a verification record, or an unrelated name. Treat those as separate hypotheses. Start with the exact zone and fully qualified owner name, then narrow the change window using the deploy timestamp and the first observed lookup failure. Query the authoritative zone and any retained zone versions; save their results before editing anything.
A resolver response alone is weak evidence. A cached positive answer can outlive deletion until its TTL expires; a negative answer can be cached too. RFC 2308 defines negative caching, while RFC 1035 describes the TTL carried by resource records. Ask an authoritative server for the current answer, and distinguish that from what users may still see through recursive resolvers. The gap between those views is part of the incident, not proof that one side is lying. For example, a support agent seeing the old destination and a developer seeing a negative answer may both be reporting accurately from different caches; neither observation identifies the deleted record's prior value.
The constraint changes the decision. For a one-person service shipping weekly, minutes spent guessing at DNS are minutes not spent fixing the actual support flow. Keep the first pass small: establish what was deleted and whether the old destination can still serve requests.
A deletion event that records only name and type is not a backup. You need either the deleted record's before-image or a preceding successful create/update event for that exact record set. Check event success, zone, owner, type, and timestamp. If the record set had multiple values, reconstruct the whole set, not one convenient value. A request log saying someone tried to delete it is not evidence that the authoritative zone accepted the change.
Here is the smallest useful reconstruction pattern. This example uses illustrative data, not a claim about any DNS provider's audit format. The retained events are ordered by sequence; before is a complete record-set snapshot captured by the change system.
type RecordSet = { owner: string; type: string; ttl: number; values: string[] };
type Change = {
sequence: number;
zone: string;
action: "upsert" | "delete";
succeeded: boolean;
before: RecordSet | null;
};
const changes: Change[] = [{
sequence: 42,
zone: "example.test",
action: "delete",
succeeded: true,
before: { owner: "help.example.test", type: "A", ttl: 300, values: ["192.0.2.10"] },
}];
function deletedBeforeImage(changes: Change[], zone: string, owner: string): RecordSet | null {
const deletion = changes
.filter(c => c.succeeded && c.zone === zone && c.action === "delete" && c.before?.owner === owner)
.sort((a, b) => b.sequence - a.sequence)[0];
return deletion?.before ?? null;
}
const candidate = deletedBeforeImage(changes, "example.test", "help.example.test");
if (!candidate) throw new Error("No verified before-image; stop and inspect zone history");
The filter deliberately fails closed. It does not publish a record, and it does not pretend that a name uniquely identifies one DNS type. In a real restore, also match the expected type and change identifier, then check whether later successful writes superseded the deletion. Preserve the original event and an immutable copy of the zone snapshot so the reconstruction can be reviewed.
Separate recovery of evidence from execution. Compare the candidate with the planned destination and with an independent inventory of the support endpoint. Check that the target is still serving the intended hostname before putting traffic back. Restore the whole record set with its intended TTL, then query the authoritative servers again for the exact owner and type. Track resolver observations separately; a successful write does not flush existing caches.
For a cutover, the trade-off is propagation delay versus cutover speed. Lowering a TTL immediately before switching does not retroactively shorten answers already cached under an earlier TTL. Plan the low-TTL window ahead of a scheduled switch, and keep the previous endpoint available while the old answers may remain cached. If the old endpoint is gone, restoring its DNS address makes the incident worse. This is why a rollback path needs a live destination as well as a saved record.
Pause here if the evidence conflicts.
A missing before-image, an unknown target, or a newer write to the same record set should trigger human review, not an automated replay.
For a single support hostname, a retained, ordered change log plus versioned zone exports can be enough. At larger scale, make each change record the complete before-and-after record sets, actor, request identifier, zone, and authoritative write result. Reconcile that log against periodic snapshots. Test a restore in a non-production zone using the same record-set semantics, including multiple values and TTLs. Alerts should report the affected owner and type, not merely a count of edits.
There is a cost to keeping history and running rehearsals. There is also a cost to guessing while support is unavailable. Outsource undifferentiated hosting if that frees time to ship weekly, but keep the evidence and rollback procedure portable: the decision to restore should not depend on remembering which dashboard button was clicked. The useful success measure is whether an operator can identify the exact prior record, verify the endpoint, and explain why a cached answer may still disagree.
See the primary specifications in References.