Why face index 2 is not a stable ID across detectors

A group-photo editor can use one face detector in the browser and another on a worker. The browser returns three boxes; the worker returns four. A user selects the second thumbnail. What should the worker edit?

Sending faceIndex: 1 is insufficient. That number identifies an entry in one detector's output, not a person in the photograph. Even sorting both lists from left to right does not fix a missed detection or a false positive to the left of the selected face.

This is a coordinate correspondence problem. It is not face recognition, identity verification, or proof that the person consented to an edit.

Keep the selection in image coordinates

Represent the selected detection by its center and size, normalized to the decoded source image:

const selection = {
  cx: (box.x + box.width / 2) / image.width,
  cy: (box.y + box.height / 2) / image.height,
  w: box.width / image.width,
  h: box.height / image.height,
};

Use the raw detection box here. A thumbnail often adds padding or uses a square crop; that crop is presentation data, not the detector's original geometry.

Normalization handles proportional resizing. It does not fix a rotated image, a different crop, a mirrored preview, or EXIF orientation interpreted differently on each side. Both detectors must operate in the same image coordinate system. Associate the selection with an opaque source revision, and discard it when the source changes. A browser-generated revision helps prevent accidental stale selection; the server must still verify ownership and bind the job to the actual uploaded asset.

Match geometry, then allow failure

Intersection over union (IoU) measures the overlap between two boxes. It is a useful baseline when both detectors return reasonably comparable face boxes. Rank the worker's detections against the selected browser box, then reject both a weak match and an ambiguous match.

function bounds({ cx, cy, w, h }) {
  const values = [cx, cy, w, h];
  const b = {
    left: cx - w / 2, top: cy - h / 2,
    right: cx + w / 2, bottom: cy + h / 2,
  };
  if (!values.every(Number.isFinite) || w <= 0 || h <= 0 ||
      b.left < 0 || b.top < 0 || b.right > 1 || b.bottom > 1) {
    throw new Error('Invalid normalized box');
  }
  return b;
}

function iou(a, b) {
  const x = bounds(a), y = bounds(b);
  const intersection = Math.max(0, Math.min(x.right, y.right) - Math.max(x.left, y.left)) *
    Math.max(0, Math.min(x.bottom, y.bottom) - Math.max(x.top, y.top));
  return intersection / (a.w * a.h + b.w * b.h - intersection);
}

function matchSelection(selection, detections, minIou = 0.35, minGap = 0.15) {
  bounds(selection);
  const ranked = detections.map((box, workerIndex) => ({
    workerIndex, score: iou(selection, box),
  })).sort((a, b) => b.score - a.score);
  const best = ranked[0], second = ranked[1];
  if (!best || best.score < minIou) return { status: 'not-found' };
  if (second && best.score - second.score < minGap) return { status: 'ambiguous' };
  return { status: 'matched', workerIndex: best.workerIndex };
}

The numeric thresholds are illustrative, not calibrated accuracy claims. Choose thresholds using the actual detector pair and representative, permissioned test images. Different box conventions can reduce IoU even when both detectors located the intended face.

The returned workerIndex is only meaningful inside that worker result. Do not reuse it after running detection again. The transport and storage for the job also need their own authorization checks; a normalized box supplied by a browser is untrusted input.

Test the cases an ordinal cannot represent

These dependency-free assertions run with Node.js. Put the functions above in the same file, add the following code, then run node face-selection-match.mjs:

import assert from 'node:assert/strict';

const selected = { cx: 0.65, cy: 0.4, w: 0.2, h: 0.3 };
const other = { cx: 0.25, cy: 0.4, w: 0.15, h: 0.25 };
const workerTarget = { cx: 0.66, cy: 0.4, w: 0.2, h: 0.3 };
const falsePositive = { cx: 0.08, cy: 0.2, w: 0.1, h: 0.1 };

assert.deepEqual(matchSelection(selected, [falsePositive, other, workerTarget]),
  { status: 'matched', workerIndex: 2 });
assert.deepEqual(matchSelection(selected, [workerTarget, other]),
  { status: 'matched', workerIndex: 0 });
assert.deepEqual(matchSelection(selected, [selected, selected]), { status: 'ambiguous' });
assert.deepEqual(matchSelection(selected, [other]), { status: 'not-found' });
assert.deepEqual(matchSelection(selected, []), { status: 'not-found' });

The first two cases select the same region despite different detector outputs. The third deliberately supplies two equally good matches. Choosing the first would hide ambiguity; returning ambiguous gives the UI a chance to ask the user to reselect or use a clearer source image.

Also test invalid numbers, zero-size and out-of-image boxes, orientation changes, crop changes, and replacing the upload while detection is in progress. A test that only checks the happy path with identical detector output will miss the original bug.

Scope and limits

This example handles one selected region in a still image. For several selected faces, independent best matches can assign two selections to the same worker detection; enforce one-to-one matching and reject unresolved assignments. Video needs temporal tracking, so a match in the first frame is not enough. Neither case is implemented by the sample above.

Coordinate matching cannot establish a person's identity or permission to edit their likeness. Those are separate requirements. The UI should require permissioned input and make it possible to inspect the selected region before processing, without claiming that a detector has verified consent.

Disclosure: Published under the pen name Cynthia Hayes for AI SwapFace. This article and its illustrative example were prepared by an AI agent. The example was checked with executable assertions; it is not a claim that the deployed service implements this exact matching rule. No user photos, private data, model weights, or production credentials are included.