Data Quality Checks That Catch Real Web Scraping Failures

A scraper can run successfully and still give you bad data. The HTTP request returns 200, your parser finds elements, the job writes rows, and nobody notices that price now contains the text from a promo banner because the site changed a CSS class.

That is the annoying part of scraping: failure often looks like success until someone uses the dataset.

Treat scraped data like an API contract

The first mistake is storing whatever the page gives you and cleaning it later. That works for one-off scripts. It breaks down when you run scrapers on a schedule or feed results into dashboards, pricing systems, search indexes, or ML jobs.

Define the shape of a valid record before you scrape too much:

{
  "product_id": "SKU-123",
  "url": "https://example.com/products/sku-123",
  "title": "USB-C Charger",
  "price_usd": 24.99,
  "currency": "USD",
  "in_stock": true,
  "scraped_at": "2026-08-18T10:15:00Z"
}

That contract should include types, required fields, accepted ranges, and any values that must be normalized. For example, do not allow price_usd to sometimes be "$24.99", sometimes 24.99, and sometimes "Contact us". Pick one representation and reject the rest.

Here is a small Pydantic model that catches common scrape issues at ingestion:

from datetime import datetime, timezone
from pydantic import BaseModel, HttpUrl, Field, field_validator, ValidationError

class ProductRow(BaseModel):
    product_id: str = Field(min_length=1)
    url: HttpUrl
    title: str = Field(min_length=1)
    price_usd: float = Field(gt=0, lt=10000)
    currency: str
    in_stock: bool
    scraped_at: datetime

    @field_validator("currency")
    @classmethod
    def currency_must_be_usd(cls, value):
        if value != "USD":
            raise ValueError("currency must be USD")
        return value

    @field_validator("scraped_at")
    @classmethod
    def scraped_at_must_have_timezone(cls, value):
        if value.tzinfo is None:
            raise ValueError("scraped_at must be timezone-aware")
        return value.astimezone(timezone.utc)

raw_row = {
    "product_id": "SKU-123",
    "url": "https://example.com/products/sku-123",
    "title": "USB-C Charger",
    "price_usd": "Contact us",
    "currency": "USD",
    "in_stock": True,
    "scraped_at": "2026-08-18T10:15:00Z"
}

try:
    row = ProductRow.model_validate(raw_row)
except ValidationError as exc:
    print(exc)

That bad row fails with a useful symptom:

price_usd
  Input should be a valid number, unable to parse string as a number

That error tells you something specific: either the product really has no public price, or your selector is no longer pointing at a numeric price. Both cases need handling, but they are different problems.

Validate before, during, and after scraping

You need checks at more than one point because different failures happen at different times.

Before scraping, validate inputs. Remove duplicate URLs, reject unsupported domains, and normalize URL parameters if they do not affect page content. A bad URL list creates duplicate records and retry noise.

During scraping, validate extracted fields before writing them as clean data. If a required field is missing, do not silently store null and hope downstream code understands it. Put the item into a re-scrape queue with the reason.

After scraping, validate aggregate behavior. One missing price might be normal. A 40 percent missing-price rate usually means the page template changed, your proxy received a bot-check page, or the site served a region-specific layout you did not expect.

A simple re-scrape queue can be enough:

valid_rows = []
rescrape_queue = []

for raw_row in scraped_rows:
    try:
        valid_rows.append(ProductRow.model_validate(raw_row).model_dump())
    except ValidationError as exc:
        rescrape_queue.append({
            "url": raw_row.get("url"),
            "product_id": raw_row.get("product_id"),
            "reason": exc.errors(),
            "raw": raw_row,
        })

print(f"valid={len(valid_rows)} rescrape={len(rescrape_queue)}")

The important part is not the queue technology. Use Redis, SQS, Postgres, or a file if the job is small. The important part is that invalid rows do not disappear and do not become trusted records.

Wire fits this kind of extraction workflow when selector failures, missing fields, retries, and validation status need to be tracked alongside the scraped records.

Watch for selector drift, not just exceptions

Many scraper bugs do not throw exceptions. BeautifulSoup, Cheerio, Playwright, and Selenium will happily return an empty list if a selector stops matching.

This code fails quietly:

price_el = soup.select_one("span.product-price")
price = price_el.text.strip() if price_el else None

If you accept None, the scraper keeps running. A dashboard might later show average price dropping because missing values were treated as zero, or a join might exclude products with null prices.

Make selector expectations explicit:

def require_text(soup, selector, field_name):
    el = soup.select_one(selector)
    if el is None:
        raise ValueError(f"missing selector for {field_name}: {selector}")

    text = el.get_text(strip=True)
    if not text:
        raise ValueError(f"empty text for {field_name}: {selector}")

    return text

raw_price = require_text(soup, "span.product-price", "price_usd")

This turns a silent data quality bug into an operational failure you can alert on. That is a good tradeoff. A failed job is easier to investigate than a clean-looking CSV full of wrong fields.

You should also store enough evidence to debug failures: URL, timestamp, scraper version, selector version, response status, content hash, and maybe a small HTML snapshot for failed pages. Do not store full pages forever unless you have a reason, since storage and privacy concerns add up quickly.

Use anomaly detection after basic rules

Start with simple rules. They catch most issues:

• Required fields cannot be blank
• Prices must be positive numbers
• Timestamps must include a timezone
• Product IDs must match the expected format
• Duplicate product IDs within the same scrape should be investigated

Then add statistical checks for things that are valid individually but suspicious in bulk.

For example, a price of 499.99 might pass schema validation. But if the same product usually sells between 19.99 and 29.99, you should flag it.

import statistics

historical_prices = [21.99, 22.49, 20.99, 23.99, 21.49, 22.99]
new_price = 499.99

mean = statistics.mean(historical_prices)
stdev = statistics.stdev(historical_prices)
z_score = abs((new_price - mean) / stdev)

if z_score > 4:
    print(f"price anomaly: {new_price} z_score={z_score:.2f}")

This is not proof that the value is wrong. It is a signal. The site might have changed pack size, the item might be out of stock and listed by a marketplace seller, or your scraper might have captured a financing amount instead of a price.

That distinction matters. Do not automatically delete anomalies unless your business rule says to. Flag them, route them for review, or re-scrape from another source.

If you buy scraped datasets rather than run the crawlers yourself, Wire is most relevant when you need to understand how extraction failures, anomaly checks, and retry decisions are represented in the delivered data.

Keep schema changes visible

Scraped data changes because websites change. Your schema will change too. Add a version field to your output and keep validation code in version control.

{
  "schema_version": "products.v3",
  "product_id": "SKU-123",
  "price_usd": 24.99
}

When a field changes meaning, create a migration or a new version. Do not quietly reuse the old column name. availability meaning in_stock | out_of_stock is different from availability meaning ships_today | preorder | discontinued.

Your next practical step: take one existing scraper and add three things to it this week: a schema validator, a re-scrape queue for invalid rows, and a daily summary showing missing-field rates by domain and scraper version.