Building a dedupe-first AI news feed — and where it fails

I'm the author of InBrief. This isn't a pitch. It's three things: why I built a dedupe-first feed for AI news, how I actually read it day to day, and the cases where it's the wrong tool for the job. If you've ever shipped an aggregator, most of the pipeline will look familiar — the interesting part is how the engineering decisions show up in reading behavior.

The problem was never "not enough AI news"

My old routine: a screenshot of a model launch on X before breakfast; three rewrites of the same launch in a team Slack channel by lunch; the Hacker News thread in the afternoon, where the comments had better sourcing than the article; two newsletters at night that overlapped by roughly half.

Three specific failure modes:

1. Duplication with variation. A big enough story gets retitled and reposted by everyone. Working out whether two items describe the same event costs more time than reading either one.
2. Telephone-game distortion. An official blog post says something with three caveats attached. By the third hop it's a flat claim with no scope, no date, no conditions.
3. Fragmentation. Announcements live on company blogs, opinions on X, deep discussion on HN and Reddit, code on GitHub, papers on Hugging Face. You either skip a few surfaces or spend the day switching tabs.

What I wanted wasn't more sources. It was one place I could open on a schedule, where something had already decided whether item B is a restatement of item A.

What the pipeline actually does

InBrief is a time-ordered list that refreshes every two hours. It pulls from RSS, Hacker News, Reddit, X, YouTube transcripts, GitHub Trending and the Hugging Face papers leaderboard — 370+ sources at the moment, including about 310 AI practitioners on X (Karpathy, Sam Altman and Simon Willison among them) and 13 YouTube channels. Items go through an LLM for filtering, scoring and deduplication, and come out with bilingual summaries in English and Chinese.

Two engineering details matter more than the rest:

• Two-stage dedupe: SimHash, then pgvector. SimHash catches near-identical text cheaply; the vector stage catches the same story told in genuinely different words. Neither alone is sufficient, because retitled reposts are sometimes character-level copies and sometimes complete rewrites.
• Scoring happens before reading, not after. Every item gets an importance score. That's what makes a list of a few hundred items survivable.

The bilingual output sounds like a feature-list item, but it's really a triage tool: when a source is written in a language I don't read comfortably, the summary tells me whether the original is worth opening.

If I had to defend one design decision hardest, it would be the dedupe stage. It isn't glamorous, but it decides whether you open a feed or a wall of the same announcement five times.

Primary sources first, commentary second

This was the original motivation for building the thing. The feed tracks official blogs — OpenAI, Google DeepMind, Google Research, Microsoft AI, Nvidia, Apple Machine Learning, Hugging Face — alongside outlets like TechCrunch AI and The Verge. Official posts and reporting sit in the same list, and ordering doesn't invert: you see the primary item, then you see how it was interpreted.

My rule of thumb:

• Model and capability launches, API and pricing changes, papers and benchmark results, company-level strategy — go back to the primary source to check the facts. Reporting tends to drop scope and caveats first.
• Hands-on impressions and industry gossip — the reporting is the better use of your time.

The gap between first-hand and second-hand isn't mainly about speed. It's about whether scope, publication date and conditions survive the trip. Keeping both in one list means you don't have to maintain a dozen blog bookmarks and a pile of RSS feeds to preserve the "official first, analysis second" order.

The comments are the content

On Hacker News I read the thread, not the title. A good chunk of HN's value is in the replies: someone posts the paper the article is summarizing, someone points out the benchmark is contaminated, someone links a working implementation. Reading headlines only is reading the table of contents.

Reddit coverage spans 11 AI-related subreddits, including r/OpenAI, r/MachineLearning and r/ClaudeAI. The research threads and the local-deployment / open-model threads are often more informative than the comment section under the news article itself.

News tells you what happened. Discussion tells you what it means and where it might be wrong. I scan the news column first, then spend two or three minutes on the community column. Some days the item worth remembering is a single sentence in an HN reply, not any of the headlines.

Score first, click later

The workflow: read the top-scored items, read the summary, decide whether it's relevant, then either skip or open the original. Low-score items with familiar titles are usually events that have already been covered repeatedly — a glance at the summary to confirm there's nothing new, then move on.

The effect isn't reading more. It's opening fewer tabs. I used to click a dozen headlines out of curiosity and find that eight of them were versions of one story. Now most of that judgment happens at the list level, and ten minutes covers more ground than it used to. Summaries also recover things I'd otherwise miss: a non-English source, or an official update with a boring title where the summary makes clear what actually changed.

When you want to ask a question instead of scan

Browsing covers the "what's new" case. For "what did anyone say about this topic between these dates," there's an Agent Skill that takes natural-language queries by category, source and time range. It runs in Claude Code, Codex, OpenClaw and Antigravity.

The collection engine is open source under MIT: https://github.com/frankzch/ai-news-brief
The Agent Skill lives here: https://github.com/frankzch/ai-news-skill

Self-hosting is real work: you need PostgreSQL with pgvector and an LLM API key. That's a technical prerequisite, not a signup form. If all you want to do is read, this section doesn't apply to you.

How it fits next to newsletters, RSS and news sites

No scoring of other tools here — just division of labor:

• Newsletters suit a fixed-time email habit. Strong rhythm; weaker for interaction and for going back to find something you half-remember.
• RSS readers give you the most control, because you pick every source. The cost is that you also maintain the source list and whatever dedupe rules you can cobble together.
• News sites update fast and have an editorial point of view, but they're narrative-first: primary sources and community discussion live somewhere else.
• InBrief is aggregate + score + dedupe behind a fixed entry point. It doesn't replace deep reading, and it definitely doesn't replace reading the paper yourself.

Who it's for, and who should skip it

A reasonable fit:

• Developers who need to keep up with AI progress but have limited time
• Product people tracking model and tooling shifts
• Anyone who'd rather have a primary source than a summary of a summary
• People trying to build a fixed information routine instead of being led by notifications

A poor fit:

• General news readers. Coverage is AI-specific; other industries aren't included.
• Anyone who wants zero technical setup but full self-hosting. PostgreSQL + pgvector + an LLM API key is the entry price.
• Anyone looking for a single source of truth. It aggregates and filters. Depth still lives in the original post, the paper and the long thread.

If you're building something similar, the lesson I'd pass on is that dedupe is a product feature, not an implementation detail. The scoring model gets the attention, but the dedupe stage decides whether anyone opens the list a second time.

Disclosure: the author is involved in the InBrief project.

Note: AI assistance (DeepSeek) was used in researching and drafting this article; it was reviewed by the author, who is responsible for its content.