a HackerNoon project

Method

A duplicated term is one word carrying two popular meanings. We call the older one the original and the newer one the duplicate, the meaning that rode in on an established name. Every page sits at deduplication.tech/term/definition1/definition2, with the original first.

Evidence

Wikipedia pageviews. Monthly views of the English Wikipedia article for each meaning, from July 2015 to last month, pulled live from the Wikimedia Pageviews API. Redirects resolve to the canonical article first. We compare the last 12 months to get the duplicate's share of attention, and smooth with a 3-month moving average to find the last month where the lines crossed.

Origin dates. Hand-researched approximate dates of first use for each meaning. "Ancient" means the sense predates modern record-keeping.

Spoken usage: YouTube and Movies & TV (in progress)

Pageviews show what people look up. To see which meaning people actually say, we're labeling real spoken mentions of every term from two sources:

How a mention gets labeled. We find every line containing the term, keep the sentence around it, and a model decides whether it means the original, the duplicate, or something else. Low-confidence lines go to a human reviewer. Each term page will get YouTube and Movies & TV filters showing the split by source and by year.

Colour system

Every colour on the site answers one question: which meaning owns the word?

Grey · the originalThe older meaning. Grey panels, grey lines, grey words in the cloud. A grey verdict means the original still holds: under 40% of attention goes to the duplicate.
Green · the duplicateThe newer meaning riding on the name. Green lines and words. A green verdict means the duplicate dominates: over 60% of attention.
Amber · contestedNeither side clearly owns the word: the duplicate holds 40–60%. In the cloud, amber words are the live fights.
Dim grey · not enough dataWe can't separate the meanings yet, usually because there's no Wikipedia article for one side or both sides share one.

Where you see it. On a term page the left half is always grey (original) and the right half black with green (duplicate), whoever is winning. The bar under the hero is the word split by ownership. The "Canonical today" and "Status" boxes take the colour of whichever side wins. In the cloud, word size is popularity (both meanings' Wikipedia views combined) and colour is ownership.

Verdicts

Over 60% duplicate share: duplicate dominant. Under 40%: original holds. In between: contested. The last 12 months of Wikipedia pageviews decide. When neither source separates the meanings (MTS, for example), the page shows no verdict rather than guess.

Limits

Pageviews measure lookups, not usage, and an article can cover more than one sense. Phrase searches miss usages that never spell the meaning out. Treat each page as evidence, not a ruling.