Titles
exact duplicates, and titles that only differ by the site name once the common | Brand suffix or prefix is removed.
GitHub Packages · MIT · Node ≥ 18
duplicate-beacon
Detect duplicate and near-duplicate titles, meta descriptions and page bodies across your built static site.
1 Point the
@mrlouiswrightscope at GitHub Packages in.npmrc@mrlouiswright:registry=https://npm.pkg.github.com2 Install
npm i -D @mrlouiswright/duplicate-beacon
Installing from GitHub Packages needs a token
Published to GitHub Packages, not npmjs.com. GitHub Packages needs a GitHub token with read:packages to install, even for public packages: add //npm.pkg.github.com/:_authToken=${GITHUB_TOKEN} to .npmrc or run npm login --registry=https://npm.pkg.github.com.
$ npx duplicate-beacon test/fixtures/badduplicate-beacon — /workspace/duplicate-beacon/test/fixtures/bad11 page(s); 11 compared (noindex and canonicalised pages excluded) HIGH b.html 2 pages share the title "Widgets | Acme": /a.html, /b.html [duplicate-title] ↳ fix: Give each page a unique, descriptive <title>.HIGH b.html 2 pages have identical main content: /a.html, /b.html [duplicate-body] ↳ fix: Consolidate them into one page, or point duplicates at the original with rel=canonical (or noindex them).MEDIUM b.html 2 pages share the meta description "Buy widgets online from Acme today at low prices.": /a.html, /b.html [duplicate-description] ↳ fix: Write a unique description for each page, or remove it and let search engines generate one.MEDIUM c.html /a.html and /c.html are 94% similar [near-duplicate-body]
Why it exists
Duplicate content splits ranking signals and makes search engines pick a URL for you. It usually comes from templates: every product page titled "Widgets | Acme", one meta description pasted into twenty pages, a print or AMP copy of every article, tag pages that are 95% the same. duplicate-beacon compares every built page with every other one, skipping pages that are noindex or already canonicalised elsewhere, and reports each group with the files, severity and a concrete fix.
CLI, library and GitHub Action. Node.js 18+, zero runtime dependencies. Works on any static build output (Astro, Next.js “output: 'export'”, Eleventy, Hugo, Jekyll, Docusaurus, VitePress, plain HTML) and runs offline: point it at the folder your build writes. By Louis Wright.
Features
Meta descriptions
exact duplicates, near-duplicates (template text with a word swapped), and descriptions that just repeat the title.
Bodies
identical main content, and near-duplicates using 5-word shingles with MinHash/LSH candidate search and exact Jaccard similarity (--threshold, default 0.9). Navigation, header, footer and aside are left out of the main-content comparison.
Boilerplate
pages whose text is mostly shared with many other pages (info).
Usage
# Example
npx duplicate-beacon dist# Example
npx duplicate-beacon dist --threshold 0.85 --fail-on medium# Example
npx duplicate-beacon public --ignore "/tags/**,/page/*" --min-words 80# Example
npx duplicate-beacon out --format json > duplicates.jsonCI / GitHub Action
permissions:
contents: read
security-events: write # only needed for the SARIF upload
jobs:
duplicate-beacon:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with: { persist-credentials: false }
- run: npm ci && npm run build # produce the static site first
- uses: MrLouisWright/duplicate-beacon@v1
with:
path: dist
fail-on: high
sarif: duplicate-beacon.sarif
- uses: github/codeql-action/upload-sarif@v3
if: always()
with: { sarif_file: duplicate-beacon.sarif }Options & flags
| Flag | Description |
|---|---|
| Site | |
--base-url <url> | Public URL of the site, e.g. https://example.com (default: inferred from canonical/og:url) |
--include <globs> | Only check these pages (file or URL path globs, comma-separated) |
--exclude <globs> | Skip these pages |
--clean-urls | Pages are served without .html (about.html -> /about) |
| Checks | |
--threshold <0-1> | Similarity for near-duplicate bodies (default 0.9) |
--min-words <n> | Ignore bodies shorter than n words (default 50) |
--ignore <globs> | Pages to leave out of the comparison |
--no-template-check | Do not report boilerplate-heavy pages |
| Common | |
-c, --config <file> | JSON config (default: .duplicate-beacon.json or "duplicateBeacon" in package.json) |
-f, --format <fmt> | text (default) | json | sarif | markdown |
--json <file> | Also write a JSON report |
--sarif <file> | Also write SARIF 2.1.0 |
--markdown <file> | Also write Markdown (e.g. for a job summary) |
--fail-on <sev> | critical | high | medium | low | info | none (default: high) |
--list-rules | Print rules and default severities |
--no-fixes | Hide fix suggestions in text output |
--no-color | Disable colours |
Full reference, config file format and API in the README.
Support this project
Support open-source security tools
I build free, open-source tools that help developers ship safer websites. Every coffee helps keep them maintained, documented and free for everyone.
Buy me a coffee (opens Stripe in a new tab)Secure checkout by Stripe
More tools by Louis Wright
58 tools
Flagship Next.js 16 App Router libraries · pre-alpha · 2
- next-tonguePre-alphaCompiler-first, type-safe i18n for the Next.js 16 App Router.ICU messages compiled at build time, works with TurbopackLocale from route params, so pages stay in the static shellZero dependencies beyond next and react
- next-heraldPre-alphaTyped metadata, safe JSON-LD, sitemaps and an SEO CI check for the Next.js 16 App Router.Returns native Metadata, Sitemap and Robots objectsXSS-safe JSON-LD with an entity graph by defaultherald check: a dependency-free CI crawler for SEO mistakes
Sentinels security scanners & CI gates · 20
- binary-sentinelStatic PE and ELF triage in pure JavaScript: headers, sections, entropy, imports, imphash, packer hints, capabilities and hardening checks.
- cookie-sentinelAudit the cookies your site sets before attackers do.
- defang-sentinelDefang and refang indicators of compromise, and stop live malicious links from getting into threat reports, tickets and docs.
- dns-sentinelAudit your domains' DNS and email security before attackers or spammers find the gaps.
- dockerfile-sentinelSecurity linter for Dockerfiles and Docker Compose files: root containers, baked-in secrets, curl | sh, unpinned and end-of-life base images, privileged services, Docker socket mounts and exposed databases.
- dotenv-sentinelValidate .env files against .env.example or a schema, catch weak and default secrets, and keep .env files out of git, including its history.
- hash-sentinelCheck file hashes against your own threat feeds, MISP and optional public lookups, offline-first.
- header-sentinelFail CI when deployed URLs ship weak HTTP security headers.
- ioc-sentinelExtract indicators of compromise from threat reports, logs, emails and binaries, then export them as CSV, STIX 2.1, JSON or SARIF.
- license-sentinelA license compliance gate for npm projects, plus CycloneDX/SPDX SBOMs and a NOTICES file.
- malware-report-sentinelTurn a suspicious file into a structured malware triage report (Markdown, HTML or JSON) using static analysis only.
- phish-sentinelOffline phishing URL heuristics with an explainable score: lookalike and homograph domains, brand impersonation, IP hosts, user@host tricks, risky TLDs, shorteners, free hosting and redirect chains.
- redirect-sentinelFind open redirects and mixed content in a built site (offline) or a live crawl: http:// scripts and styles on HTTPS pages, insecure forms, client-side redirects fed by the URL, open redirect rules in _redirects/vercel.json/nginx, HTTPS downgrades, and optional safe redirect probing.
- sbom-diff-sentinelSee exactly what a pull request changes in your software bill of materials, and block the risky changes.
- securitytxt-sentinelValidate security.txt (RFC 9116) and robots.txt for your domains, or as local files in CI before you deploy them.
- sri-sentinelSubresource Integrity for your built site: generate, write and check it in CI.
- static-link-sentinelFail CI on broken links before you deploy.
- supply-chain-sentinelOne supply-chain gate for npm projects: risky dependency changes, license policy, new vulnerabilities and SBOM drift, and GitHub Actions hardening, with one config, one report and one exit code.
- tls-sentinelCheck TLS endpoints for expiring or untrusted certificates, hostname mismatches, missing intermediates, weak keys and signatures, TLS 1.0/1.1, weak ciphers and missing TLS 1.3.
- workflow-sentinelCatch insecure GitHub Actions workflows before they're merged.
Gauges package & repo quality · 8
- commit-gaugeA lightweight Conventional Commits linter for git hooks and CI, with clear fix-up suggestions.
- dead-deps-gaugeFind dependencies you declare but never use, and imports that are missing from package.json.
- engines-gaugeMake engines.node, your CI matrix, .nvmrc and your dependencies' engine requirements agree.
- monorepo-gaugeHygiene checks for npm, yarn and pnpm workspaces: circular internal deps, mismatched shared deps, version drift and missing fields.
- pkg-field-gaugeValidate package.json before you publish: exports/main/types consistency, files, repository, keywords and sideEffects.
- publish-gaugeCheck what npm publish will actually ship, before you publish it.
- readme-gaugeLint READMEs for missing sections, broken badges, stale install commands and missing usage examples.
- semver-gaugeA CI gate that checks your version bump matches what changed, and that the CHANGELOG has a real entry for it.
Beacons SEO & discoverability · 8
- answer-beaconCheck that your built static site is ready for answer engines and AI search: direct answers, FAQ schema, llms.txt, AI crawler access and E-E-A-T signals.
- canonical-beaconCheck canonical tags and hreflang clusters in your built static site: one canonical per page, pointing at a real, indexable, final URL, and hreflang clusters that are valid, self-referencing and reciprocal.
- heading-beaconAudit the heading structure of your built static site: one visible H1 per page, a logical H1 → H2 → H3 outline, and headings that actually say something.
- index-beaconCheck which pages of your built static site search engines are told not to index or follow, via meta robots and X-Robots-Tag, against the rules you actually intend.
- meta-beaconAudit titles, meta descriptions, Open Graph and Twitter/X cards across a built static site.
- orphan-beaconFind the pages in your built static site that nothing links to, and measure how many clicks every page is from the homepage.
- schema-beaconValidate JSON-LD and microdata against schema.org and Google rich-result requirements, across your whole built site.
- sitemap-beaconAudit XML sitemaps against your built static site: every URL should be a real, indexable, canonical page, and every such page should be listed.
Wardens privacy, compliance & accessibility · 8
- a11y-wardenWCAG 2.2 accessibility audit of static HTML for CI, with a baseline so you can adopt it on an existing site and only fail on new problems.
- consent-wardenCheck that your cookie banner actually behaves: nothing non-essential loads before opt-in, Reject is as easy as Accept, and consent can be withdrawn.
- dsar-wardenAssemble Data Subject Access Request exports from many systems into one structured, checksummed bundle, and catch the mistakes before you send it.
- gpc-wardenCheck that your built site honours Global Privacy Control: it reads the signal and suppresses sale/share trackers when it is on.
- pii-log-wardenFind and redact personal data in log files: emails, phone numbers, IP addresses, card numbers and national IDs.
- policy-drift-wardenCompare the trackers and third parties your built site actually loads with what its privacy policy says, and fail CI when they drift apart.
- retention-wardenLint your data-retention policy as code: every data category gets a period, a legal basis and an action at expiry.
- tracker-wardenInventory every third-party tracker, pixel and SDK your built static site loads, with vendor, company and category, and gate CI on an approved-vendor list.
Pulses web performance · 8
- bundle-pulseA CI gate for JavaScript and CSS bundle size: budgets on raw, gzip and brotli sizes, and a diff against your base branch.
- cache-pulseCheck that your static hosting config caches the right things: fingerprinted assets for a year, HTML revalidated, and nothing stale after a deploy.
- font-pulseAudit how your built site loads web fonts: font-display, preloads, WOFF2, subsetting, unused weights and total font payload.
- image-pulseFind oversized, unoptimised and wrongly formatted images in your built site, before they slow down your Largest Contentful Paint.
- lazy-pulseLazy-load what is below the fold, never the LCP image: a static lazy-loading audit for your built site.
- render-pulseFind the CSS and JavaScript that block first paint in your built site, with the defer, async, media or critical-CSS fix for each.
- thirdparty-pulseHow much does each third-party script cost your pages? Estimated bytes, requests and main-thread time per page and per vendor, checked against budgets.
- vitals-pulseFail CI when Core Web Vitals get worse: LCP, CLS and an INP proxy, gated on thresholds and on regressions against your baseline.