What supports The Word Index commonness tiers

Published for direct use; search approval pending. Its mechanically validated result is available, but the page remains noindex and outside sitemaps while human editorial and source-rights review is incomplete.

By The Word Index Updated July 2026

Result

Only 8.22% of safe records receive a tier directly from an eligible observed rank; the remainder use declared fallback or demotion rules.

Analysis population
394,445
Records assigned a tier from an eligible observed rank
8.22%
Records assigned by fallback or demotion
91.78%

Only 8.22% of safe records receive a tier directly from an eligible observed rank; the remainder use declared fallback or demotion rules.

Why the label is build-specific

The observed counts come from one pinned subtitle-derived artifact. Tier fallbacks also use exact source-list membership. A tier is therefore a reproducible ordering aid for this build, not universal commonness.

Higher-ranked and full result populations by length

The adjacent table supplies every exact value in the chart.

Read the accessible data table.

Population and exclusions

The exact population used by this version of the analysis.
Population stage Records (records)
Raw lexical ledger 394595
Excluded by product policy 150
Safe normalized-spelling records 394445
Records meeting this study's inclusion criteria 394445

How much of the index has observed frequency evidence

Observed evidence and tier assignment are deliberately separate.
Evidence class Records (records) Share (percent)
Retained subtitle-corpus observation 37318 9.46
Tier assigned from an eligible observed rank 32414 8.22
Tier assigned by a fallback or demotion rule 362031 91.78

Every tier assignment rule

Counts for the exact tier and reason stored on every safe record.
Tier Assignment reason Records (records) Share (percent)
1 observed-top-10000 9943 2.52
2 dual-source-membership-fallback 149 0.04
2 observed-rank-10001-25000 14951 3.79
3 enable-membership-fallback 140103 35.52
3 observed-below-rank-25000 7520 1.91
3 unreliable-token-demotion 14 0.00
4 corroborated-source-fallback 442 0.11
4 observed-tier-ineligible-corroborated-fallback 1302 0.33
5 observed-tier-ineligible-single-source-fallback 3582 0.91
5 single-source-fallback 216369 54.85
5 unreliable-token-demotion 70 0.02

How the extended result population changes with length

Counts compare tiers 1-2 with the complete five-tier safe population.
Letters Tiers 1-2 (records) All tiers (records) Additional records (records) Tiers 1-2 share (percent) All-to-higher ratio (ratio)
1 2 26 24 7.69 13.00
2 89 498 409 17.87 5.60
3 709 2290 1581 30.96 3.23
4 2067 7320 5253 28.24 3.54
5 3160 16135 12975 19.58 5.11
6 4035 30344 26309 13.30 7.52
7 4264 42758 38494 9.97 10.03
8 3698 52746 49048 7.01 14.26
9 2887 54755 51868 5.27 18.97
10 1918 49834 47916 3.85 25.98
11 1142 41449 40307 2.76 36.30
12 616 32573 31957 1.89 52.88
13 300 23627 23327 1.27 78.76
14 113 16243 16130 0.70 143.74
15 32 10335 10303 0.31 322.97
16 8 6230 6222 0.13 778.75
17 3 3610 3607 0.08 1203.33
18 0 1857 1857 0.00 not-defined
19 0 971 971 0.00 not-defined
20 0 465 465 0.00 not-defined
21 0 207 207 0.00 not-defined
22 0 94 94 0.00 not-defined
23 0 40 40 0.00 not-defined
24 0 19 19 0.00 not-defined
25 0 9 9 0.00 not-defined
27 0 4 4 0.00 not-defined
28 0 3 3 0.00 not-defined
29 0 2 2 0.00 not-defined
31 0 1 1 0.00 not-defined

Methodology

Research question: How much of the compiled lexical release has direct observed frequency evidence, and how much is ranked by bounded fallback rules?

Why it is useful: Users can judge what a commonness label means and avoid treating an estimated tier as a universal fact.

Included

  • commonness.safe-compiled-record: Include each safe normalized spelling retained in the compiled lexical release.

Excluded

  • commonness.product-policy: Exclude spellings matched by the exact first-party content-safety policy before aggregation.

Transformations

  1. commonness.evidence-class-v1: Classify each retained record by its sealed frequency-tier reason and presence of direct observed corpus evidence.

Calculations

research.count-v1
Count retained records in a declared evidence or tier group. Formula: count(records).
research.share-v1
Calculate an evidence class's exact share of the safe compiled population. Formula: 100 * numerator / denominator.
research.ratio-v1
Calculate an exact count ratio used to compare evidence cohorts. Formula: numerator / denominator.

Limitations

  • FrequencyWords is a bounded 2018 corpus-derived list and is not a timeless census of English.
  • Ranks are computed only among eligible normalized spellings after deterministic filtering.
  • Fallback source membership can prioritize results but is not direct usage evidence.
  • The study does not claim that a tier proves familiarity for an individual reader or community.

Versions, sources, and review

Study ID
study.commonness-evidence
Study version
1.0.0
Lexical data version
2026.07-lexical.3
Lexical build
b5dc0fb56b8de890
Study build
b25eeb6d0926c39e
Analysis contract
542778f7e22c8e11
Exact result digest
b0412cdb7abb7d98
Factual review
machine validated
Editorial review
pending
Source-rights review
pending
Search eligibility
No — The numeric artifact is available to users but remains noindex pending editorial and legal review.

dwyl-english-words git:8179fe68775df3f553ef19520db065228e65d1d3

source-membership; surface-spelling-as-listed.

Licence: Unlicense notice; upstream dataset rights unresolved. Legal review: pending. Redistribution review: pending.

  • The pinned repository licence is the Unlicense, but the pinned README says copyright in the extracted source list remains with Infochimps. Human legal review of the upstream rights chain is required.
  • Do not assert a redistribution right until the Infochimps-to-dwyl rights chain has been reviewed by a human.
  • Dialect or scope: unspecified

dwyl/english-words; the repository carries an Unlicense notice, while its pinned README attributes upstream copyright to Infochimps

enable artifact-sha256:d3fbe8485022088fcf527edcde2fbdc18b4bbc141ac58123c9adb462e086eaf7

source-membership; surface-spelling-as-listed.

Licence: Licence terms not stated by the source. Legal review: pending. Redistribution review: pending.

  • The downloaded artifact is content-pinned by SHA-256, but it contains no licence notice and the configured source page does not state reuse terms.
  • The artifact is reproducibly pinned but no affirmative reuse grant has been located; redistribution approval remains pending.
  • Dialect or scope: North-American-oriented; exact edition metadata unavailable

ENABLE (Enhanced North American Benchmark Lexicon); exact reuse terms are pending legal review

google-10000-english git:bdf4c221bc120b0b7f6c3f1eff1cc1abb975f8d8

source-membership; source-order.

Licence: No explicit licence grant in pinned repository metadata. Legal review: pending. Redistribution review: pending.

  • The exact pinned LICENSE.md describes provenance but contains no explicit public-domain dedication or licence grant. Human review is required.
  • The pinned repository documents provenance but does not supply an affirmative licence grant; do not describe it as public domain.
  • Dialect or scope: USA no-swears variant

first20hours/google-10000-english, derived from the Google Web Trillion Word Corpus and Peter Norvig's compilation

frequencywords-en-2018 git:525f9b560de45753a5ea01069454e72e9aa541c6; corpus-release:2018

source-token-observation; source-token-count.

Licence: CC BY-SA 4.0 (content). Legal review: pending. Redistribution review: pending.

  • The exact pinned README states MIT for code and CC BY-SA 4.0 for content. Human review is required for attribution and ShareAlike obligations on compiled and derived data.
  • Content is declared CC BY-SA 4.0. Attribution and ShareAlike treatment of the compiled frequency evidence and derived tiers require human approval.
  • Dialect or scope: unspecified

FrequencyWords content derived from OpenSubtitles, licensed CC BY-SA 4.0

offensive-term-policy 2026.07-review-pending.2

content_filter; search_results; result_counts.

Licence: Reuse terms not yet approved. Legal review: pending. Redistribution review: pending.

  • This bounded policy list cannot determine whether every usage is offensive or benign.

The Word Index first-party content-safety exclusion policy

Citations

  1. FrequencyWords English 2018, FrequencyWords, version git:525f9b560de45753a5ea01069454e72e9aa541c6; corpus-release:2018 (dataset).
  2. The Word Index data methodology, The Word Index, version 2026.07-lexical.3 (method).
  3. The Word Index content-safety exclusion policy, The Word Index, version 2026.07-review-pending.2 (policy).

Data and reproducibility

Source-rights review is pending. The available file contains aggregate computed measures, not a redistributed spelling list. Availability does not grant broader reuse rights; consult the source and licence register before reuse.

Download the canonical aggregate CSV

View the canonical JSON study bundle.

Reproduce this version from a clean checkout:

make studies_reproduce STUDY=commonness-evidence