Loading...
Diachronic corpus linguistics toolkit for Soviet census methodology texts (1926-1989): VLM extraction, orthography canonicalization, lemmatization, concordance, keyword, and structural-register analysis.
The Soviet census did not only count identities; it helped define the official vocabulary for nationality, ethnicity, language, and administrative belonging. I built the extraction infrastructure that turns scanned census front matter into an analyzable corpus, then used corpus linguistics methods to track how methodology language became more nominal, formulaic, and agency-erasing through 1979 before partially reversing in 1989.
Soviet Corpus is research infrastructure for a diachronic corpus linguistics study of Soviet census methodology texts. The public proof is the extraction pipeline, the normalized corpus, and the A1-A7 analysis layer showing how Soviet census register shifted from 1926 through 1989.
The source repository and raw extracted corpus remain private. This page shows aggregate metrics, architecture summaries, documentation-derived excerpts, and generated charts. It excludes secrets, raw scans, raw private datasets, and unreleased thesis text.
Published Soviet census methodology/preface series analyzed as the main chain.
Methodology and preface pages only, across the primary chain plus 1937/1939 vignettes.
All extracted page types, including structural metadata.
1926 has 37,819 tokens; 1989 has 3,116 tokens in the methodology+preface subset.
Nominal density rises from 1926 to a 1979 peak, then partially reverses in 1989.
Vocabulary richness contracts by roughly 40% from 1926 to the 1979 minimum; 1989 is raw TTR because it falls below the 5k STTR floor.
The README frames the project as a diachronic corpus linguistics study of Soviet census methodology texts, with VLM extraction as the infrastructure that built the corpus.
The A7 findings report POS distribution, vocabulary richness, and reflexive density on the methodology+preface subset.
The pipeline preserves original surface forms while canonicalizing historical orthography before pymorphy2 lemmatization, then supports concordance, keyword, diachronic/CITT, and structural analyses.
Computed on methodology+preface pages only. 1989 falls below the 5k-token STTR floor and is reported as raw TTR.
| Year | Tokens | N:V ratio | STTR/TTR | Reading |
|---|---|---|---|---|
| 1926 | 37819 | 5.84 | 0.287 | baseline |
| 1959 | 9528 | 8.27 | 0.201 | rising nominalization |
| 1970 | 7560 | 13.04 | 0.192 | rising nominalization |
| 1979 | 6351 | 16.2 | 0.173 | N:V peak; STTR minimum |
| 1989 | 3116 | 10.04 | 0.222 | partial glasnost-era reversal; raw TTR |
| Included publicly | Excluded publicly |
|---|---|
| Aggregate A7 metrics and generated charts | Raw scanned volumes and raw extracted corpus pages |
| Architecture/process summaries | Secrets, .env values, credentials, API tokens |
| Small documentation-derived excerpts and tables | Unreleased thesis prose or private datasets |