OCR quality estimate for Compact Memory pages
http://data.judaicalink.org/datasets/cm-ocr-quality
Search this dataset In the catalogue · Query
Description
- rdf:type
- void:Dataset
- rdf:type
- dcat:Dataset
- dcterms:abstract
- An estimate of the OCR quality of every page of Compact Memory, and the mean per journal.
- dcterms:description
An estimate of the OCR quality of every page of Compact Memory, and the mean per journal. Per page (`jl:ocrKnownWordShare`): the share of its words (letters only, at least two characters, lowercased) that occur at least 10 times in every 20th page of the corpus. Per journal (`jl…
An estimate of the OCR quality of every page of Compact Memory, and the mean per journal. Per page (`jl:ocrKnownWordShare`): the share of its words (letters only, at least two characters, lowercased) that occur at least 10 times in every 20th page of the corpus. Per journal (`jl:meanOcrKnownWordShare`): the mean over its pages. An estimate, not a measurement: Hebrew, Yiddish and Russian pages score lower partly because the word list is dominated by German. The statements are about the pages and journals of compact-memory; the texts come from the full text of the FID portal's Compact Memory search.- identifier
- cm-ocr-quality
- dcterms:license
- https://creativecommons.org/publicdomain/zero/1.0/
- dcterms:title
- OCR quality estimate for Compact Memory pages
- dcterms:type
- datasets/vocabulary/method/automated
- dcterms:type
- datasets/vocabulary/method/statistical
- dcterms:type
- datasets/vocabulary/kind/complement
- dcat:hasVersion
- datasets/cm-ocr-quality/2026-10-12
- dcat:landingPage
- https://judaicalink.org/
datasets/cm-ocr-quality/2026-10-12
- rdf:type
- void:Dataset
- rdf:type
- dcat:Dataset
- rdf:type
- prov:Entity
- void:triples
- 846808
- dcterms:creator
- datasets/agents/persons/1
- dcterms:issued
- 2026-10-12
- void:classes
- 0
- void:distinctSubjects
- 846808
- void:properties
- 2
- dcat:distribution
- dumps/cm-ocr-quality/2026-10-12/cm-ocr-quality-meta.ttl.gz
- dcat:distribution
- dumps/cm-ocr-quality/2026-10-12/cm-ocr-quality.ttl.gz
- dcat:isVersionOf
- datasets/cm-ocr-quality
- dcat:version
- 2026-10-12
- prov:wasDerivedFrom
- datasets/compact-memory/2026-10-12
- prov:wasGeneratedBy
- datasets/cm-ocr-quality/2026-10-12/generation
datasets/cm-ocr-quality/2026-10-12/generation
- rdf:type
- prov:Activity
- rdfs:comment
Input not published (inputs:compact-memory/fulltext-2026-10-05-v2, 43 files, 2,097,436,345 bytes; named by checksum in the run record): Compact Memory full text for the Solr core cm_meta (bundle v2). 846,500 pages of Compact Memory as documents for the Solr core cm_meta, one JSON…
Input not published (inputs:compact-memory/fulltext-2026-10-05-v2, 43 files, 2,097,436,345 bytes; named by checksum in the run record): Compact Memory full text for the Solr core cm_meta (bundle v2). 846,500 pages of Compact Memory as documents for the Solr core cm_meta, one JSON object per line in 35 gzipped parts (docs-NNNN.jsonl.gz): page VLID, journal, date of issue, page label, volume, OCR text. manifest.json lists the parts with their checksums, the pages left out (no OCR file, no metadata, no text) and the corrections (Hebrew calendar years converted to Gregorian, which can be one year off; '(null)' removed; volumes rebuilt from the MODS records). date_review.tsv lists 513 pages whose year is isolated from the rest of their journal (not changed). The bundle also holds its README.md, the scripts that made and corrected it (extract_cm_fulltext.py, fix_bundle.py with fix_bundle.log), the loader for Solr (load_cm_fulltext.py) and dropped.tsv.gz (the pages left out, with the reason). Origin: Made on 2026-10-05 by extract_cm_fulltext.py (cm_indexer in the repository of the FID portal, handed over to Frankfurt) from the OCR XML files of Compact Memory on the JudaicaLink labs server in Mannheim, the page table (compact-memory-supp) and the MODS records (inputs:compact-memory/mods-2020-06-15); a copy is on that server.- rdfs:comment
- Word list: 116,604 words occurring at least 10 times in every 20th page.
- rdfs:seeAlso
- https://github.com/judaicalink/judaicalink-data/blob/main/generators/cm-ocr-quality/runs/2026-10-12.json
- prov:endedAtTime
- 2026-10-12T07:58:18Z
- prov:startedAtTime
- 2026-10-12T07:55:18Z
- prov:used
- dumps/compact-memory/2026-10-12/compact-memory-pages.ttl.gz
- prov:used
- https://github.com/judaicalink/judaicalink-data/blob/b8e227c3dc120eeabdda242096eca3683fbc2683/generators/cm-ocr-quality/generate.py
- prov:wasAssociatedWith
- datasets/agents/persons/1
- prov:wasAssociatedWith
- datasets/agents/software/judaicalink-data
datasets/cm-ocr-quality/2026-10-12/metadata
- rdf:type
- prov:Activity
- rdfs:comment
- Rendered by the framework from the catalogue entry, the run record of the version and statistics measured on its files (decision 0018).
- prov:endedAtTime
- 2026-10-12T07:59:32Z
- prov:used
- https://github.com/judaicalink/judaicalink-data/tree/b8e227c3dc120eeabdda242096eca3683fbc2683
- prov:wasAssociatedWith
- datasets/agents/software/judaicalink-data
dumps/cm-ocr-quality/2026-10-12/cm-ocr-quality-meta.ttl.gz
- rdf:type
- dcat:Distribution
- dcterms:description
- Metadata of this version (this file).
- dcat:accessURL
- https://web.judaicalink.org/dataindex/?file=cm-ocr-quality/2026-10-12/cm-ocr-quality-meta.ttl.gz
- dcat:compressFormat
- http://www.iana.org/assignments/media-types/application/gzip
- dcat:mediaType
- http://www.iana.org/assignments/media-types/text/turtle
- prov:wasGeneratedBy
- datasets/cm-ocr-quality/2026-10-12/metadata
dumps/cm-ocr-quality/2026-10-12/cm-ocr-quality.ttl.gz
- rdf:type
- dcat:Distribution
- void:triples
- 846808
- dcat:accessURL
- https://web.judaicalink.org/dataindex/?file=cm-ocr-quality/2026-10-12/cm-ocr-quality.ttl.gz
- dcat:compressFormat
- http://www.iana.org/assignments/media-types/application/gzip
- dcat:mediaType
- http://www.iana.org/assignments/media-types/text/turtle
- http://spdx.org/rdf/terms#checksum
- dumps/cm-ocr-quality/2026-10-12/cm-ocr-quality.ttl.gz#sha256
- dcat:byteSize
- 7353618
dumps/cm-ocr-quality/2026-10-12/cm-ocr-quality.ttl.gz#sha256
- rdf:type
- http://spdx.org/rdf/terms#Checksum
- http://spdx.org/rdf/terms#algorithm
- http://spdx.org/rdf/terms#checksumAlgorithm_sha256
- http://spdx.org/rdf/terms#checksumValue
- 50de7954aa34356a4e7795d7ada520065bfb3756969fb50e53bb6f4829207afb