snowchive-meta/README.md
2026-11-19 04:39:11 +02:00

9.5 KiB

snowchive-meta

Metadata and text versions for the Snowchive archive of Snowden documents. This repo holds only the .json and .md files.

To download the PDFs, page images and thumbnails:

https://snowchive.org/about#downloading-all-data

The Snowchive server code lives here:

https://git.disroot.org/libroot/snowchive-server

Metadata layout

data/
  <document-id>/
    metadata.json
    <document-id>.md       # text version of the document
    versions/<id>/<id>.md  # other published document versions (some documents)
  FOIA/<document-id>/...             # FOIA releases (238)
  SIDtoday-archive/<document-id>/... # NSA SIDtoday newsletters (2,142)
project/
  assemblies.json  # documents rebuilt from separately published parts
  glossary.json    # codewords and terms, with definitions
  related.json     # document id -> list of related document ids
  changelog.json   # archive change log

metadata.json

Field Description
id The document's stable identifier: the name of its directory and the last part of its URL (/doc/{id}). It is not written inside metadata.json - the directory name is the id.
aka_id Other names the same document was released under, most often the file name another publisher or a mirror gave its PDF. A list of entries, each described below.
aka_id[].id The other name itself. Searchable, so a document can be found by a file name it circulated under elsewhere.
aka_id[].doc_source The web page where the document was released under that name, linked as "page" next to the alias.
aka_id[].doc_url The direct link to the PDF released under that name, linked as "PDF" next to the alias.
title The document's title, written by the curators - usually the title the document gives itself or the one its publisher used.
description A short summary of what the document is and says, shown under the publication list and in search results and cards. Descriptions are the work of archivists without inside knowledge - corrections welcome.
long_description A longer account of the contents where one has been written, behind "Show more info" on the document page; often empty. A description: search looks in both descriptions.
tags Free-text subjects added by the curators. Unlike the other lists they are not necessarily taken from the document's own vocabulary, so they are the loosest thing to browse or search by.
type The kind of release: REPORTED (a leaked document as an outlet published it), SIDTODAY (an article of the NSA's internal SIDtoday newsletter from The Intercept's bulk release) or FOIA (obtained through a public-records request). When it is empty, a document filed in the SIDtoday or FOIA folder takes that type and any other is REPORTED.
collection Which of the three collections - the tabs on Browse - the document belongs to: news (Reported), sidtoday-archive or foia. Documents filed under the SIDtoday and FOIA folders belong to those collections whatever this says; elsewhere an empty value means Reported, or FOIA for a document of type FOIA. FOIA releases are left out of browsing and search unless you include them in Preferences.
pages The page count of the PDF. When it is missing, the number of page images is used instead.
pdf_sha256 The SHA-256 of the PDF as archived.
archive_comment A note from the curators about this archive's copy - how it was assembled, where it was found, what differs between releases, or a known problem with it.

.document

Field Description
document.creator The agency that wrote the document, in its short form (NSA, GCHQ, FISC, DOJ, CSE ...) - not the agency it is about and not the one that published it. Empty when the author is not known.
document.date_created When the document was written, as precise as the document allows: 2013-03-20, 2013-03, 2013, or empty when unknown. Nothing is invented to fill the gaps, and nothing is guessed from the publication date.
document.banner The marking line verbatim - what the document says about itself, qualifiers and all. Intranet pages often carry a ceiling banner ("DYNAMIC PAGE - HIGHEST POSSIBLE CLASSIFICATION IS ...") while their content is portion-marked unclassified; the banner records the claim, not an assessment. Empty means the artifact has no marking line - nothing is reconstructed from portion marks.
document.classification The curator's assessment of the content's classification, as component values: the level (TOP SECRET, SECRET, CONFIDENTIAL, UNCLASSIFIED) and any control systems (COMINT, SI, TK, STRAP 1 ...), judged from portion marks. If a document contains multiple classifications, highest is used here.
document.dissemination The curator's assessment of the handling controls, as component values (FOUO, NOFORN, ORCON, REL TO ...). Assessed the same way as the classification.
document.genre The document's form, from a fixed vocabulary: slides, report, wiki, manual, guide, newsletter, other.
document.series.name For a document that prints a series roster (currently only in SIDtoday): the series name verbatim, as the document prints it. Documents with the same name are listed together on the document page and under Series.
document.series.part This document's 1-based position in the printed roster. Order comes from the artifact; parts of a series not yet in the archive show as gaps in the numbering, never renumbered away.
document.programs The codenames and programs the document names (XKEYSCORE, PRISM, ...), as curated values for browsing and search; the Glossary (project/glossary.json) explains many of them.
document.countries The countries the document mentions, by their English names. A mention, not a claim that the country was a target or a partner.
document.people The people who appear in the document: named in its text, or recognised in its images - for example Mobile Networks in My NOC World (gchq-mobile-networks-in-my-noc-world) lists King Charles and Queen Camilla, who appear only in a photograph on one of its slides. Usually written "Surname, Given name". A person appearing in these documents implies nothing about them beyond that appearance. The journalists who published the document are under publication[].reporters, not here.

.publication[]

One entry per story or book that published (or reported) the document.

Field Description
publication[].publisher Who published it: a news outlet, a book's publisher, or for a FOIA release the organization that obtained and released it. Browsable under Publishers - except entries marked reporting_only.
publication[].title The headline of the article, or the book's title. Shown on the document page and names the story on Articles.
publication[].link The URL of the article. It is the story's identity: documents whose entries share a link were published in the same report and are listed together on its article page.
publication[].date_published When this publication appeared, in the same YYYY-MM-DD / YYYY-MM / YYYY forms as the document date.
publication[].reporters The journalists bylined on the story, written "Surname, Given name". Browsable under Journalists and searchable with journalist:.
publication[].contributors People credited on the story without a byline - additional reporting, research, technical analysis. Shown on the document page and found by an all-fields search, but not listed among the journalists.
publication[].note Anything particular to this publication: that it showed only some of the pages, published them in a different order, redacted differently, or printed them in a book appendix.
publication[].version When this publication released a different copy from the one shown on the document page, the name of that copy in the document's versions/ folder; its entry then links to "view this copy". Without it, the entry published the copy shown on the page.
publication[].doc_source The web page where this publication presented the document (the outlet's own document page, a DocumentCloud page), linked as "document page". Where a copy came from is recorded here, on the publication that released it, not on the document.
publication[].doc_url The direct link to the PDF file this publication released, linked as "original PDF".
publication[].reporting_only true when the outlet reported the story without publishing the document - a reporting partner. The entry keeps its byline and its article, but the outlet is not counted as one of the document's publishers and no copy is linked from it.

.versions

Other published copies of a document (e.g. less redacted, with extra pages, or a different scan) are kept in its versions/ folder, one sub-folder per copy, each with its own PDF, text and page images. The versions object adds details about them, keyed by the copy's folder name.

Field Description
versions.{name}.changed_pages The pages of that copy, by its own page numbers, that differ from the copy shown on the document page. They are linked as "changed pages" next to the copy.
versions.{name}.pdf_sha256 The SHA-256 of that copy's PDF.

Issues/contact

Submit issues/feedback:

https://git.disroot.org/libroot/snowchive-meta

or to Codeberg (mirror): https://codeberg.org/libroot/snowchive-meta

Contact the maintainers via email:

https://libroot.org/about