snowchive-server/README.md
2026-11-19 04:40:59 +02:00

7.9 KiB

Snowchive server

A self-contained web server for browsing and searching an archive of published Snowden documents in Snowchive website. The server includes faceted browsing, a boolean/fielded search engine with stemming and accent folding, a SIDtoday serial reader, a read-only JSON API, sitemaps, and per-visitor preferences, pins and history that live only in the visitor's cookies. The server uses just Go's standard library.

No JavaScript is required to use the website. A file in static/main.js adds progressive enhancements only. No database and no external services.

See it in action: https://snowchive.org

If you are looking for Snowchive data or metadata:

Data (PDFs, page images):

https://snowchive.org/about#downloading-all-data

Metadata (with extracted texts):

https://git.disroot.org/libroot/snowchive-meta

Run

Full Snowchive data is required (containing all the PDFs and metadata) to run the server. If you don't yet have the data, download it first: https://snowchive.org/about#downloading-all-data

go run . -root=/path/to/snowchive-data/

Building

If you want to build a binary to deploy one executable for production use:

go build -ldflags "-X main.buildCommit=$(git rev-parse --short HEAD)" -o snowchive .
./snowchive -root=/path/to/snowchive-data/ -base=https://snowchive.example.org

Flags

flag purpose
-root Snowchive data directory holding data/ (the corpus, one directory per document) and project/ (glossary.json, assemblies.json, changelog.json, related.json) (default: the working directory, $SNOWCHIVE_ROOT)
-listen TCP address (default 127.0.0.1:8080, $SNOWCHIVE_LISTEN).
-base canonical scheme://host for absolute URLs in robots.txt and the sitemap; set it on anything internet-facing. Without it those URLs follow each request's host.
-no-api disable the JSON API ($SNOWCHIVE_NO_API=1)
-accel-files PREFIX behind nginx only: answer document files with an X-Accel-Redirect into this internal location (e.g. /_files/, an alias of the data directory), so nginx sends the bytes and this server only decides whether the file is served ($SNOWCHIVE_ACCEL_FILES)
-accel-expires DURATION let the proxy cache a page served to a reader without cookies for this long (e.g. 10m, sent as X-Accel-Expires); browsers keep the 10-second max-age. Clear the proxy cache after a restart with new data ($SNOWCHIVE_ACCEL_EXPIRES)
-dev serve templates/ and static/ from the working directory, re-parse per request, verbose errors. Never in production.
-check load the corpus, print the warnings and the tally, then exit non-zero if anything was reported - run it after a curation pass or in CI.
-rebuild-related compute related.json entries for documents that lack one, or whose list links to a document no longer in the corpus, drop the entries of documents that are gone, then exit. It only fills gaps, so after a scoring change delete related.json first to rebuild every list.

What it costs to run

Measured on a 2-core VM against the 3,106-document corpus: ~7 seconds to load (warm page cache), ~260 MB of live heap, and ~750 MB of address space under sustained concurrent load.

Directory layout

data/
  <document-id>/                one directory per document
    metadata.json               document metadata
    <anything>.pdf              document PDF file
    <anything>.md               extracted text transcript from PDF
    thumb.jpg, thumb.gif        document thumbnail images
    images/page-N.jpg           PDF page scans
    versions/<slug>/            another published copy that differs (e.g. less
                                redacted, extra pages): its own PDF,
                                .md and images/ (optional)
  FOIA/<document-id>/           the FOIA document collection
  SIDtoday-archive/<id>/        the bulk SIDtoday document collection

The files about the archive as a whole are in the project directory (-project):

project/
  glossary.json                 format documented in glossary.go
  assemblies.json               see "Assemblies" below
  changelog.json                snowchive changelog, format in changelog.go
  related.json                  documents relating to each other, generated by
                                -rebuild-related

The corpus is read exactly once at startup and everything derived from it is computed there. All of it lives in one Corpus value (corpus.go), which is built whole and pre-rendered. The server writes nothing while it runs. Only -rebuild-related writes a file (related.json, in the project directory).

About metadata

metadata.json carries the title, description, tags, the document block (creator, date_created, programs, classification, dissemination, banner, genre, series, countries, people) and the publication list (publisher, title, link, reporters, contributors, date_published, plus doc_source and doc_url for the copy that entry published, and note and version when that copy differs from the primary one). Every field except title is optional. There is no document-level doc_source/doc_url, because where a copy came from is a property of the publication that published it.

A publication entry is an outlet that published the document. A partner in a joint investigation that published the story but no documents is the one exception (they have entry "reporting_only": true).

A publication that released a different copy of the same document is in version directory. The directory versions/<slug>/ is in the document directory, holding that copy's PDF, .md and images/. The document page shows the publication's note and links the copy, which has its own page at /doc/<id>/versions/<slug>. Versions never count as documents. Search covers the lines of a version's transcript that the main transcript lacks; results label those matches with the copy they came from. A version no publication names is listed as another copy.

The versions object records, per version directory, the SHA-256 of that copy's PDF and the pages that differ from the main document, by the number in the page image's filename: "versions": {"<slug>": {"pdf_sha256": "...", "changed_pages": [7, 12]}}.

aka_id lists other names the document was published under, as an array of objects: {"id": "...", "doc_source": "...", "doc_url": "..."}. An aka is a pointer and stores no file, which is what separates it from a versions/ entry.

More info about the metadata: https://git.disroot.org/libroot/snowchive-meta

Assemblies

assemblies.json is a hand-maintained list of sets of pages drawn from several documents and versions, for material that was published in pieces, such as a slide deck. Each gets a page at /explore/assemblies/<id> showing all its slides in one grid and lightbox, listed at /explore/assemblies

[{
  "id": "prism-overview",
  "title": "PRISM/US-984XN briefing",
  "note": "Shown at the top: how the set was put together and what is unproven.",
  "slides": [
    {"doc": "<document-id>", "pages": "1-7"},
    {"doc": "<document-id>", "version": "<slug>", "pages": "8", "note": "Shown under the slide."},
    {"doc": "<document-id>", "pages": "9,11",
     "alt": [{"doc": "<document-id>", "version": "<slug>", "pages": "1-2"}]}
  ]
}]

An assembly is the archive's own arrangement, it's not an original document and stores no files, and is neither counted nor searched.

Tests

./tests/run.sh

TODO. The tests are currently written largely by Claude.

Other/contact

The maintainers of this project do not endorse any public Snowchive instance other than the official snowchive.org instance (unless you are a reputable organization that wants to host a public instance).

Please submit issues/feedback on the Snowchive server's repository:

https://git.disroot.org/libroot/snowchive-server

or to Codeberg (mirror): https://codeberg.org/libroot/snowchive-server

Contact the maintainers via email:

https://libroot.org/about