7.9 KiB
Snowchive server
A self-contained web server for browsing and searching an archive of published Snowden documents in Snowchive website. The server includes faceted browsing, a boolean/fielded search engine with stemming and accent folding, a SIDtoday serial reader, a read-only JSON API, sitemaps, and per-visitor preferences, pins and history that live only in the visitor's cookies. The server uses just Go's standard library.
No JavaScript is required to use the website. A file in static/main.js adds
progressive enhancements only. No database and no external services.
See it in action: https://snowchive.org
If you are looking for Snowchive data or metadata:
Data (PDFs, page images):
https://snowchive.org/about#downloading-all-data
Metadata (with extracted texts):
https://git.disroot.org/libroot/snowchive-meta
Run
Full Snowchive data is required (containing all the PDFs and metadata) to run the server. If you don't yet have the data, download it first: https://snowchive.org/about#downloading-all-data
go run . -root=/path/to/snowchive-data/
Building
If you want to build a binary to deploy one executable for production use:
go build -ldflags "-X main.buildCommit=$(git rev-parse --short HEAD)" -o snowchive .
./snowchive -root=/path/to/snowchive-data/ -base=https://snowchive.example.org
Flags
| flag | purpose |
|---|---|
-root |
Snowchive data directory holding data/ (the corpus, one directory per document) and project/ (glossary.json, assemblies.json, changelog.json, related.json) (default: the working directory, $SNOWCHIVE_ROOT) |
-listen |
TCP address (default 127.0.0.1:8080, $SNOWCHIVE_LISTEN). |
-base |
canonical scheme://host for absolute URLs in robots.txt and the sitemap; set it on anything internet-facing. Without it those URLs follow each request's host. |
-no-api |
disable the JSON API ($SNOWCHIVE_NO_API=1) |
-accel-files PREFIX |
behind nginx only: answer document files with an X-Accel-Redirect into this internal location (e.g. /_files/, an alias of the data directory), so nginx sends the bytes and this server only decides whether the file is served ($SNOWCHIVE_ACCEL_FILES) |
-accel-expires DURATION |
let the proxy cache a page served to a reader without cookies for this long (e.g. 10m, sent as X-Accel-Expires); browsers keep the 10-second max-age. Clear the proxy cache after a restart with new data ($SNOWCHIVE_ACCEL_EXPIRES) |
-dev |
serve templates/ and static/ from the working directory, re-parse per request, verbose errors. Never in production. |
-check |
load the corpus, print the warnings and the tally, then exit non-zero if anything was reported - run it after a curation pass or in CI. |
-rebuild-related |
compute related.json entries for documents that lack one, or whose list links to a document no longer in the corpus, drop the entries of documents that are gone, then exit. It only fills gaps, so after a scoring change delete related.json first to rebuild every list. |
What it costs to run
Measured on a 2-core VM against the 3,106-document corpus: ~7 seconds to load (warm page cache), ~260 MB of live heap, and ~750 MB of address space under sustained concurrent load.
Directory layout
data/
<document-id>/ one directory per document
metadata.json document metadata
<anything>.pdf document PDF file
<anything>.md extracted text transcript from PDF
thumb.jpg, thumb.gif document thumbnail images
images/page-N.jpg PDF page scans
versions/<slug>/ another published copy that differs (e.g. less
redacted, extra pages): its own PDF,
.md and images/ (optional)
FOIA/<document-id>/ the FOIA document collection
SIDtoday-archive/<id>/ the bulk SIDtoday document collection
The files about the archive as a whole are in the project directory
(-project):
project/
glossary.json format documented in glossary.go
assemblies.json see "Assemblies" below
changelog.json snowchive changelog, format in changelog.go
related.json documents relating to each other, generated by
-rebuild-related
The corpus is read exactly once at startup and everything derived from it is
computed there. All of it lives in one Corpus value (corpus.go), which is
built whole and pre-rendered. The server writes nothing while it runs. Only
-rebuild-related writes a file (related.json, in the project directory).
About metadata
metadata.json carries the title, description, tags, the document
block (creator, date_created, programs, classification, dissemination,
banner, genre, series, countries, people) and the publication list
(publisher, title, link, reporters, contributors, date_published, plus
doc_source and doc_url for the copy that entry published, and note
and version when that copy differs from the primary one). Every field
except title is optional. There is no document-level doc_source/doc_url,
because where a copy came from is a property of the publication that published
it.
A publication entry is an outlet that published the document. A partner in a
joint investigation that published the story but no documents is the one
exception (they have entry "reporting_only": true).
A publication that released a different copy of the same document is in
version directory. The directory versions/<slug>/ is in the document
directory, holding that copy's PDF, .md and images/. The document page shows
the publication's note and links the copy, which has its own page at
/doc/<id>/versions/<slug>. Versions never count as documents. Search covers
the lines of a version's transcript that the main transcript lacks; results
label those matches with the copy they came from. A version no publication names
is listed as another copy.
The versions object records, per version directory, the SHA-256 of that copy's
PDF and the pages that differ from the main document, by the number in the page
image's filename:
"versions": {"<slug>": {"pdf_sha256": "...", "changed_pages": [7, 12]}}.
aka_id lists other names the document was published under, as an array of
objects: {"id": "...", "doc_source": "...", "doc_url": "..."}. An aka is a
pointer and stores no file, which is what separates it from a versions/ entry.
More info about the metadata: https://git.disroot.org/libroot/snowchive-meta
Assemblies
assemblies.json is a hand-maintained list of sets of pages drawn from several
documents and versions, for material that was published in pieces, such as a
slide deck. Each gets a page at /explore/assemblies/<id> showing all its
slides in one grid and lightbox, listed at /explore/assemblies
[{
"id": "prism-overview",
"title": "PRISM/US-984XN briefing",
"note": "Shown at the top: how the set was put together and what is unproven.",
"slides": [
{"doc": "<document-id>", "pages": "1-7"},
{"doc": "<document-id>", "version": "<slug>", "pages": "8", "note": "Shown under the slide."},
{"doc": "<document-id>", "pages": "9,11",
"alt": [{"doc": "<document-id>", "version": "<slug>", "pages": "1-2"}]}
]
}]
An assembly is the archive's own arrangement, it's not an original document and stores no files, and is neither counted nor searched.
Tests
./tests/run.sh
TODO. The tests are currently written largely by Claude.
Other/contact
The maintainers of this project do not endorse any public Snowchive instance other than the official snowchive.org instance (unless you are a reputable organization that wants to host a public instance).
Please submit issues/feedback on the Snowchive server's repository:
https://git.disroot.org/libroot/snowchive-server
or to Codeberg (mirror): https://codeberg.org/libroot/snowchive-server
Contact the maintainers via email: