# Snowchive server A self-contained web server for browsing and searching an archive of published Snowden documents in Snowchive website. The server includes faceted browsing, a boolean/fielded search engine with stemming and accent folding, a SIDtoday serial reader, a read-only JSON API, sitemaps, and per-visitor preferences, pins and history that live only in the visitor's cookies. The server uses just Go's standard library. No JavaScript is required to use the website. A file in `static/main.js` adds progressive enhancements only. No database and no external services. See it in action: https://snowchive.org If you are looking for Snowchive data or metadata: Data (PDFs, page images): https://snowchive.org/about#downloading-all-data Metadata (with extracted texts): https://git.disroot.org/libroot/snowchive-meta ## Run Full Snowchive data is required (containing all the PDFs and metadata) to run the server. If you don't yet have the data, download it first: https://snowchive.org/about#downloading-all-data ```sh go run . -root=/path/to/snowchive-data/ ``` ## Building If you want to build a binary to deploy one executable for production use: ```sh go build -ldflags "-X main.buildCommit=$(git rev-parse --short HEAD)" -o snowchive . ./snowchive -root=/path/to/snowchive-data/ -base=https://snowchive.example.org ``` ## Flags | flag | purpose | |---|---| | `-root` | Snowchive data directory holding `data/` (the corpus, one directory per document) and `project/` (`glossary.json`, `assemblies.json`, `changelog.json`, `related.json`) (default: the working directory, `$SNOWCHIVE_ROOT`) | | `-listen` | TCP address (default `127.0.0.1:8080`, `$SNOWCHIVE_LISTEN`). | | `-base` | canonical `scheme://host` for absolute URLs in robots.txt and the sitemap; set it on anything internet-facing. Without it those URLs follow each request's host. | | `-no-api` | disable the JSON API (`$SNOWCHIVE_NO_API=1`) | | `-accel-files PREFIX` | behind nginx only: answer document files with an `X-Accel-Redirect` into this internal location (e.g. `/_files/`, an `alias` of the data directory), so nginx sends the bytes and this server only decides whether the file is served (`$SNOWCHIVE_ACCEL_FILES`) | | `-accel-expires DURATION` | let the proxy cache a page served to a reader without cookies for this long (e.g. `10m`, sent as `X-Accel-Expires`); browsers keep the 10-second `max-age`. Clear the proxy cache after a restart with new data (`$SNOWCHIVE_ACCEL_EXPIRES`) | | `-dev` | serve `templates/` and `static/` from the working directory, re-parse per request, verbose errors. Never in production. | | `-check` | load the corpus, print the warnings and the tally, then exit non-zero if anything was reported - run it after a curation pass or in CI. | | `-rebuild-related` | compute `related.json` entries for documents that lack one, or whose list links to a document no longer in the corpus, drop the entries of documents that are gone, then exit. It only fills gaps, so after a scoring change delete `related.json` first to rebuild every list. | ## What it costs to run Measured on a 2-core VM against the 3,106-document corpus: ~7 seconds to load (warm page cache), ~260 MB of live heap, and ~750 MB of address space under sustained concurrent load. ## Directory layout ``` data/ / one directory per document metadata.json document metadata .pdf document PDF file .md extracted text transcript from PDF thumb.jpg, thumb.gif document thumbnail images images/page-N.jpg PDF page scans versions// another published copy that differs (e.g. less redacted, extra pages): its own PDF, .md and images/ (optional) FOIA// the FOIA document collection SIDtoday-archive// the bulk SIDtoday document collection ``` The files about the archive as a whole are in the project directory (`-project`): ``` project/ glossary.json format documented in glossary.go assemblies.json see "Assemblies" below changelog.json snowchive changelog, format in changelog.go related.json documents relating to each other, generated by -rebuild-related ``` The corpus is read exactly once at startup and everything derived from it is computed there. All of it lives in one `Corpus` value (`corpus.go`), which is built whole and pre-rendered. The server writes nothing while it runs. Only `-rebuild-related` writes a file (`related.json`, in the project directory). ## About metadata `metadata.json` carries the title, description, tags, the `document` block (creator, date_created, programs, classification, dissemination, banner, genre, series, countries, people) and the `publication` list (publisher, title, link, reporters, contributors, date_published, plus `doc_source` and `doc_url` for the copy that entry published, and `note` and `version` when that copy differs from the primary one). Every field except `title` is optional. There is no document-level `doc_source`/`doc_url`, because where a copy came from is a property of the publication that published it. A publication entry is an outlet that published the document. A partner in a joint investigation that published the story but no documents is the one exception (they have entry `"reporting_only": true`). A publication that released a different copy of the same document is in `version` directory. The directory `versions//` is in the document directory, holding that copy's PDF, `.md` and `images/`. The document page shows the publication's `note` and links the copy, which has its own page at `/doc//versions/`. Versions never count as documents. Search covers the lines of a version's transcript that the main transcript lacks; results label those matches with the copy they came from. A version no publication names is listed as another copy. The `versions` object records, per version directory, the SHA-256 of that copy's PDF and the pages that differ from the main document, by the number in the page image's filename: `"versions": {"": {"pdf_sha256": "...", "changed_pages": [7, 12]}}`. `aka_id` lists other names the document was published under, as an array of objects: `{"id": "...", "doc_source": "...", "doc_url": "..."}`. An aka is a pointer and stores no file, which is what separates it from a `versions/` entry. More info about the metadata: https://git.disroot.org/libroot/snowchive-meta ## Assemblies `assemblies.json` is a hand-maintained list of sets of pages drawn from several documents and versions, for material that was published in pieces, such as a slide deck. Each gets a page at `/explore/assemblies/` showing all its slides in one grid and lightbox, listed at `/explore/assemblies` ```json [{ "id": "prism-overview", "title": "PRISM/US-984XN briefing", "note": "Shown at the top: how the set was put together and what is unproven.", "slides": [ {"doc": "", "pages": "1-7"}, {"doc": "", "version": "", "pages": "8", "note": "Shown under the slide."}, {"doc": "", "pages": "9,11", "alt": [{"doc": "", "version": "", "pages": "1-2"}]} ] }] ``` An assembly is the archive's own arrangement, it's not an original document and stores no files, and is neither counted nor searched. ## Tests ./tests/run.sh TODO. The tests are currently written largely by Claude. ## Other/contact The maintainers of this project do not endorse any public Snowchive instance other than the official snowchive.org instance (unless you are a reputable organization that wants to host a public instance). Please submit issues/feedback on the Snowchive server's repository: https://git.disroot.org/libroot/snowchive-server or to Codeberg (mirror): https://codeberg.org/libroot/snowchive-server Contact the maintainers via email: https://libroot.org/about