177 lines
7.9 KiB
Markdown
177 lines
7.9 KiB
Markdown
# Snowchive server
|
|
|
|
A self-contained web server for browsing and searching an archive of published
|
|
Snowden documents in Snowchive website. The server includes faceted browsing, a
|
|
boolean/fielded search engine with stemming and accent folding, a SIDtoday
|
|
serial reader, a read-only JSON API, sitemaps, and per-visitor preferences, pins
|
|
and history that live only in the visitor's cookies. The server uses just Go's
|
|
standard library.
|
|
|
|
No JavaScript is required to use the website. A file in `static/main.js` adds
|
|
progressive enhancements only. No database and no external services.
|
|
|
|
See it in action: https://snowchive.org
|
|
|
|
If you are looking for Snowchive data or metadata:
|
|
|
|
Data (PDFs, page images):
|
|
|
|
https://snowchive.org/about#downloading-all-data
|
|
|
|
Metadata (with extracted texts):
|
|
|
|
https://git.disroot.org/libroot/snowchive-meta
|
|
|
|
## Run
|
|
|
|
Full Snowchive data is required (containing all the PDFs and metadata) to run
|
|
the server. If you don't yet have the data, download it first:
|
|
https://snowchive.org/about#downloading-all-data
|
|
|
|
```sh
|
|
go run . -root=/path/to/snowchive-data/
|
|
```
|
|
|
|
## Building
|
|
|
|
If you want to build a binary to deploy one executable for production use:
|
|
|
|
```sh
|
|
go build -ldflags "-X main.buildCommit=$(git rev-parse --short HEAD)" -o snowchive .
|
|
./snowchive -root=/path/to/snowchive-data/ -base=https://snowchive.example.org
|
|
```
|
|
|
|
## Flags
|
|
|
|
| flag | purpose |
|
|
|---|---|
|
|
| `-root` | Snowchive data directory holding `data/` (the corpus, one directory per document) and `project/` (`glossary.json`, `assemblies.json`, `changelog.json`, `related.json`) (default: the working directory, `$SNOWCHIVE_ROOT`) |
|
|
| `-listen` | TCP address (default `127.0.0.1:8080`, `$SNOWCHIVE_LISTEN`). |
|
|
| `-base` | canonical `scheme://host` for absolute URLs in robots.txt and the sitemap; set it on anything internet-facing. Without it those URLs follow each request's host. |
|
|
| `-no-api` | disable the JSON API (`$SNOWCHIVE_NO_API=1`) |
|
|
| `-accel-files PREFIX` | behind nginx only: answer document files with an `X-Accel-Redirect` into this internal location (e.g. `/_files/`, an `alias` of the data directory), so nginx sends the bytes and this server only decides whether the file is served (`$SNOWCHIVE_ACCEL_FILES`) |
|
|
| `-accel-expires DURATION` | let the proxy cache a page served to a reader without cookies for this long (e.g. `10m`, sent as `X-Accel-Expires`); browsers keep the 10-second `max-age`. Clear the proxy cache after a restart with new data (`$SNOWCHIVE_ACCEL_EXPIRES`) |
|
|
| `-dev` | serve `templates/` and `static/` from the working directory, re-parse per request, verbose errors. Never in production. |
|
|
| `-check` | load the corpus, print the warnings and the tally, then exit non-zero if anything was reported - run it after a curation pass or in CI. |
|
|
| `-rebuild-related` | compute `related.json` entries for documents that lack one, or whose list links to a document no longer in the corpus, drop the entries of documents that are gone, then exit. It only fills gaps, so after a scoring change delete `related.json` first to rebuild every list. |
|
|
|
|
## What it costs to run
|
|
|
|
Measured on a 2-core VM against the 3,106-document corpus: ~7 seconds to load
|
|
(warm page cache), ~260 MB of live heap, and ~750 MB of address space under
|
|
sustained concurrent load.
|
|
|
|
## Directory layout
|
|
|
|
```
|
|
data/
|
|
<document-id>/ one directory per document
|
|
metadata.json document metadata
|
|
<anything>.pdf document PDF file
|
|
<anything>.md extracted text transcript from PDF
|
|
thumb.jpg, thumb.gif document thumbnail images
|
|
images/page-N.jpg PDF page scans
|
|
versions/<slug>/ another published copy that differs (e.g. less
|
|
redacted, extra pages): its own PDF,
|
|
.md and images/ (optional)
|
|
FOIA/<document-id>/ the FOIA document collection
|
|
SIDtoday-archive/<id>/ the bulk SIDtoday document collection
|
|
```
|
|
|
|
The files about the archive as a whole are in the project directory
|
|
(`-project`):
|
|
|
|
```
|
|
project/
|
|
glossary.json format documented in glossary.go
|
|
assemblies.json see "Assemblies" below
|
|
changelog.json snowchive changelog, format in changelog.go
|
|
related.json documents relating to each other, generated by
|
|
-rebuild-related
|
|
```
|
|
|
|
The corpus is read exactly once at startup and everything derived from it is
|
|
computed there. All of it lives in one `Corpus` value (`corpus.go`), which is
|
|
built whole and pre-rendered. The server writes nothing while it runs. Only
|
|
`-rebuild-related` writes a file (`related.json`, in the project directory).
|
|
|
|
## About metadata
|
|
|
|
`metadata.json` carries the title, description, tags, the `document`
|
|
block (creator, date_created, programs, classification, dissemination,
|
|
banner, genre, series, countries, people) and the `publication` list
|
|
(publisher, title, link, reporters, contributors, date_published, plus
|
|
`doc_source` and `doc_url` for the copy that entry published, and `note`
|
|
and `version` when that copy differs from the primary one). Every field
|
|
except `title` is optional. There is no document-level `doc_source`/`doc_url`,
|
|
because where a copy came from is a property of the publication that published
|
|
it.
|
|
|
|
A publication entry is an outlet that published the document. A partner in a
|
|
joint investigation that published the story but no documents is the one
|
|
exception (they have entry `"reporting_only": true`).
|
|
|
|
A publication that released a different copy of the same document is in
|
|
`version` directory. The directory `versions/<slug>/` is in the document
|
|
directory, holding that copy's PDF, `.md` and `images/`. The document page shows
|
|
the publication's `note` and links the copy, which has its own page at
|
|
`/doc/<id>/versions/<slug>`. Versions never count as documents. Search covers
|
|
the lines of a version's transcript that the main transcript lacks; results
|
|
label those matches with the copy they came from. A version no publication names
|
|
is listed as another copy.
|
|
|
|
The `versions` object records, per version directory, the SHA-256 of that copy's
|
|
PDF and the pages that differ from the main document, by the number in the page
|
|
image's filename:
|
|
`"versions": {"<slug>": {"pdf_sha256": "...", "changed_pages": [7, 12]}}`.
|
|
|
|
`aka_id` lists other names the document was published under, as an array of
|
|
objects: `{"id": "...", "doc_source": "...", "doc_url": "..."}`. An aka is a
|
|
pointer and stores no file, which is what separates it from a `versions/` entry.
|
|
|
|
More info about the metadata: https://git.disroot.org/libroot/snowchive-meta
|
|
|
|
## Assemblies
|
|
|
|
`assemblies.json` is a hand-maintained list of sets of pages drawn from several
|
|
documents and versions, for material that was published in pieces, such as a
|
|
slide deck. Each gets a page at `/explore/assemblies/<id>` showing all its
|
|
slides in one grid and lightbox, listed at `/explore/assemblies`
|
|
|
|
```json
|
|
[{
|
|
"id": "prism-overview",
|
|
"title": "PRISM/US-984XN briefing",
|
|
"note": "Shown at the top: how the set was put together and what is unproven.",
|
|
"slides": [
|
|
{"doc": "<document-id>", "pages": "1-7"},
|
|
{"doc": "<document-id>", "version": "<slug>", "pages": "8", "note": "Shown under the slide."},
|
|
{"doc": "<document-id>", "pages": "9,11",
|
|
"alt": [{"doc": "<document-id>", "version": "<slug>", "pages": "1-2"}]}
|
|
]
|
|
}]
|
|
```
|
|
An assembly is the archive's own arrangement, it's not an original document and
|
|
stores no files, and is neither counted nor searched.
|
|
|
|
## Tests
|
|
|
|
./tests/run.sh
|
|
|
|
TODO. The tests are currently written largely by Claude.
|
|
|
|
## Other/contact
|
|
|
|
The maintainers of this project do not endorse any public Snowchive instance
|
|
other than the official snowchive.org instance (unless you are a reputable
|
|
organization that wants to host a public instance).
|
|
|
|
Please submit issues/feedback on the Snowchive server's repository:
|
|
|
|
https://git.disroot.org/libroot/snowchive-server
|
|
|
|
or to Codeberg (mirror): https://codeberg.org/libroot/snowchive-server
|
|
|
|
Contact the maintainers via email:
|
|
|
|
https://libroot.org/about
|