first commit

This commit is contained in:
libroot 2026-11-19 04:40:59 +02:00
commit 8c4dcae475
Signed by: libroot
GPG key ID: 2AC68C029EBE9ACE
89 changed files with 24623 additions and 0 deletions

177
README.md Normal file
View file

@ -0,0 +1,177 @@
# Snowchive server
A self-contained web server for browsing and searching an archive of published
Snowden documents in Snowchive website. The server includes faceted browsing, a
boolean/fielded search engine with stemming and accent folding, a SIDtoday
serial reader, a read-only JSON API, sitemaps, and per-visitor preferences, pins
and history that live only in the visitor's cookies. The server uses just Go's
standard library.
No JavaScript is required to use the website. A file in `static/main.js` adds
progressive enhancements only. No database and no external services.
See it in action: https://snowchive.org
If you are looking for Snowchive data or metadata:
Data (PDFs, page images):
https://snowchive.org/about#downloading-all-data
Metadata (with extracted texts):
https://git.disroot.org/libroot/snowchive-meta
## Run
Full Snowchive data is required (containing all the PDFs and metadata) to run
the server. If you don't yet have the data, download it first:
https://snowchive.org/about#downloading-all-data
```sh
go run . -root=/path/to/snowchive-data/
```
## Building
If you want to build a binary to deploy one executable for production use:
```sh
go build -ldflags "-X main.buildCommit=$(git rev-parse --short HEAD)" -o snowchive .
./snowchive -root=/path/to/snowchive-data/ -base=https://snowchive.example.org
```
## Flags
| flag | purpose |
|---|---|
| `-root` | Snowchive data directory holding `data/` (the corpus, one directory per document) and `project/` (`glossary.json`, `assemblies.json`, `changelog.json`, `related.json`) (default: the working directory, `$SNOWCHIVE_ROOT`) |
| `-listen` | TCP address (default `127.0.0.1:8080`, `$SNOWCHIVE_LISTEN`). |
| `-base` | canonical `scheme://host` for absolute URLs in robots.txt and the sitemap; set it on anything internet-facing. Without it those URLs follow each request's host. |
| `-no-api` | disable the JSON API (`$SNOWCHIVE_NO_API=1`) |
| `-accel-files PREFIX` | behind nginx only: answer document files with an `X-Accel-Redirect` into this internal location (e.g. `/_files/`, an `alias` of the data directory), so nginx sends the bytes and this server only decides whether the file is served (`$SNOWCHIVE_ACCEL_FILES`) |
| `-accel-expires DURATION` | let the proxy cache a page served to a reader without cookies for this long (e.g. `10m`, sent as `X-Accel-Expires`); browsers keep the 10-second `max-age`. Clear the proxy cache after a restart with new data (`$SNOWCHIVE_ACCEL_EXPIRES`) |
| `-dev` | serve `templates/` and `static/` from the working directory, re-parse per request, verbose errors. Never in production. |
| `-check` | load the corpus, print the warnings and the tally, then exit non-zero if anything was reported - run it after a curation pass or in CI. |
| `-rebuild-related` | compute `related.json` entries for documents that lack one, or whose list links to a document no longer in the corpus, drop the entries of documents that are gone, then exit. It only fills gaps, so after a scoring change delete `related.json` first to rebuild every list. |
## What it costs to run
Measured on a 2-core VM against the 3,106-document corpus: ~7 seconds to load
(warm page cache), ~260 MB of live heap, and ~750 MB of address space under
sustained concurrent load.
## Directory layout
```
data/
<document-id>/ one directory per document
metadata.json document metadata
<anything>.pdf document PDF file
<anything>.md extracted text transcript from PDF
thumb.jpg, thumb.gif document thumbnail images
images/page-N.jpg PDF page scans
versions/<slug>/ another published copy that differs (e.g. less
redacted, extra pages): its own PDF,
.md and images/ (optional)
FOIA/<document-id>/ the FOIA document collection
SIDtoday-archive/<id>/ the bulk SIDtoday document collection
```
The files about the archive as a whole are in the project directory
(`-project`):
```
project/
glossary.json format documented in glossary.go
assemblies.json see "Assemblies" below
changelog.json snowchive changelog, format in changelog.go
related.json documents relating to each other, generated by
-rebuild-related
```
The corpus is read exactly once at startup and everything derived from it is
computed there. All of it lives in one `Corpus` value (`corpus.go`), which is
built whole and pre-rendered. The server writes nothing while it runs. Only
`-rebuild-related` writes a file (`related.json`, in the project directory).
## About metadata
`metadata.json` carries the title, description, tags, the `document`
block (creator, date_created, programs, classification, dissemination,
banner, genre, series, countries, people) and the `publication` list
(publisher, title, link, reporters, contributors, date_published, plus
`doc_source` and `doc_url` for the copy that entry published, and `note`
and `version` when that copy differs from the primary one). Every field
except `title` is optional. There is no document-level `doc_source`/`doc_url`,
because where a copy came from is a property of the publication that published
it.
A publication entry is an outlet that published the document. A partner in a
joint investigation that published the story but no documents is the one
exception (they have entry `"reporting_only": true`).
A publication that released a different copy of the same document is in
`version` directory. The directory `versions/<slug>/` is in the document
directory, holding that copy's PDF, `.md` and `images/`. The document page shows
the publication's `note` and links the copy, which has its own page at
`/doc/<id>/versions/<slug>`. Versions never count as documents. Search covers
the lines of a version's transcript that the main transcript lacks; results
label those matches with the copy they came from. A version no publication names
is listed as another copy.
The `versions` object records, per version directory, the SHA-256 of that copy's
PDF and the pages that differ from the main document, by the number in the page
image's filename:
`"versions": {"<slug>": {"pdf_sha256": "...", "changed_pages": [7, 12]}}`.
`aka_id` lists other names the document was published under, as an array of
objects: `{"id": "...", "doc_source": "...", "doc_url": "..."}`. An aka is a
pointer and stores no file, which is what separates it from a `versions/` entry.
More info about the metadata: https://git.disroot.org/libroot/snowchive-meta
## Assemblies
`assemblies.json` is a hand-maintained list of sets of pages drawn from several
documents and versions, for material that was published in pieces, such as a
slide deck. Each gets a page at `/explore/assemblies/<id>` showing all its
slides in one grid and lightbox, listed at `/explore/assemblies`
```json
[{
"id": "prism-overview",
"title": "PRISM/US-984XN briefing",
"note": "Shown at the top: how the set was put together and what is unproven.",
"slides": [
{"doc": "<document-id>", "pages": "1-7"},
{"doc": "<document-id>", "version": "<slug>", "pages": "8", "note": "Shown under the slide."},
{"doc": "<document-id>", "pages": "9,11",
"alt": [{"doc": "<document-id>", "version": "<slug>", "pages": "1-2"}]}
]
}]
```
An assembly is the archive's own arrangement, it's not an original document and
stores no files, and is neither counted nor searched.
## Tests
./tests/run.sh
TODO. The tests are currently written largely by Claude.
## Other/contact
The maintainers of this project do not endorse any public Snowchive instance
other than the official snowchive.org instance (unless you are a reputable
organization that wants to host a public instance).
Please submit issues/feedback on the Snowchive server's repository:
https://git.disroot.org/libroot/snowchive-server
or to Codeberg (mirror): https://codeberg.org/libroot/snowchive-server
Contact the maintainers via email:
https://libroot.org/about