first commit
This commit is contained in:
commit
8c4dcae475
89 changed files with 24623 additions and 0 deletions
177
README.md
Normal file
177
README.md
Normal file
|
|
@ -0,0 +1,177 @@
|
|||
# Snowchive server
|
||||
|
||||
A self-contained web server for browsing and searching an archive of published
|
||||
Snowden documents in Snowchive website. The server includes faceted browsing, a
|
||||
boolean/fielded search engine with stemming and accent folding, a SIDtoday
|
||||
serial reader, a read-only JSON API, sitemaps, and per-visitor preferences, pins
|
||||
and history that live only in the visitor's cookies. The server uses just Go's
|
||||
standard library.
|
||||
|
||||
No JavaScript is required to use the website. A file in `static/main.js` adds
|
||||
progressive enhancements only. No database and no external services.
|
||||
|
||||
See it in action: https://snowchive.org
|
||||
|
||||
If you are looking for Snowchive data or metadata:
|
||||
|
||||
Data (PDFs, page images):
|
||||
|
||||
https://snowchive.org/about#downloading-all-data
|
||||
|
||||
Metadata (with extracted texts):
|
||||
|
||||
https://git.disroot.org/libroot/snowchive-meta
|
||||
|
||||
## Run
|
||||
|
||||
Full Snowchive data is required (containing all the PDFs and metadata) to run
|
||||
the server. If you don't yet have the data, download it first:
|
||||
https://snowchive.org/about#downloading-all-data
|
||||
|
||||
```sh
|
||||
go run . -root=/path/to/snowchive-data/
|
||||
```
|
||||
|
||||
## Building
|
||||
|
||||
If you want to build a binary to deploy one executable for production use:
|
||||
|
||||
```sh
|
||||
go build -ldflags "-X main.buildCommit=$(git rev-parse --short HEAD)" -o snowchive .
|
||||
./snowchive -root=/path/to/snowchive-data/ -base=https://snowchive.example.org
|
||||
```
|
||||
|
||||
## Flags
|
||||
|
||||
| flag | purpose |
|
||||
|---|---|
|
||||
| `-root` | Snowchive data directory holding `data/` (the corpus, one directory per document) and `project/` (`glossary.json`, `assemblies.json`, `changelog.json`, `related.json`) (default: the working directory, `$SNOWCHIVE_ROOT`) |
|
||||
| `-listen` | TCP address (default `127.0.0.1:8080`, `$SNOWCHIVE_LISTEN`). |
|
||||
| `-base` | canonical `scheme://host` for absolute URLs in robots.txt and the sitemap; set it on anything internet-facing. Without it those URLs follow each request's host. |
|
||||
| `-no-api` | disable the JSON API (`$SNOWCHIVE_NO_API=1`) |
|
||||
| `-accel-files PREFIX` | behind nginx only: answer document files with an `X-Accel-Redirect` into this internal location (e.g. `/_files/`, an `alias` of the data directory), so nginx sends the bytes and this server only decides whether the file is served (`$SNOWCHIVE_ACCEL_FILES`) |
|
||||
| `-accel-expires DURATION` | let the proxy cache a page served to a reader without cookies for this long (e.g. `10m`, sent as `X-Accel-Expires`); browsers keep the 10-second `max-age`. Clear the proxy cache after a restart with new data (`$SNOWCHIVE_ACCEL_EXPIRES`) |
|
||||
| `-dev` | serve `templates/` and `static/` from the working directory, re-parse per request, verbose errors. Never in production. |
|
||||
| `-check` | load the corpus, print the warnings and the tally, then exit non-zero if anything was reported - run it after a curation pass or in CI. |
|
||||
| `-rebuild-related` | compute `related.json` entries for documents that lack one, or whose list links to a document no longer in the corpus, drop the entries of documents that are gone, then exit. It only fills gaps, so after a scoring change delete `related.json` first to rebuild every list. |
|
||||
|
||||
## What it costs to run
|
||||
|
||||
Measured on a 2-core VM against the 3,106-document corpus: ~7 seconds to load
|
||||
(warm page cache), ~260 MB of live heap, and ~750 MB of address space under
|
||||
sustained concurrent load.
|
||||
|
||||
## Directory layout
|
||||
|
||||
```
|
||||
data/
|
||||
<document-id>/ one directory per document
|
||||
metadata.json document metadata
|
||||
<anything>.pdf document PDF file
|
||||
<anything>.md extracted text transcript from PDF
|
||||
thumb.jpg, thumb.gif document thumbnail images
|
||||
images/page-N.jpg PDF page scans
|
||||
versions/<slug>/ another published copy that differs (e.g. less
|
||||
redacted, extra pages): its own PDF,
|
||||
.md and images/ (optional)
|
||||
FOIA/<document-id>/ the FOIA document collection
|
||||
SIDtoday-archive/<id>/ the bulk SIDtoday document collection
|
||||
```
|
||||
|
||||
The files about the archive as a whole are in the project directory
|
||||
(`-project`):
|
||||
|
||||
```
|
||||
project/
|
||||
glossary.json format documented in glossary.go
|
||||
assemblies.json see "Assemblies" below
|
||||
changelog.json snowchive changelog, format in changelog.go
|
||||
related.json documents relating to each other, generated by
|
||||
-rebuild-related
|
||||
```
|
||||
|
||||
The corpus is read exactly once at startup and everything derived from it is
|
||||
computed there. All of it lives in one `Corpus` value (`corpus.go`), which is
|
||||
built whole and pre-rendered. The server writes nothing while it runs. Only
|
||||
`-rebuild-related` writes a file (`related.json`, in the project directory).
|
||||
|
||||
## About metadata
|
||||
|
||||
`metadata.json` carries the title, description, tags, the `document`
|
||||
block (creator, date_created, programs, classification, dissemination,
|
||||
banner, genre, series, countries, people) and the `publication` list
|
||||
(publisher, title, link, reporters, contributors, date_published, plus
|
||||
`doc_source` and `doc_url` for the copy that entry published, and `note`
|
||||
and `version` when that copy differs from the primary one). Every field
|
||||
except `title` is optional. There is no document-level `doc_source`/`doc_url`,
|
||||
because where a copy came from is a property of the publication that published
|
||||
it.
|
||||
|
||||
A publication entry is an outlet that published the document. A partner in a
|
||||
joint investigation that published the story but no documents is the one
|
||||
exception (they have entry `"reporting_only": true`).
|
||||
|
||||
A publication that released a different copy of the same document is in
|
||||
`version` directory. The directory `versions/<slug>/` is in the document
|
||||
directory, holding that copy's PDF, `.md` and `images/`. The document page shows
|
||||
the publication's `note` and links the copy, which has its own page at
|
||||
`/doc/<id>/versions/<slug>`. Versions never count as documents. Search covers
|
||||
the lines of a version's transcript that the main transcript lacks; results
|
||||
label those matches with the copy they came from. A version no publication names
|
||||
is listed as another copy.
|
||||
|
||||
The `versions` object records, per version directory, the SHA-256 of that copy's
|
||||
PDF and the pages that differ from the main document, by the number in the page
|
||||
image's filename:
|
||||
`"versions": {"<slug>": {"pdf_sha256": "...", "changed_pages": [7, 12]}}`.
|
||||
|
||||
`aka_id` lists other names the document was published under, as an array of
|
||||
objects: `{"id": "...", "doc_source": "...", "doc_url": "..."}`. An aka is a
|
||||
pointer and stores no file, which is what separates it from a `versions/` entry.
|
||||
|
||||
More info about the metadata: https://git.disroot.org/libroot/snowchive-meta
|
||||
|
||||
## Assemblies
|
||||
|
||||
`assemblies.json` is a hand-maintained list of sets of pages drawn from several
|
||||
documents and versions, for material that was published in pieces, such as a
|
||||
slide deck. Each gets a page at `/explore/assemblies/<id>` showing all its
|
||||
slides in one grid and lightbox, listed at `/explore/assemblies`
|
||||
|
||||
```json
|
||||
[{
|
||||
"id": "prism-overview",
|
||||
"title": "PRISM/US-984XN briefing",
|
||||
"note": "Shown at the top: how the set was put together and what is unproven.",
|
||||
"slides": [
|
||||
{"doc": "<document-id>", "pages": "1-7"},
|
||||
{"doc": "<document-id>", "version": "<slug>", "pages": "8", "note": "Shown under the slide."},
|
||||
{"doc": "<document-id>", "pages": "9,11",
|
||||
"alt": [{"doc": "<document-id>", "version": "<slug>", "pages": "1-2"}]}
|
||||
]
|
||||
}]
|
||||
```
|
||||
An assembly is the archive's own arrangement, it's not an original document and
|
||||
stores no files, and is neither counted nor searched.
|
||||
|
||||
## Tests
|
||||
|
||||
./tests/run.sh
|
||||
|
||||
TODO. The tests are currently written largely by Claude.
|
||||
|
||||
## Other/contact
|
||||
|
||||
The maintainers of this project do not endorse any public Snowchive instance
|
||||
other than the official snowchive.org instance (unless you are a reputable
|
||||
organization that wants to host a public instance).
|
||||
|
||||
Please submit issues/feedback on the Snowchive server's repository:
|
||||
|
||||
https://git.disroot.org/libroot/snowchive-server
|
||||
|
||||
or to Codeberg (mirror): https://codeberg.org/libroot/snowchive-server
|
||||
|
||||
Contact the maintainers via email:
|
||||
|
||||
https://libroot.org/about
|
||||
Loading…
Add table
Add a link
Reference in a new issue