first commit
This commit is contained in:
commit
8c4dcae475
89 changed files with 24623 additions and 0 deletions
544
templates/about.html
Normal file
544
templates/about.html
Normal file
|
|
@ -0,0 +1,544 @@
|
|||
{{ define "title" }}About - Snowden Archive{{ end }}
|
||||
{{ define "content" }}
|
||||
<div class="about-page">
|
||||
<h1>About this archive</h1>
|
||||
|
||||
<p>This is a searchable archive of all published documents from the 2013
|
||||
Snowden disclosures. The archive includes documents published alongside
|
||||
news reporting, The Intercept's bulk SIDtoday release, and FOIA material
|
||||
(official releases from FOIA litigation and FISC and ODNI
|
||||
declassifications).</p>
|
||||
<p>Every document carries its original PDF, page images, extracted text,
|
||||
and curated metadata - agencies, programs, countries, classification
|
||||
markings, people, and the journalists and articles that published it.</p>
|
||||
<p>Use <a href="/browse">Browse</a> to explore by category,
|
||||
<a href="/search">Search</a> for fielded boolean queries across text and
|
||||
metadata, <a href="/articles">Articles</a> to follow the disclosures story
|
||||
by story, the <a href="/glossary">Glossary</a> to decode codenames and
|
||||
acronyms, and <a href="/explore">Explore</a> to dig deeper into ranked
|
||||
views and lists.</p>
|
||||
|
||||
<nav class="toc" aria-label="Table of contents">
|
||||
<strong>Contents</strong>
|
||||
<ul>
|
||||
<li><a href="#privacy">Privacy</a></li>
|
||||
<li><a href="#whats-in">What's in this archive?</a></li>
|
||||
<li><a href="#search">How to search</a></li>
|
||||
{{ if .APIEnabled }}<li><a href="#api">Downloading data as JSON</a>
|
||||
<ul><li><a href="#metadata">Metadata structure</a></li></ul>
|
||||
</li>{{ else }}<li><a href="#metadata">Metadata structure</a></li>{{ end }}
|
||||
<li><a href="#download-archive">Downloading the archive</a>
|
||||
<ul>
|
||||
<li><a href="#downloading-all-data">Downloading all data</a></li>
|
||||
<li><a href="#downloading-texts">Downloading only extracted texts and metadata</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li><a href="#own-copy">Can I run my own copy or mirror this?</a></li>
|
||||
<li><a href="#contact">Contact</a></li>
|
||||
<li><a href="#contributing">Contributing</a></li>
|
||||
<li><a href="#thanks">Thank you</a></li>
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
<section id="privacy">
|
||||
<h2>Privacy</h2>
|
||||
<p>By default, the server keeps no state about its visitors. Pins and
|
||||
history live only in your own browser's cookies, and history features
|
||||
are off unless you opt in via <a href="/preferences">Preferences</a>.
|
||||
Anonymous page views set no cookies at all. All core features on this
|
||||
website work without JavaScript.</p>
|
||||
<p>Note that you can't verify that this server doesn't log and analyze
|
||||
your visits. Use Tor to hide your real IP address, or
|
||||
<a href="#own-copy">set up</a> your own instance on your machine to
|
||||
browse fully offline.</p>
|
||||
</section>
|
||||
|
||||
<section id="whats-in">
|
||||
<h2>What's in this archive?</h2>
|
||||
<p>By default, two collections are shown as tabs on the
|
||||
<a href="/browse">Browse</a> page: <strong>Reported</strong> documents
|
||||
(leaked files as the outlets that broke the stories presented them) and
|
||||
the <strong>SIDtoday Archive</strong> (The Intercept's bulk release of
|
||||
the NSA's internal SIDtoday newsletters). The third collection,
|
||||
<strong>FOIA</strong> (official releases from FOIA litigation and FISC
|
||||
and ODNI declassifications), can be included via
|
||||
<a href="/preferences">Preferences</a>.</p>
|
||||
<p>The metadata and many of the extracted texts are curated by hand and
|
||||
may contain errors or omissions, so treat them as a guide to the
|
||||
documents rather than as established fact, and check the original
|
||||
documents and reporting before relying on them.</p>
|
||||
</section>
|
||||
|
||||
<section id="search">
|
||||
<h2>How to search</h2>
|
||||
<p>The <a href="/search">Search</a> page supports fielded boolean
|
||||
queries. Restrict a term to a field (title, full text, programs,
|
||||
countries, journalists, and so on), combine clauses with AND/OR, quote
|
||||
<code>"text"</code> to find it anywhere, match exact whole words with
|
||||
<code>=...</code> (<code>publisher:="The Guardian"</code> matches the
|
||||
whole value), and use <code>*</code> wildcards (<code>tap*</code>,
|
||||
<code>*apping</code>).</p>
|
||||
|
||||
<p>Ways to write a value, in any row or after any operator:</p>
|
||||
<ul>
|
||||
<li><code>cable tapping</code> finds both words anywhere, in their other forms too (<em>tapping</em> also finds <em>tapped</em>).</li>
|
||||
<li><code>"apping"</code> finds that text anywhere, even inside a word, like <code>grep -i</code> (it finds, for example, <em>mapping</em> and <em>tapping</em>).</li>
|
||||
<li><code>=tap</code> or <code>="cable tapping"</code> finds exactly those whole words, so <code>=tap</code> skips <em>taps</em> and <em>strap</em>. On a metadata field it matches one whole value, so <code>agency:=NSA</code> finds documents whose agency is NSA but not <em>NSA, GCHQ</em>.</li>
|
||||
<li>The <code>*</code> wildcard finds part of a word: <code>tap*</code> finds words starting with <em>tap</em>, <code>*land</code> words ending in <em>land</em>, and <code>*keys*</code> words containing <em>keys</em>. A leading <code>*</code> needs at least three letters after it.</li>
|
||||
<li><code>"upstream collection"~10</code> finds the words within 10 words of each other (in any order, common stopwords ignored).</li>
|
||||
<li>A leading <code>-</code> excludes within its row: <code>xkeyscore -tempora</code> finds <em>xkeyscore</em> without <em>tempora</em>, and the row keeps that meaning under OR and NOT.</li>
|
||||
</ul>
|
||||
|
||||
<p>The simple search box (on the home page, on the 404 page, and via
|
||||
the URL parameter <code>?q=</code>) understands field operators:</p>
|
||||
<ul>
|
||||
<li><code>program:XKEYSCORE country:Germany cable</code> or <code>journalist:="Poitras, Laura"</code>.</li>
|
||||
<li>A leading <code>-</code> negates (<code>-tag:=junk</code>, <code>-draft</code>).</li>
|
||||
<li><code>date:2013</code>, <code>date:2013..2015</code> and <code>published:2014-06</code> filter by date.</li>
|
||||
<li>Operators: text, title, content, description, agency, journalist, publisher, classification, program, dissemination, banner, genre, series, country, people, tag, type.</li>
|
||||
<li>Bare words search all fields.</li>
|
||||
</ul>
|
||||
</section>
|
||||
|
||||
<section id="api">
|
||||
{{ if .APIEnabled }}
|
||||
<h2>Downloading data as JSON</h2>
|
||||
<p>API endpoints:</p>
|
||||
<div class="table-scroll">
|
||||
<table class="data-table about-table">
|
||||
<thead><tr><th>Endpoint</th><th>Returns</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><a href="/api/docs"><code>/api/docs</code></a></td><td>Metadata for every document in the archive, with pagination.</td></tr>
|
||||
<tr><td><code>/api/doc/{id}</code></td><td>A single document's metadata and extracted text, where <code>{id}</code> is the document's identifier.</td></tr>
|
||||
<tr><td><a href="/api/search?q=xkeyscore"><code>/api/search?q=xkeyscore</code></a></td><td>Search results for term xkeyscore. It accepts the same parameters as the Search page, so you can also pass <code>fqf</code>/<code>fqv</code>/<code>fqc</code> clause parameters for fielded queries.</td></tr>
|
||||
<tr><td><a href="/api/facets"><code>/api/facets</code></a></td><td>Counts for every browsable category (programs, agencies, countries, people, and so on).</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<p>Responses are plain JSON. For paginated responses, use the URL
|
||||
parameter <code>p</code> to navigate pages (e.g.
|
||||
<a href="/api/docs?p=2"><code>/api/docs?p=2</code></a>).</p>
|
||||
<p>See the <a href="#download-archive">Downloading the archive</a>
|
||||
section on how to download the whole archive.</p>
|
||||
<p><strong>Important if you are a crawler, robot or AI trying to
|
||||
download large amounts of data from this website:</strong> please use
|
||||
the downloadable datasets from the
|
||||
<a href="#download-archive">Downloading the archive</a> section. The
|
||||
information on this website is identical to the datasets, so you gain
|
||||
nothing by crawling this website.</p>
|
||||
{{ end }}
|
||||
|
||||
<details class="about-collapse">
|
||||
<summary>{{ if .APIEnabled }}<h3 id="metadata">Metadata structure</h3>{{ else }}<h2 id="metadata">Metadata structure</h2>{{ end }}</summary>
|
||||
<p>Every document ships a curated <code>metadata.json</code>:</p>
|
||||
|
||||
<h4>The document record</h4>
|
||||
<dl class="field-dict">
|
||||
<dt><code>id</code></dt>
|
||||
<dd>The document's stable identifier: the name of its directory
|
||||
and the last part of its URL (<code>/doc/{id}</code>). It is not
|
||||
written inside <code>metadata.json</code> - the directory name is
|
||||
the id.</dd>
|
||||
|
||||
<dt><code>aka_id</code></dt>
|
||||
<dd>Other names the same document was released under, most often
|
||||
the file name another publisher or a mirror gave its PDF. A list
|
||||
of entries, each described below.</dd>
|
||||
|
||||
<dt><code>aka_id[].id</code></dt>
|
||||
<dd>The other name itself. Searchable, so a document can be found
|
||||
by a file name it circulated under elsewhere.</dd>
|
||||
|
||||
<dt><code>aka_id[].doc_source</code></dt>
|
||||
<dd>The web page where the document was released under that name,
|
||||
linked as “page” next to the alias.</dd>
|
||||
|
||||
<dt><code>aka_id[].doc_url</code></dt>
|
||||
<dd>The direct link to the PDF released under that name, linked
|
||||
as “PDF” next to the alias.</dd>
|
||||
|
||||
<dt><code>title</code></dt>
|
||||
<dd>The document's title, written by the curators - usually the
|
||||
title the document gives itself or the one its publisher used.</dd>
|
||||
|
||||
<dt><code>description</code></dt>
|
||||
<dd>A short summary of what the document is and says, shown under
|
||||
the publication list and in search results and cards.
|
||||
Descriptions are the work of archivists without inside knowledge -
|
||||
corrections welcome.</dd>
|
||||
|
||||
<dt><code>long_description</code></dt>
|
||||
<dd>A longer account of the contents where one has been written,
|
||||
behind “Show more info” on the document page; often
|
||||
empty. A <code>description:</code> search looks in both
|
||||
descriptions.</dd>
|
||||
|
||||
<dt><code>tags</code></dt>
|
||||
<dd>Free-text subjects added by the curators. Unlike the other
|
||||
lists they are not necessarily taken from the document's own
|
||||
vocabulary, so they are the loosest thing to browse or search by.
|
||||
</dd>
|
||||
|
||||
<dt><code>type</code></dt>
|
||||
<dd>The kind of release: <code>REPORTED</code> (a leaked document
|
||||
as an outlet published it), <code>SIDTODAY</code> (an article of
|
||||
the NSA's internal SIDtoday newsletter from The Intercept's bulk
|
||||
release) or <code>FOIA</code> (obtained through a public-records
|
||||
request). When it is empty, a document filed in the SIDtoday or FOIA
|
||||
folder takes that type and any other is <code>REPORTED</code>.</dd>
|
||||
|
||||
<dt><code>collection</code></dt>
|
||||
<dd>Which of the three collections - the tabs on
|
||||
<a href="/browse">Browse</a> - the document belongs to:
|
||||
<code>news</code> (Reported), <code>sidtoday-archive</code> or
|
||||
<code>foia</code>. Documents filed under the SIDtoday and FOIA
|
||||
folders belong to those collections whatever this says; elsewhere
|
||||
an empty value means Reported, or FOIA for a document of type
|
||||
<code>FOIA</code>. FOIA releases are left out of
|
||||
browsing and search unless you include them in
|
||||
<a href="/preferences">Preferences</a>.</dd>
|
||||
|
||||
<dt><code>pages</code></dt>
|
||||
<dd>The page count of the PDF. When it is missing, the number of
|
||||
page images is used instead.</dd>
|
||||
|
||||
<dt><code>pdf_sha256</code></dt>
|
||||
<dd>The SHA-256 of the PDF as archived.</dd>
|
||||
|
||||
<dt><code>archive_comment</code></dt>
|
||||
<dd>A note from the curators about this archive's copy - how it
|
||||
was assembled, where it was found, what differs between releases,
|
||||
or a known problem with it.</dd>
|
||||
</dl>
|
||||
|
||||
<h4>The document itself: <code>document</code></h4>
|
||||
<dl class="field-dict">
|
||||
<dt><code>document.creator</code></dt>
|
||||
<dd>The agency that <em>wrote</em> the document, in its short form
|
||||
(<code>NSA</code>, <code>GCHQ</code>, <code>FISC</code>,
|
||||
<code>DOJ</code>, <code>CSE</code> ...) - not the agency it is
|
||||
about and not the one that published it. Empty when the author is
|
||||
not known.</dd>
|
||||
|
||||
<dt><code>document.date_created</code></dt>
|
||||
<dd>When the document was written, as precise as the document
|
||||
allows: <code>2013-03-20</code>, <code>2013-03</code>,
|
||||
<code>2013</code>, or empty when unknown. Nothing is invented to
|
||||
fill the gaps, and nothing is guessed from the publication date.
|
||||
</dd>
|
||||
|
||||
<dt><code>document.banner</code></dt>
|
||||
<dd>The marking line <strong>verbatim</strong> - what the
|
||||
document says about itself, qualifiers and all. Intranet pages
|
||||
often carry a ceiling banner (“DYNAMIC PAGE - HIGHEST
|
||||
POSSIBLE CLASSIFICATION IS ...”) while their content is
|
||||
portion-marked unclassified; the banner records the claim, not an
|
||||
assessment. Empty means the artifact has no marking line - nothing
|
||||
is reconstructed from portion marks.</dd>
|
||||
|
||||
<dt><code>document.classification</code></dt>
|
||||
<dd>The curator's <strong>assessment</strong> of the content's
|
||||
classification, as component values: the level
|
||||
(<code>TOP SECRET</code>, <code>SECRET</code>,
|
||||
<code>CONFIDENTIAL</code>, <code>UNCLASSIFIED</code>) and any
|
||||
control systems (<code>COMINT</code>, <code>SI</code>,
|
||||
<code>TK</code>, <code>STRAP 1</code> ...), judged from portion
|
||||
marks. If a document contains multiple classifications, highest is
|
||||
used here.</dd>
|
||||
|
||||
<dt><code>document.dissemination</code></dt>
|
||||
<dd>The curator's assessment of the handling controls, as
|
||||
component values (<code>FOUO</code>, <code>NOFORN</code>,
|
||||
<code>ORCON</code>, <code>REL TO ...</code>). Assessed the same
|
||||
way as the classification.</dd>
|
||||
|
||||
<dt><code>document.genre</code></dt>
|
||||
<dd>The document's form, from a fixed vocabulary:
|
||||
<code>slides</code>, <code>report</code>, <code>wiki</code>,
|
||||
<code>manual</code>, <code>guide</code>,
|
||||
<code>newsletter</code>, <code>other</code>.</dd>
|
||||
|
||||
<dt><code>document.series.name</code></dt>
|
||||
<dd>For a document that prints a series roster (currently only in
|
||||
SIDtoday): the series name verbatim, as the document prints it.
|
||||
Documents with the same name are listed together on the document
|
||||
page and under <a href="/browse/series">Series</a>.</dd>
|
||||
|
||||
<dt><code>document.series.part</code></dt>
|
||||
<dd>This document's 1-based position in the printed roster. Order
|
||||
comes from the artifact; parts of a series not yet in the archive
|
||||
show as gaps in the numbering, never renumbered away.</dd>
|
||||
|
||||
<dt><code>document.programs</code></dt>
|
||||
<dd>The codenames and programs the document names
|
||||
(<code>XKEYSCORE</code>, <code>PRISM</code>, ...), as curated
|
||||
values for browsing and search; the
|
||||
<a href="/glossary">Glossary</a> explains many of them.</dd>
|
||||
|
||||
<dt><code>document.countries</code></dt>
|
||||
<dd>The countries the document mentions, by their English names.
|
||||
A mention, not a claim that the country was a target or a partner.
|
||||
</dd>
|
||||
|
||||
<dt><code>document.people</code></dt>
|
||||
<dd>The people who appear in the document: named in its text, or
|
||||
recognised in its images - for example
|
||||
<a href="/doc/gchq-mobile-networks-in-my-noc-world">Mobile Networks
|
||||
in My NOC World</a> lists King Charles and Queen Camilla,
|
||||
who appear only in a photograph on one of its slides. Usually
|
||||
written “Surname, Given name”. A person appearing in
|
||||
these documents implies nothing about them beyond that appearance.
|
||||
The journalists who published the document are under
|
||||
<code>publication[].reporters</code>, not here.</dd>
|
||||
</dl>
|
||||
|
||||
<h4>Where it was published: <code>publication[]</code></h4>
|
||||
<p>One entry per story or book that published (or reported) the
|
||||
document. Browse-by-publication-date counts a document under its
|
||||
<em>oldest</em> dated entry; a publication-date search matches
|
||||
<em>any</em> entry's date - or, combined with a journalist or
|
||||
publisher, that entry's date.</p>
|
||||
<dl class="field-dict">
|
||||
<dt><code>publication[].publisher</code></dt>
|
||||
<dd>Who published it: a news outlet, a book's publisher, or for a
|
||||
FOIA release the organization that obtained and released it.
|
||||
Browsable under <a href="/browse/publisher">Publishers</a> - except
|
||||
entries marked <code>reporting_only</code>.</dd>
|
||||
|
||||
<dt><code>publication[].title</code></dt>
|
||||
<dd>The headline of the article, or the book's title. Shown on the
|
||||
document page and names the story on
|
||||
<a href="/articles">Articles</a>.</dd>
|
||||
|
||||
<dt><code>publication[].link</code></dt>
|
||||
<dd>The URL of the article. It is the story's identity: documents
|
||||
whose entries share a link were published in the same report and
|
||||
are listed together on its article page.</dd>
|
||||
|
||||
<dt><code>publication[].date_published</code></dt>
|
||||
<dd>When this publication appeared, in the same
|
||||
<code>YYYY-MM-DD</code> / <code>YYYY-MM</code> /
|
||||
<code>YYYY</code> forms as the document date.</dd>
|
||||
|
||||
<dt><code>publication[].reporters</code></dt>
|
||||
<dd>The journalists bylined on the story, written “Surname,
|
||||
Given name”. Browsable under
|
||||
<a href="/browse/journalist">Journalists</a> and searchable with
|
||||
<code>journalist:</code>.</dd>
|
||||
|
||||
<dt><code>publication[].contributors</code></dt>
|
||||
<dd>People credited on the story without a byline - additional
|
||||
reporting, research, technical analysis. Shown on the document
|
||||
page and found by an all-fields search, but not listed among the
|
||||
journalists.</dd>
|
||||
|
||||
<dt><code>publication[].note</code></dt>
|
||||
<dd>Anything particular to this publication: that it showed only
|
||||
some of the pages, published them in a different order, redacted
|
||||
differently, or printed them in a book appendix.</dd>
|
||||
|
||||
<dt><code>publication[].version</code></dt>
|
||||
<dd>When this publication released a different copy from the one
|
||||
shown on the document page, the name of that copy in the
|
||||
document's <code>versions/</code> folder; its entry then links to
|
||||
“view this copy”. Without it, the entry published the
|
||||
copy shown on the page.</dd>
|
||||
|
||||
<dt><code>publication[].doc_source</code></dt>
|
||||
<dd>The web page where this publication presented the document
|
||||
(the outlet's own document page, a DocumentCloud page), linked as
|
||||
“document page”. Where a copy came from is recorded
|
||||
here, on the publication that released it, not on the
|
||||
document.</dd>
|
||||
|
||||
<dt><code>publication[].doc_url</code></dt>
|
||||
<dd>The direct link to the PDF file this publication released,
|
||||
linked as “original PDF”.</dd>
|
||||
|
||||
<dt><code>publication[].reporting_only</code></dt>
|
||||
<dd><code>true</code> when the outlet reported the story without
|
||||
publishing the document - a reporting partner. The entry keeps its
|
||||
byline and its article, but the outlet is not counted as one of
|
||||
the document's publishers and no copy is linked from it.</dd>
|
||||
</dl>
|
||||
|
||||
<h4>Other copies: <code>versions</code></h4>
|
||||
<p>Other published copies of a document (e.g. less redacted, with extra
|
||||
pages, or a different scan) are kept in its <code>versions/</code>
|
||||
folder, one sub-folder per copy, each with its own PDF, text and page
|
||||
images. The <code>versions</code> object adds details about them,
|
||||
keyed by the copy's folder name.</p>
|
||||
<dl class="field-dict">
|
||||
<dt><code>versions.{name}.changed_pages</code></dt>
|
||||
<dd>The pages of that copy, by its own page numbers, that differ
|
||||
from the copy shown on the document page. They are linked as
|
||||
“changed pages” next to the copy.</dd>
|
||||
|
||||
<dt><code>versions.{name}.pdf_sha256</code></dt>
|
||||
<dd>The SHA-256 of that copy's PDF.</dd>
|
||||
</dl>
|
||||
</details>
|
||||
</section>
|
||||
|
||||
<section id="download-archive">
|
||||
<h2>Downloading the archive</h2>
|
||||
<p>Each release comes in two packages:</p>
|
||||
<ul>
|
||||
<li><strong>full</strong>, ~18 GB: all the PDFs, page images (the
|
||||
PDF pages as JPEG files), extracted texts, and metadata,</li>
|
||||
<li><strong>pdfs</strong>, ~3.5 GB: the same without the page
|
||||
images.</li>
|
||||
</ul>
|
||||
<p>Both extract to <code>data/</code> and <code>project/</code>, ready
|
||||
to <a href="#own-copy">run your own copy</a>. Each release also
|
||||
includes the source of the server version it was published with, and
|
||||
the last six releases stay online. For only the extracted texts and
|
||||
metadata (~58 MB), see <a href="#downloading-texts">below</a>.</p>
|
||||
|
||||
<h3 id="downloading-all-data">Downloading all data</h3>
|
||||
|
||||
<h4>HTTP</h4>
|
||||
<p><a href="https://data.snowchive.org/files/">https://data.snowchive.org/files/</a></p>
|
||||
<details>
|
||||
<summary>See a curl example</summary>
|
||||
<pre>
|
||||
# download the latest release:
|
||||
rel=$(curl -fsS https://data.snowchive.org/files/latest.txt)
|
||||
base="https://data.snowchive.org/files/$rel"
|
||||
curl -f -C - -O "$base/snowchive-$rel-full.tar" # "pdfs" instead of "full" for no page images
|
||||
|
||||
# verify (optional):
|
||||
curl -fsS -O "$base/SHA256SUMS" -O "$base/SHA256SUMS.gpg"
|
||||
curl -fsS -O "https://libroot.org/public/libroot.asc"
|
||||
# "gpg --show-keys libroot.asc" should show 3B054BC2E7B2CC825FF23E94108F97B8FC5B2EBC
|
||||
gpg --import libroot.asc
|
||||
gpg --verify SHA256SUMS.gpg SHA256SUMS
|
||||
sha256sum --check --ignore-missing SHA256SUMS
|
||||
|
||||
# extract to ./snowchive-data/:
|
||||
mkdir -p snowchive-data && tar -xf "snowchive-$rel-full.tar" -C snowchive-data
|
||||
</pre>
|
||||
</details>
|
||||
|
||||
<h4>Torrent</h4>
|
||||
<p>Each release folder on
|
||||
<a href="https://data.snowchive.org/files/">https://data.snowchive.org/files/</a>
|
||||
has a <code>.torrent</code> for both packages.</p>
|
||||
|
||||
<h3 id="downloading-texts">Downloading only extracted texts and metadata</h3>
|
||||
<p>Repository for the extracted texts and metadata:
|
||||
<a href="https://git.disroot.org/libroot/snowchive-meta">https://git.disroot.org/libroot/snowchive-meta</a></p>
|
||||
<pre>
|
||||
git clone https://git.disroot.org/libroot/snowchive-meta
|
||||
cd snowchive-meta/data
|
||||
|
||||
# grep only the extracted texts:
|
||||
grep "fiber optic" -ri --include=*.md
|
||||
|
||||
# grep only the metadata:
|
||||
grep "fiber optic" -ri --include=*.json
|
||||
</pre>
|
||||
</section>
|
||||
|
||||
<section id="own-copy">
|
||||
<h2>Can I run my own copy or mirror this?</h2>
|
||||
<p>Yes. The server software behind this archive is free software with
|
||||
no database and no external dependencies, designed to run offline or,
|
||||
for example, as a Tor hidden service.</p>
|
||||
<p>First, get a <a href="#downloading-all-data">release package</a>
|
||||
and extract it: <strong>full</strong> for everything, or
|
||||
<strong>pdfs</strong> for a site that works the same but shows no page
|
||||
images.</p>
|
||||
<p>Then get the server (Go required). Each release folder has the
|
||||
source of the server version it was published with,
|
||||
<code>snowchive-server-<commit>.tar.gz</code>, covered by the same
|
||||
<code>SHA256SUMS</code> - the version known to work with that data:</p>
|
||||
<pre>
|
||||
tar -xzf snowchive-server-<commit>.tar.gz
|
||||
cd snowchive-server-<commit>
|
||||
go run . -root=/path/to/snowchive-data/ -listen=127.0.0.1:8081
|
||||
</pre>
|
||||
<p>Or clone the latest version from the
|
||||
<a href="https://git.disroot.org/libroot/snowchive-server">snowchive-server</a>
|
||||
repository:</p>
|
||||
<pre>
|
||||
git clone https://git.disroot.org/libroot/snowchive-server
|
||||
cd snowchive-server
|
||||
go run . -root=/path/to/snowchive-data/ -listen=127.0.0.1:8081
|
||||
</pre>
|
||||
<p>Recommended minimal setup for production instances:</p>
|
||||
<pre>
|
||||
go build -ldflags "-X main.buildCommit=$(git rev-parse --short HEAD)" -o snowchive .
|
||||
./snowchive -root=/path/to/snowchive-data/ -listen=127.0.0.1:8081 -base=https://archive.example.com
|
||||
</pre>
|
||||
<p>System requirements: at least 300 MB of RAM (1 GB or more
|
||||
recommended for production).</p>
|
||||
<p>The maintainers of this project do not endorse any public Snowchive
|
||||
instance other than the official snowchive.org instance (unless you
|
||||
are a reputable organization that wants to host a public
|
||||
instance).</p>
|
||||
<p>See the
|
||||
<a href="https://git.disroot.org/libroot/snowchive-server">snowchive-server README</a>
|
||||
for the full documentation.</p>
|
||||
</section>
|
||||
|
||||
<section id="contact">
|
||||
<h2>Contact</h2>
|
||||
<p>If the archive is missing a document, or some metadata is wrong,
|
||||
please submit an issue on the project repository:
|
||||
<a href="https://git.disroot.org/libroot/snowchive-meta/issues">https://git.disroot.org/libroot/snowchive-meta/issues</a></p>
|
||||
<p>You can also report issues and errors or give feedback directly to
|
||||
the project maintainers via email:
|
||||
<a href="https://libroot.org/about">https://libroot.org/about</a></p>
|
||||
<p>Please submit server-related issues to the Snowchive server
|
||||
repository:
|
||||
<a href="https://git.disroot.org/libroot/snowchive-server/issues">https://git.disroot.org/libroot/snowchive-server/issues</a></p>
|
||||
</section>
|
||||
|
||||
<section id="contributing">
|
||||
<h2>Contributing</h2>
|
||||
<p>We welcome anyone who wants to contribute to the project, for
|
||||
example by fixing errors in the metadata or in the extracted texts.
|
||||
Please see the <a href="#contact">Contact</a> section above.</p>
|
||||
<p><small>Running this project doesn't cost much, but if you wish, you
|
||||
can send a donation to:</small></p>
|
||||
<dl class="field-dict">
|
||||
<dt><small>XMR</small></dt>
|
||||
<dd><small><code style="word-break:break-all;">85gTBgYuLB4Mvo1ZCGo62126hBQQ678UeESTuSi3L9SECHXr6S7mXE28yAKvURgdJneGKqbPDcBydFfTkK7H5bvh2WXE4oT</code></small></dd>
|
||||
<dt><small>BTC</small></dt>
|
||||
<dd><small><code style="word-break:break-all;">bc1q5f5mc4pytajrn2xrp3lsftx64fw6hsdxxvs3ha</code></small></dd>
|
||||
</dl>
|
||||
</section>
|
||||
|
||||
<section id="thanks">
|
||||
<h2>Thank you</h2>
|
||||
<p>We want to thank</p>
|
||||
<ul>
|
||||
<li>all the journalists, reporters, and other people who broke the
|
||||
stories or helped bring them to light,</li>
|
||||
<li>other archives and the people behind them, such as
|
||||
<ul>
|
||||
<li>University of Toronto's and Canadian Journalists for
|
||||
Free Expression's The Snowden Surveillance Archive (shut
|
||||
down in mid-2024),</li>
|
||||
<li>Courage Foundation's Snowden Revelations (shut down in
|
||||
early 2022),</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>organizations, resources and blogs such as
|
||||
<ul>
|
||||
<li><a href="https://www.eff.org/nsa-spying/nsadocs">EFF's NSA document collection</a>,</li>
|
||||
<li><a href="https://www.aclu.org/nsa-documents-released-to-the-public-since-june-2013">ACLU's NSA document collection</a>,</li>
|
||||
<li><a href="https://christopher-parsons.com/resources/the-sigint-summaries/">Christopher Parsons' SIGINT Summaries</a>,</li>
|
||||
<li><a href="https://electrospaces.net">Electrospaces</a>,</li>
|
||||
<li><a href="https://github.com/iamcryptoki/snowden-archive">iamcryptoki/snowden-archive</a>,</li>
|
||||
<li><a href="https://cryptome.org">Cryptome</a>,</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>and of course, Edward Snowden himself.</li>
|
||||
</ul>
|
||||
</section>
|
||||
|
||||
</div>
|
||||
{{ end }}
|
||||
{{ template "base.html" . }}
|
||||
Loading…
Add table
Add a link
Reference in a new issue