first commit

This commit is contained in:
libroot 2026-11-19 04:40:59 +02:00
commit 8c4dcae475
Signed by: libroot
GPG key ID: 2AC68C029EBE9ACE
89 changed files with 24623 additions and 0 deletions

661
LICENSE Normal file
View file

@ -0,0 +1,661 @@
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU Affero General Public License is a free, copyleft license for
software and other kinds of works, specifically designed to ensure
cooperation with the community in the case of network server software.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
our General Public Licenses are intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
Developers that use our General Public Licenses protect your rights
with two steps: (1) assert copyright on the software, and (2) offer
you this License which gives you legal permission to copy, distribute
and/or modify the software.
A secondary benefit of defending all users' freedom is that
improvements made in alternate versions of the program, if they
receive widespread use, become available for other developers to
incorporate. Many developers of free software are heartened and
encouraged by the resulting cooperation. However, in the case of
software used on network servers, this result may fail to come about.
The GNU General Public License permits making a modified version and
letting the public access it on a server without ever releasing its
source code to the public.
The GNU Affero General Public License is designed specifically to
ensure that, in such cases, the modified source code becomes available
to the community. It requires the operator of a network server to
provide the source code of the modified version running there to the
users of that server. Therefore, public use of a modified version, on
a publicly accessible server, gives the public access to the source
code of the modified version.
An older license, called the Affero General Public License and
published by Affero, was designed to accomplish similar goals. This is
a different license, not a version of the Affero GPL, but Affero has
released a new version of the Affero GPL which permits relicensing under
this license.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU Affero General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Remote Network Interaction; Use with the GNU General Public License.
Notwithstanding any other provision of this License, if you modify the
Program, your modified version must prominently offer all users
interacting with it remotely through a computer network (if your version
supports such interaction) an opportunity to receive the Corresponding
Source of your version by providing access to the Corresponding Source
from a network server at no charge, through some standard or customary
means of facilitating copying of software. This Corresponding Source
shall include the Corresponding Source for any work covered by version 3
of the GNU General Public License that is incorporated pursuant to the
following paragraph.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the work with which it is combined will remain governed by version
3 of the GNU General Public License.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU Affero General Public License from time to time. Such new versions
will be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU Affero General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU Affero General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU Affero General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If your software can interact with users remotely through a computer
network, you should also make sure that it provides a way for users to
get its source. For example, if your program is a web application, its
interface could display a "Source" link that leads users to an archive
of the code. There are many ways you could offer source, and different
solutions will be better for different programs; see section 13 for the
specific requirements.
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU AGPL, see
<https://www.gnu.org/licenses/>.

177
README.md Normal file
View file

@ -0,0 +1,177 @@
# Snowchive server
A self-contained web server for browsing and searching an archive of published
Snowden documents in Snowchive website. The server includes faceted browsing, a
boolean/fielded search engine with stemming and accent folding, a SIDtoday
serial reader, a read-only JSON API, sitemaps, and per-visitor preferences, pins
and history that live only in the visitor's cookies. The server uses just Go's
standard library.
No JavaScript is required to use the website. A file in `static/main.js` adds
progressive enhancements only. No database and no external services.
See it in action: https://snowchive.org
If you are looking for Snowchive data or metadata:
Data (PDFs, page images):
https://snowchive.org/about#downloading-all-data
Metadata (with extracted texts):
https://git.disroot.org/libroot/snowchive-meta
## Run
Full Snowchive data is required (containing all the PDFs and metadata) to run
the server. If you don't yet have the data, download it first:
https://snowchive.org/about#downloading-all-data
```sh
go run . -root=/path/to/snowchive-data/
```
## Building
If you want to build a binary to deploy one executable for production use:
```sh
go build -ldflags "-X main.buildCommit=$(git rev-parse --short HEAD)" -o snowchive .
./snowchive -root=/path/to/snowchive-data/ -base=https://snowchive.example.org
```
## Flags
| flag | purpose |
|---|---|
| `-root` | Snowchive data directory holding `data/` (the corpus, one directory per document) and `project/` (`glossary.json`, `assemblies.json`, `changelog.json`, `related.json`) (default: the working directory, `$SNOWCHIVE_ROOT`) |
| `-listen` | TCP address (default `127.0.0.1:8080`, `$SNOWCHIVE_LISTEN`). |
| `-base` | canonical `scheme://host` for absolute URLs in robots.txt and the sitemap; set it on anything internet-facing. Without it those URLs follow each request's host. |
| `-no-api` | disable the JSON API (`$SNOWCHIVE_NO_API=1`) |
| `-accel-files PREFIX` | behind nginx only: answer document files with an `X-Accel-Redirect` into this internal location (e.g. `/_files/`, an `alias` of the data directory), so nginx sends the bytes and this server only decides whether the file is served (`$SNOWCHIVE_ACCEL_FILES`) |
| `-accel-expires DURATION` | let the proxy cache a page served to a reader without cookies for this long (e.g. `10m`, sent as `X-Accel-Expires`); browsers keep the 10-second `max-age`. Clear the proxy cache after a restart with new data (`$SNOWCHIVE_ACCEL_EXPIRES`) |
| `-dev` | serve `templates/` and `static/` from the working directory, re-parse per request, verbose errors. Never in production. |
| `-check` | load the corpus, print the warnings and the tally, then exit non-zero if anything was reported - run it after a curation pass or in CI. |
| `-rebuild-related` | compute `related.json` entries for documents that lack one, or whose list links to a document no longer in the corpus, drop the entries of documents that are gone, then exit. It only fills gaps, so after a scoring change delete `related.json` first to rebuild every list. |
## What it costs to run
Measured on a 2-core VM against the 3,106-document corpus: ~7 seconds to load
(warm page cache), ~260 MB of live heap, and ~750 MB of address space under
sustained concurrent load.
## Directory layout
```
data/
<document-id>/ one directory per document
metadata.json document metadata
<anything>.pdf document PDF file
<anything>.md extracted text transcript from PDF
thumb.jpg, thumb.gif document thumbnail images
images/page-N.jpg PDF page scans
versions/<slug>/ another published copy that differs (e.g. less
redacted, extra pages): its own PDF,
.md and images/ (optional)
FOIA/<document-id>/ the FOIA document collection
SIDtoday-archive/<id>/ the bulk SIDtoday document collection
```
The files about the archive as a whole are in the project directory
(`-project`):
```
project/
glossary.json format documented in glossary.go
assemblies.json see "Assemblies" below
changelog.json snowchive changelog, format in changelog.go
related.json documents relating to each other, generated by
-rebuild-related
```
The corpus is read exactly once at startup and everything derived from it is
computed there. All of it lives in one `Corpus` value (`corpus.go`), which is
built whole and pre-rendered. The server writes nothing while it runs. Only
`-rebuild-related` writes a file (`related.json`, in the project directory).
## About metadata
`metadata.json` carries the title, description, tags, the `document`
block (creator, date_created, programs, classification, dissemination,
banner, genre, series, countries, people) and the `publication` list
(publisher, title, link, reporters, contributors, date_published, plus
`doc_source` and `doc_url` for the copy that entry published, and `note`
and `version` when that copy differs from the primary one). Every field
except `title` is optional. There is no document-level `doc_source`/`doc_url`,
because where a copy came from is a property of the publication that published
it.
A publication entry is an outlet that published the document. A partner in a
joint investigation that published the story but no documents is the one
exception (they have entry `"reporting_only": true`).
A publication that released a different copy of the same document is in
`version` directory. The directory `versions/<slug>/` is in the document
directory, holding that copy's PDF, `.md` and `images/`. The document page shows
the publication's `note` and links the copy, which has its own page at
`/doc/<id>/versions/<slug>`. Versions never count as documents. Search covers
the lines of a version's transcript that the main transcript lacks; results
label those matches with the copy they came from. A version no publication names
is listed as another copy.
The `versions` object records, per version directory, the SHA-256 of that copy's
PDF and the pages that differ from the main document, by the number in the page
image's filename:
`"versions": {"<slug>": {"pdf_sha256": "...", "changed_pages": [7, 12]}}`.
`aka_id` lists other names the document was published under, as an array of
objects: `{"id": "...", "doc_source": "...", "doc_url": "..."}`. An aka is a
pointer and stores no file, which is what separates it from a `versions/` entry.
More info about the metadata: https://git.disroot.org/libroot/snowchive-meta
## Assemblies
`assemblies.json` is a hand-maintained list of sets of pages drawn from several
documents and versions, for material that was published in pieces, such as a
slide deck. Each gets a page at `/explore/assemblies/<id>` showing all its
slides in one grid and lightbox, listed at `/explore/assemblies`
```json
[{
"id": "prism-overview",
"title": "PRISM/US-984XN briefing",
"note": "Shown at the top: how the set was put together and what is unproven.",
"slides": [
{"doc": "<document-id>", "pages": "1-7"},
{"doc": "<document-id>", "version": "<slug>", "pages": "8", "note": "Shown under the slide."},
{"doc": "<document-id>", "pages": "9,11",
"alt": [{"doc": "<document-id>", "version": "<slug>", "pages": "1-2"}]}
]
}]
```
An assembly is the archive's own arrangement, it's not an original document and
stores no files, and is neither counted nor searched.
## Tests
./tests/run.sh
TODO. The tests are currently written largely by Claude.
## Other/contact
The maintainers of this project do not endorse any public Snowchive instance
other than the official snowchive.org instance (unless you are a reputable
organization that wants to host a public instance).
Please submit issues/feedback on the Snowchive server's repository:
https://git.disroot.org/libroot/snowchive-server
or to Codeberg (mirror): https://codeberg.org/libroot/snowchive-server
Contact the maintainers via email:
https://libroot.org/about

60
agencies.go Normal file
View file

@ -0,0 +1,60 @@
// agencies.go - display names for the acronyms used as document creators
package main
import "strings"
var agencyNames = map[string]string{
"CSE": "Communications Security Establishment (Canada)",
"DFS": "Directorate for Signals Intelligence (Japan)",
"DNI": "Director of National Intelligence (US)",
"DOD": "Department of Defense (US)",
"DOJ": "Department of Justice (US)",
"DOS": "Department of State (US)",
"DSD": "Defence Signals Directorate (Australia; now ASD)",
"FISC": "Foreign Intelligence Surveillance Court (US)",
"FVEY": "Five Eyes (US, UK, Canada, Australia, New Zealand)",
"GCHQ": "Government Communications Headquarters (UK)",
"GCSB": "Government Communications Security Bureau (New Zealand)",
"MI5": "Security Service (UK)",
"NATO": "North Atlantic Treaty Organization",
"NCTC": "National Counterterrorism Center (US)",
"NSA": "National Security Agency (US)",
"ODNI": "Office of the Director of National Intelligence (US)",
"SIS": "Secret Intelligence Service (UK)",
"SWAC": "Southwest Analysis Center",
"USG": "United States Government",
}
func agencyName(creator string) string {
var out strings.Builder
known := false
start := 0
flush := func(end int) {
part := creator[start:end]
key := strings.ToUpper(strings.TrimSpace(part))
if name, ok := agencyNames[key]; ok {
known = true
out.WriteString(name)
} else {
out.WriteString(strings.TrimSpace(part))
}
}
for i := 0; i < len(creator); i++ {
switch creator[i] {
case '/':
flush(i)
out.WriteString(" / ")
start = i + 1
case ',':
flush(i)
out.WriteString(", ")
start = i + 1
}
}
flush(len(creator))
if !known {
return ""
}
return out.String()
}

335
api.go Normal file
View file

@ -0,0 +1,335 @@
// api.go - read-only JSON API mirroring the UI's features.
//
// Endpoints (all GET):
//
// /api/docs list documents; params: coll (repeatable:
// news, sidtoday-archive, foia), type
// (repeatable), sort (any list-page sort: title,
// agency, date, pubdate, classification, ...; see
// docSorts), order (asc|desc), p, per_page (max
// 500, or "all" for exports)
// /api/doc/{id} full document: metadata, full text, images,
// related IDs, published-together IDs,
// version_copies (other published copies: PDF,
// page images, transcript, publication indices)
// /api/search params: q (simple) OR fqv/fqf/fqc clause
// triples (same field codes and value syntax as
// the UI: word, "text", =exact, *wildcard*),
// plus coll and type (both repeatable, as on
// /api/docs; no coll means every collection),
// date_doc_start/_end, date_pub_start/_end
// (partial END dates are inclusive: 2013 means
// through 2013-12-31), sort, order, and the same
// p / per_page pagination as /api/docs (default
// 50, max 500, "all" for every match without
// snippets see apiPageSize). results come
// as one flat list, mirroring the search page:
// all collections rank equally. a result whose match
// is in another published copy of the document
// lists that copy in matched_versions (slug,
// label, url, snippet)
// /api/facets facet maps (whole corpus, like the UI; the
// one news-scoped map is news_types) plus
// collection counts
package main
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"log"
"net/http"
"net/url"
"strconv"
"strings"
"time"
)
const (
apiPerPageDefault = 50
apiMaxPerPage = 500
)
func writeJSON(w http.ResponseWriter, status int, v interface{}) {
b, err := json.Marshal(v)
if err != nil {
log.Printf("API: marshal error: %v", err)
w.Header().Set("Cache-Control", "no-store")
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.WriteHeader(http.StatusInternalServerError)
w.Write([]byte(`{"error":"internal error"}`))
return
}
writeJSONBytes(w, status, b)
}
func writeJSONBytes(w http.ResponseWriter, status int, b []byte) {
setJSONHeaders(w)
w.WriteHeader(status)
w.Write(b)
}
func setJSONHeaders(w http.ResponseWriter) {
cc := w.Header().Get("Cache-Control")
if cc == "" {
cc = "public, max-age=60"
w.Header().Set("Cache-Control", cc)
}
w.Header().Set("Content-Type", "application/json; charset=utf-8")
if !strings.Contains(cc, "private") {
w.Header().Set("Access-Control-Allow-Origin", "*")
}
}
func apiError(w http.ResponseWriter, status int, msg string) {
writeJSON(w, status, map[string]string{"error": msg})
}
type apiDocSummary struct {
Document
URL string `json:"url"`
PDF string `json:"pdf"`
}
func docSummary(d Document) apiDocSummary {
return apiDocSummary{
Document: d,
URL: docPath(d.ID),
PDF: filePath(d.ID, d.PDFFile),
}
}
type apiDocDetail struct {
apiDocSummary
FullText string `json:"full_text,omitempty"`
Images []string `json:"images,omitempty"`
Related []string `json:"related,omitempty"`
PublishedTogether []string `json:"published_together,omitempty"`
VersionCopies []apiVersion `json:"version_copies,omitempty"`
}
type apiSearchResult struct {
apiDocSummary
Score float64 `json:"score"`
Snippet string `json:"snippet,omitempty"`
MatchedVersions []apiMatchedVersion `json:"matched_versions,omitempty"`
}
func (co *Corpus) handleAPIDocs(w http.ResponseWriter, r *http.Request) {
q := r.URL.Query()
typeFilter, _ := selectedTypes(q)
// coll=, repeatable, as /api/search and the pages take it.
pool := filterByCollection(co.everyDoc, collectionFilter(q))
docs := filterByType(append([]*Document(nil), pool...), typeFilter)
sortDocs(docs, q.Get("sort"), q.Get("order"))
perPage := apiPageSize(q)
page, _ := strconv.Atoi(q.Get("p"))
start, end, page, totalPages := paginate(len(docs), page, perPage)
out := make([]apiDocSummary, 0, end-start)
for _, d := range docs[start:end] {
out = append(out, docSummary(*d))
}
writeJSON(w, http.StatusOK, map[string]interface{}{
"total": len(docs),
"page": page,
"per_page": apiRespPerPage(perPage, len(docs)),
"total_pages": totalPages,
"docs": out,
})
}
func apiPageSize(q url.Values) int {
if q.Get("per_page") == "all" {
return 0
}
if v, err := strconv.Atoi(q.Get("per_page")); err == nil && v > 0 {
if v > apiMaxPerPage {
return apiMaxPerPage
}
return v
}
return apiPerPageDefault
}
func apiRespPerPage(perPage, total int) int {
if perPage > 0 {
return perPage
}
if total == 0 {
return 1
}
return total
}
func (co *Corpus) handleAPIDoc(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id")
doc := co.docIndex[id]
if doc == nil {
apiError(w, http.StatusNotFound, "document not found")
return
}
var pubIDs []string
for _, d := range co.publishedTogether(doc) {
pubIDs = append(pubIDs, d.ID)
}
writeJSON(w, http.StatusOK, apiDocDetail{
apiDocSummary: docSummary(*doc),
FullText: doc.FullText,
Images: doc.Images,
Related: co.archive.RelatedDocs[id],
PublishedTogether: pubIDs,
VersionCopies: apiVersions(doc),
})
}
func (co *Corpus) handleAPISearch(w http.ResponseWriter, r *http.Request) {
q := r.URL.Query()
sp := readSearchParams(q)
if len(sp.Queries) == 0 {
apiError(w, http.StatusBadRequest,
"missing query: pass ?q=term or fqv/fqf/fqc clause parameters")
return
}
if sp.OverClauses > 0 {
apiError(w, http.StatusBadRequest, fmt.Sprintf(
"too many clauses: %d (the limit is %d)", sp.OverClauses, maxSearchClauses))
return
}
if sp.OverWords > 0 {
apiError(w, http.StatusBadRequest, fmt.Sprintf(
"too many words: %d (the limit is %d)", sp.OverWords, maxSearchWords))
return
}
scored, _ := co.executeSearch(sp)
snippetTerm, _ := snippetAnchor(sp.Queries, sp.Fields, sp.Conjs)
perPage := apiPageSize(q)
page, _ := strconv.Atoi(q.Get("p"))
start, end, page, totalPages := paginate(len(scored), page, perPage)
withSnippets := snippetTerm != "" && perPage > 0
results := make([]apiSearchResult, 0, end-start)
for _, s := range scored[start:end] {
res := apiSearchResult{apiDocSummary: docSummary(*s.Doc), Score: s.Score}
if withSnippets {
if body := s.Doc.bodySnippets(snippetTerm, 1); len(body) > 0 {
res.Snippet = body[0]
}
for _, h := range versionHits(s.Doc, snippetTerm, 1) {
res.MatchedVersions = append(res.MatchedVersions, apiMatchedVersion{
Slug: h.Version.Slug,
Label: h.Version.Label,
URL: versionPath(s.Doc.ID, h.Version.Slug),
Snippet: h.Snippets[0],
})
}
}
results = append(results, res)
}
body := map[string]interface{}{
"total": len(scored),
"page": page,
"per_page": apiRespPerPage(perPage, len(scored)),
"total_pages": totalPages,
"results": results,
}
if snippetTerm != "" && !withSnippets {
body["snippets_omitted"] = true
}
writeJSON(w, http.StatusOK, body)
}
func (co *Corpus) buildFacetsJSON() {
b, err := json.Marshal(map[string]interface{}{
"note": "facet counts cover the whole corpus; news_types covers the news collection only",
"collections": co.archive.CollectionCounts,
"sidtoday_total": co.archive.SIDTodayCount,
"types": co.archive.Types,
"news_types": co.archive.NewsTypes,
"agencies": co.archive.Facets["agency"],
"programs": co.archive.Facets["program"],
"journalists": co.archive.Facets["journalist"],
"publishers": co.archive.Facets["publisher"],
"countries": co.archive.Facets["country"],
"people": co.archive.Facets["people"],
"classification": co.archive.Facets["classification"],
"dissemination": co.archive.Facets["dissemination"],
"banners": co.archive.Facets["banner"],
"genres": co.archive.Facets["genre"],
"series": co.archive.Facets["series"],
"tags": co.archive.Facets["tag"],
"pub_years": co.archive.Facets["published"],
"doc_years": co.archive.Facets["docdate"],
"by_collection": co.archive.FacetsByColl,
})
if err != nil {
log.Fatalf("facets: marshal failed: %v", err)
}
co.facetsJSON = b
sum := sha256.Sum256(b)
co.facetsETag = `"` + hex.EncodeToString(sum[:8]) + `"`
log.Printf("Facets JSON pre-built: %d bytes", len(co.facetsJSON))
}
func (co *Corpus) handleAPIFacets(w http.ResponseWriter, r *http.Request) {
setJSONHeaders(w)
w.Header().Set("ETag", co.facetsETag)
http.ServeContent(w, r, "", time.Time{}, bytes.NewReader(co.facetsJSON))
}
// "version" as in document versions (metadata.json can have "versions" object)
type apiVersion struct {
Slug string `json:"slug"`
URL string `json:"url"`
PDF string `json:"pdf"`
PDFSha256 string `json:"pdf_sha256,omitempty"`
Images []string `json:"images,omitempty"`
FullText string `json:"full_text,omitempty"`
Publications []int `json:"publications,omitempty"`
ChangedPages []int `json:"changed_pages,omitempty"`
}
func apiVersions(doc *Document) []apiVersion {
var out []apiVersion
for _, v := range doc.Versions {
out = append(out, apiVersion{
Slug: v.Slug,
URL: versionPath(doc.ID, v.Slug),
PDF: versionFilePath(doc.ID, v.Slug, v.PDFFile),
PDFSha256: v.SHA,
Images: v.Images,
FullText: v.FullText,
Publications: v.PubIndex,
ChangedPages: changedPageNumbers(v.ChangedPages),
})
}
return out
}
type apiMatchedVersion struct {
Slug string `json:"slug"`
Label string `json:"label"`
URL string `json:"url"`
Snippet string `json:"snippet"`
}
func changedPageNumbers(pages []ChangedPage) []int {
var out []int
for _, p := range pages {
out = append(out, p.Page)
}
return out
}

211
articles.go Normal file
View file

@ -0,0 +1,211 @@
// articles.go - news articles as first-class objects.
//
// An article's identity is its publication LINK. Giving a publication entry
// a title is what promotes its article to a story.
package main
import (
"crypto/sha256"
"encoding/hex"
"net/http"
"net/url"
"slices"
"sort"
"strings"
)
type Article struct {
Key string // short stable hash of the link (the /article/{key} path)
Title string
Link string
Host string // display fallback when Title is empty
Publisher string
Date string // earliest date_published seen
Journalists []string // union across docs, first-seen order
DocIDs []string
Book bool // published by a book publisher, not a news outlet
}
// bookPublishers are the publication entries that are books, not news
// articles (Greenwald's No Place to Hide, Gellman's Dark Mirror, Poitras's
// Astro Noise).
var bookPublishers = map[string]bool{
"Metropolitan Books": true,
"Penguin Press": true,
"Yale University Press": true,
}
// articleKeyFor derives the stable URL key for a link
func articleKeyFor(link string) string {
h := sha256.Sum256([]byte(link))
return hex.EncodeToString(h[:])[:12]
}
func hostOf(link string) string {
u, err := url.Parse(link)
if err != nil || u.Host == "" {
return link
}
return strings.TrimPrefix(u.Host, "www.")
}
// buildArticles indexes every publication entry that has a link
func (co *Corpus) buildArticles() {
co.articleByLink = make(map[string]*Article)
co.articleIndex = make(map[string]*Article)
co.articles = nil
for _, d := range co.archive.Docs {
for _, p := range d.Publication {
if p.Link == "" {
continue
}
a := co.articleByLink[p.Link]
if a == nil {
a = &Article{
Key: articleKeyFor(p.Link),
Link: p.Link,
Host: hostOf(p.Link),
}
co.articleByLink[p.Link] = a
co.articleIndex[a.Key] = a
}
if a.Title == "" && p.Title != "" {
a.Title = p.Title
}
if a.Publisher == "" && p.Publisher != "" {
a.Publisher = p.Publisher
a.Book = bookPublishers[p.Publisher]
}
if p.DatePublished != "" && (a.Date == "" || p.DatePublished < a.Date) {
a.Date = p.DatePublished
}
for _, j := range p.Journalists {
if j == "" {
continue
}
if !slices.ContainsFunc(a.Journalists, func(x string) bool { return sameValue(x, j) }) {
a.Journalists = append(a.Journalists, j)
}
}
dup := false
for _, id := range a.DocIDs {
if id == d.ID {
dup = true
break
}
}
if !dup {
a.DocIDs = append(a.DocIDs, d.ID)
}
}
}
for _, a := range co.articleByLink {
co.articles = append(co.articles, *a)
}
// Newest first; undated articles sort last; ties by title then link so
// the order is deterministic.
sort.Slice(co.articles, func(i, j int) bool {
di, dj := co.articles[i].Date, co.articles[j].Date
if di == "" && dj != "" {
return false
}
if di != "" && dj == "" {
return true
}
if di != dj {
return di > dj
}
ti := strings.ToLower(displayTitle(co.articles[i]))
tj := strings.ToLower(displayTitle(co.articles[j]))
if ti != tj {
return ti < tj
}
return co.articles[i].Link < co.articles[j].Link
})
co.buildArticleGroups()
}
// inCollection reports whether one of the article's documents is in coll.
func (a *Article) inCollection(co *Corpus, coll string) bool {
for _, id := range a.DocIDs {
if d := co.docIndex[id]; d != nil && d.Collection == coll {
return true
}
}
return false
}
func displayTitle(a Article) string {
if a.Title != "" {
return a.Title
}
return a.Host
}
type articleYearGroup struct {
Year string
Articles []Article
}
func (co *Corpus) buildArticleGroups() {
co.articleGroups, co.articlesTitled, co.articlesBooks = nil, 0, 0
for _, a := range co.articles {
if a.Title == "" {
continue
}
co.articlesTitled++
if a.Book {
co.articlesBooks++
}
year := "Undated"
if len(a.Date) >= 4 {
year = a.Date[:4]
}
if len(co.articleGroups) == 0 || co.articleGroups[len(co.articleGroups)-1].Year != year {
co.articleGroups = append(co.articleGroups, articleYearGroup{Year: year})
}
co.articleGroups[len(co.articleGroups)-1].Articles = append(co.articleGroups[len(co.articleGroups)-1].Articles, a)
}
}
func (co *Corpus) handleArticles(w http.ResponseWriter, r *http.Request) {
render(w, r, &articlesPage{
Groups: co.articleGroups,
GroupsHTML: co.preHTML("articles"),
Total: co.articlesTitled - co.articlesBooks,
Books: co.articlesBooks,
})
}
func (co *Corpus) handleArticleView(w http.ResponseWriter, r *http.Request) {
a := co.articleIndex[r.PathValue("key")]
if a == nil {
notFound(w, r)
return
}
prefs := readPrefs(r)
scope := pageScope(nil, prefs, false)
docs := make([]*Document, 0, len(a.DocIDs))
for _, id := range a.DocIDs {
if d := co.docIndex[id]; d != nil {
docs = append(docs, d)
scope = scope.including(d.Collection)
}
}
data := &articlePage{
Base: pageWith(prefs),
Article: *a,
Total: len(docs),
BrowseQS: scope.suffix(),
Cards: newCardView(prefs, docs),
}
// A partner that only reported the story has no publisher page.
if scope.hasValue(co, "publisher", a.Publisher) {
data.PublisherURL = scope.link(categoryValueURL("publisher", a.Publisher))
}
render(w, r, data)
}

273
assemblies.go Normal file
View file

@ -0,0 +1,273 @@
// assemblies.go - hand-built sets of pages drawn from several documents,
// loaded from data/assemblies.json.
package main
import (
"encoding/json"
"fmt"
"log"
"net/http"
"net/url"
"os"
"strconv"
"strings"
)
type assemblyRef struct {
Doc string `json:"doc"`
Version string `json:"version,omitempty"`
Pages string `json:"pages"`
}
type assemblySlideSpec struct {
assemblyRef
Note string `json:"note,omitempty"`
Alt []assemblyRef `json:"alt,omitempty"`
}
type assemblySpec struct {
ID string `json:"id"`
Title string `json:"title"`
Note string `json:"note,omitempty"`
Slides []assemblySlideSpec `json:"slides"`
}
type AssemblyCopy struct {
Image string
Size imageSize
DocID string
DocTitle string
Version string
Page int
Source string // the outlets that published this copy
Path string // the document or version page it came from
}
type AssemblySlide struct {
AssemblyCopy
Pos int
Note string
Alts []AssemblyCopy
}
type Assembly struct {
ID string
Title string
Note string
Slides []AssemblySlide
DocList []string // documents the slides are shown from, in slide order
Copies int // copies the slides are shown from: a document or one of its versions
}
// parsePageList expands "1-3,7" into page numbers, keeping the order given.
func parsePageList(s string) ([]int, error) {
var out []int
for _, part := range strings.Split(s, ",") {
part = strings.TrimSpace(part)
first, last, isRange := strings.Cut(part, "-")
if !isRange {
last = first
}
a, err := strconv.Atoi(strings.TrimSpace(first))
if err != nil {
return nil, fmt.Errorf("%q is not a page or a page range", part)
}
b, err := strconv.Atoi(strings.TrimSpace(last))
if err != nil {
return nil, fmt.Errorf("%q is not a page or a page range", part)
}
if a < 1 || b < a {
return nil, fmt.Errorf("%q is not a page or a page range", part)
}
for n := a; n <= b; n++ {
out = append(out, n)
}
}
return out, nil
}
// copySource names the outlets that published a copy, oldest entry first as
// the metadata lists them. Partners that published no documents are left out.
func copySource(d *Document, slug string) string {
var names []string
seen := map[string]bool{}
for _, p := range d.Publication {
key := valueKey(p.Publisher)
if p.ReportingOnly || p.Version != slug || key == "" || seen[key] {
continue
}
seen[key] = true
names = append(names, p.Publisher)
}
return strings.Join(names, ", ")
}
func (co *Corpus) resolveAssemblyRef(label string, r assemblyRef) ([]*AssemblyCopy, bool) {
pages, err := parsePageList(r.Pages)
if err != nil {
warnf("%s: %s: pages: %v - skipped", label, r.Doc, err)
return nil, false
}
out := make([]*AssemblyCopy, len(pages))
d := co.docIndex[r.Doc]
if d == nil {
warnf("%s: no document %q - skipped", label, r.Doc)
return out, true
}
images, sizes, base, path := d.Images, d.ImageSizes, imageBase(d.ID, ""), docPath(d.ID)
if r.Version != "" {
v := d.VersionFor(r.Version)
if v == nil {
warnf("%s: %s has no version %q - skipped", label, r.Doc, r.Version)
return out, true
}
images, sizes, base, path = v.Images, v.ImageSizes, imageBase(d.ID, v.Slug), versionPath(d.ID, v.Slug)
}
byNumber := make(map[int]int, len(images))
for i, name := range images {
if n, err := pageNumber(name); err == nil {
byNumber[n] = i
}
}
source := copySource(d, r.Version)
for i, n := range pages {
k, ok := byNumber[n]
if !ok {
warnf("%s: %s %s has no page image %d - skipped", label, r.Doc, r.Version, n)
continue
}
out[i] = &AssemblyCopy{
Image: base + url.PathEscape(images[k]), Size: sizes[k], DocID: d.ID, DocTitle: d.Title,
Version: r.Version, Page: n, Source: source, Path: path,
}
}
return out, true
}
func (co *Corpus) buildAssembly(spec assemblySpec) *Assembly {
label := "assemblies.json: " + spec.ID
a := &Assembly{ID: spec.ID, Title: strings.TrimSpace(spec.Title), Note: strings.TrimSpace(spec.Note)}
seenImage := map[string]bool{}
seenDoc := map[string]bool{}
seenCopy := map[[2]string]bool{}
for _, s := range spec.Slides {
copies, ok := co.resolveAssemblyRef(label, s.assemblyRef)
if !ok {
continue
}
alts := make([][]*AssemblyCopy, 0, len(s.Alt))
for _, ref := range s.Alt {
c, ok := co.resolveAssemblyRef(label, ref)
if !ok {
continue
}
if len(c) != len(copies) {
warnf("%s: alt %s lists %d pages for a run of %d - skipped", label, ref.Doc, len(c), len(copies))
continue
}
alts = append(alts, c)
}
for i, c := range copies {
if c == nil {
continue
}
if seenImage[c.Image] {
warnf("%s: %s page %d is listed twice", label, c.DocID, c.Page)
}
seenImage[c.Image] = true
if !seenDoc[c.DocID] {
seenDoc[c.DocID] = true
a.DocList = append(a.DocList, c.DocID)
}
slide := AssemblySlide{AssemblyCopy: *c, Pos: len(a.Slides) + 1, Note: s.Note}
seenCopy[[2]string{c.DocID, c.Version}] = true
for _, alt := range alts {
if alt[i] != nil {
slide.Alts = append(slide.Alts, *alt[i])
}
}
a.Slides = append(a.Slides, slide)
}
}
a.Copies = len(seenCopy)
return a
}
func (co *Corpus) loadAssemblies() {
co.assemblies = nil
co.assemblyIndex = map[string]*Assembly{}
co.assembliesByDoc = map[string][]*Assembly{}
p := projectPath("assemblies.json")
b, err := os.ReadFile(p)
if err != nil {
log.Printf("Note: %s not found - no assemblies.", p)
return
}
var specs []assemblySpec
if err := json.Unmarshal(b, &specs); err != nil {
warnf("%s is malformed: %v - assemblies disabled.", p, err)
return
}
for _, spec := range specs {
spec.ID = strings.TrimSpace(spec.ID)
if spec.ID == "" || !docIDSafe(spec.ID) {
warnf("assemblies.json: id %q is empty or unsafe in a URL - skipped", spec.ID)
continue
}
if co.assemblyIndex[spec.ID] != nil {
warnf("assemblies.json: id %q appears twice - keeping the first", spec.ID)
continue
}
a := co.buildAssembly(spec)
if len(a.Slides) == 0 {
warnf("assemblies.json: %s has no slides that can be shown - skipped", spec.ID)
continue
}
co.assemblies = append(co.assemblies, a)
co.assemblyIndex[a.ID] = a
linked := map[string]bool{}
link := func(id string) {
if !linked[id] {
linked[id] = true
co.assembliesByDoc[id] = append(co.assembliesByDoc[id], a)
}
}
for _, s := range a.Slides {
link(s.DocID)
for _, alt := range s.Alts {
link(alt.DocID)
}
}
}
log.Printf("Assemblies loaded: %d", len(co.assemblies))
}
// assemblyPath is an assembly's page; assemblies live under /explore.
const assembliesPath = "/explore/assemblies"
func assemblyPath(id string) string { return assembliesPath + "/" + url.PathEscape(id) }
func (co *Corpus) handleAssemblies(w http.ResponseWriter, r *http.Request) {
prefs := readPrefs(r)
render(w, r, &assembliesPage{
Base: pageWith(prefs),
Assemblies: co.assemblies,
TextOnly: prefs.TextOnly,
})
}
func (co *Corpus) handleAssemblyView(w http.ResponseWriter, r *http.Request) {
a := co.assemblyIndex[r.PathValue("id")]
if a == nil {
notFound(w, r)
return
}
prefs := readPrefs(r)
render(w, r, &assemblyPage{
Base: pageWith(prefs),
Assembly: a,
TextOnly: prefs.TextOnly,
})
}

2273
browse.go Normal file

File diff suppressed because it is too large Load diff

102
changelog.go Normal file
View file

@ -0,0 +1,102 @@
// changelog.go - the archive changelog (/changelog).
//
// The changelog is a HAND-MAINTAINED file: data/changelog.json. Shape:
//
// [
// {
// "date": "2026-06-11", // YYYY-MM-DD (or full RFC3339 if you care)
// "type": "added", // added | removed | updated | note
// "id": "some-doc-id", // which document (omit for type=note)
// "title": "Document title", // shown in the headline (omit for note)
// "components": ["pdf"], // optional, for updated: what changed
// "note": "free text" // the body for type=note, optional otherwise
// }
// ]
package main
import (
"encoding/json"
"log"
"net/http"
"os"
"sort"
"strings"
"time"
)
type ChangeEntry struct {
Date string `json:"date"`
Type string `json:"type"`
ID string `json:"id,omitempty"`
Title string `json:"title,omitempty"`
Components []string `json:"components,omitempty"`
Note string `json:"note,omitempty"`
}
func (co *Corpus) loadChangelog() {
p := projectPath("changelog.json")
b, err := os.ReadFile(p)
if err != nil {
log.Printf("Note: %s not found - the changelog will be empty (see the format at the top of changelog.go).", p)
return
}
if err := json.Unmarshal(b, &co.changelog); err != nil {
warnf("%s is malformed: %v - changelog disabled.", p, err)
co.changelog = nil
return
}
sort.SliceStable(co.changelog, func(i, j int) bool {
return co.changelog[i].Date > co.changelog[j].Date
})
for _, e := range co.changelog {
if !validChangelogDate(e.Date) {
warnf("changelog entry %q has date %q - must be YYYY-MM-DD or RFC3339", e.Headline(), e.Date)
}
if e.Type != "removed" && e.ID != "" && co.docIndex[e.ID] == nil {
warnf("changelog entry %q names document %q, which is not in the corpus", e.Headline(), e.ID)
}
}
log.Printf("Changelog loaded: %d entries", len(co.changelog))
}
func validChangelogDate(d string) bool {
if _, err := time.Parse("2006-01-02", d); err == nil {
return true
}
_, err := time.Parse(time.RFC3339, d)
return err == nil
}
func (e ChangeEntry) Headline() string {
switch e.Type {
case "added":
return "Added: " + e.Title
case "removed":
return "Removed: " + e.Title
case "updated":
s := "Updated: " + e.Title
if len(e.Components) > 0 {
s += " (" + strings.Join(e.Components, ", ") + ")"
}
return s
default: // note
if e.Note != "" {
return e.Note
}
return "Archive note"
}
}
func (e ChangeEntry) DocURL() string {
if e.ID != "" && e.Type != "removed" {
return docPath(e.ID)
}
return "/changelog"
}
func (co *Corpus) handleChangelog(w http.ResponseWriter, r *http.Request) {
render(w, r, &changelogPage{
Entries: co.changelog,
Total: len(co.changelog),
})
}

298
codewords.go Normal file
View file

@ -0,0 +1,298 @@
package main
import (
"html/template"
"net/http"
"net/url"
"sort"
"strings"
)
// The codeword concordance: an automatically extracted index of ALL-CAPS
// tokens across the corpus - program names, compartments, org designators.
var codewordStoplist = map[string]bool{
"REDACTED": true,
"DYNAMIC": true,
"PAGE": true,
"HIGHEST": true,
"POSSIBLE": true,
"DERIVED": true,
"DECLASSIFY": true,
"JAN": true,
"ADDED": true,
"CLASSIFICATION": true,
}
type codewordEntry struct {
Token string
Count int // total occurrences across the corpus
Docs int // number of documents containing it
First string // earliest document year containing it ("" if only undated docs)
Last string // latest such year
}
func isCodewordToken(w string) bool {
if len(w) < 3 || w[0] < 'A' || w[0] > 'Z' {
return false
}
letters := 0
var seen [26]bool
distinct := 0
for i := 0; i < len(w); i++ {
c := w[i]
switch {
case c >= 'A' && c <= 'Z':
letters++
if !seen[c-'A'] {
seen[c-'A'] = true
distinct++
}
case c >= '0' && c <= '9':
// digits allowed
default:
return false
}
}
return letters >= 3 && distinct >= 2
}
func scanRuns(text string, mention func(w string)) {
start := -1
for j := 0; j <= len(text); j++ {
alnum := false
if j < len(text) {
ch := text[j]
alnum = (ch >= 'A' && ch <= 'Z') || (ch >= 'a' && ch <= 'z') || (ch >= '0' && ch <= '9')
}
if alnum {
if start < 0 {
start = j
}
continue
}
if start >= 0 {
mention(text[start:j])
start = -1
}
}
}
func (co *Corpus) buildCodewords() {
co.codewords, co.codewordTable = nil, nil
caps := map[string]int{}
other := map[string]int{}
docCaps := make([][]string, len(co.archive.Docs))
for i := range co.archive.Docs {
d := &co.archive.Docs[i]
here := map[string]bool{}
count := func(w string) {
switch {
case codewordStoplist[w]:
// boilerplate or archive markup (codewordStoplist)
case isCodewordToken(w):
caps[w]++
here[w] = true
default:
other[strings.ToLower(w)]++
}
}
scanRuns(d.Title, count)
scanRuns(d.FullText, count)
for w := range here {
docCaps[i] = append(docCaps[i], w)
}
}
for w, cnt := range caps {
if cnt < 2 || cnt <= other[strings.ToLower(w)] {
continue
}
co.codewords = append(co.codewords, codewordEntry{Token: w, Count: cnt})
}
kept := make(map[string]*codewordEntry, len(co.codewords))
for i := range co.codewords {
kept[co.codewords[i].Token] = &co.codewords[i]
}
for i := range co.archive.Docs {
d := &co.archive.Docs[i]
year := docYear(d)
inDoc := map[string]bool{}
exact := func(w string) {
if e := kept[w]; e != nil {
inDoc[e.Token] = true
}
}
folded := func(w string) {
if e := kept[strings.ToUpper(w)]; e != nil {
inDoc[e.Token] = true
}
}
for _, w := range docCaps[i] {
exact(w)
}
scanRuns(d.Description, exact)
scanRuns(d.LongDescription, exact)
for _, p := range d.DocMeta.Programs {
scanRuns(p, folded)
}
for _, t := range d.Tags {
scanRuns(t, folded)
}
for w := range inDoc {
e := kept[w]
e.Docs++
if year == "" {
continue
}
if e.First == "" || year < e.First {
e.First = year
}
if year > e.Last {
e.Last = year
}
}
}
sort.Slice(co.codewords, func(i, j int) bool {
if co.codewords[i].Count != co.codewords[j].Count {
return co.codewords[i].Count > co.codewords[j].Count
}
return co.codewords[i].Token < co.codewords[j].Token
})
lowerProg := make(map[string]string, len(co.archive.Facets["program"]))
for p := range co.archive.Facets["program"] {
lowerProg[valueKey(p)] = p
}
shown := co.codewords
if len(shown) > codewordRows {
shown = shown[:codewordRows]
}
co.codewordTable = make([]codewordRow, 0, len(shown))
for _, e := range shown {
row := codewordRow{codewordEntry: e, Gloss: co.glossaryLookup(e.Token), SearchURL: tokenSearchURL(e.Token)}
if p, ok := lowerProg[valueKey(e.Token)]; ok {
row.ProgURL = wholeArchive.link(categoryValueURL("program", p))
}
if e.First != "" {
row.FirstURL = yearTokenURL(e.Token, e.First)
}
if e.Last != "" {
row.LastURL = yearTokenURL(e.Token, e.Last)
}
if row.Gloss != nil {
row.GlossURL = template.URL("/glossary#term-" + url.PathEscape(row.Gloss.Term))
}
co.codewordTable = append(co.codewordTable, row)
}
co.buildCodewordViews()
}
func yearTokenURL(token, year string) template.URL {
return template.URL(wholeArchive.link(dateRangeSearchURL("TE", exactClause(token), [2]string{"date_doc", year})))
}
func tokenSearchURL(token string) template.URL {
return template.URL(wholeArchive.link("/search?q=" + url.QueryEscape(exactClause(token))))
}
type codewordRow struct {
codewordEntry
Gloss *GlossaryEntry
ProgURL string
SearchURL template.URL
FirstURL, LastURL template.URL
GlossURL template.URL
}
// codewordRows caps the page
const codewordRows = 500
var codewordCols = []sortCol{
{Key: "token", Label: "Token"},
{Key: "count", Label: "Occurrences", DescFirst: true},
{Key: "docs", Label: "Documents", DescFirst: true},
{Key: "first", Label: "First seen"},
{Key: "last", Label: "Last seen", DescFirst: true},
{Key: "gloss", Label: "Glossary"},
}
func (co *Corpus) handleCodewords(w http.ResponseWriter, r *http.Request) {
q := r.URL.Query()
sortBy := sortToken(q.Get("sort"))
order := sortToken(q.Get("order"))
view, ok := co.codewordViews[sortBy+"/"+order]
if !ok {
view, ok = co.codewordViews[sortBy+"/"+codewordDefaultOrder(sortBy)]
}
if !ok {
view = co.codewordViews["count/desc"]
}
render(w, r, &codewordsPage{
Rows: view.Rows,
RowsHTML: co.preHTML("codewords:" + view.Key),
Total: len(co.codewords),
Shown: len(view.Rows),
Heads: view.Heads,
})
}
type codewordView struct {
Key string
Rows []codewordRow
Heads []sortHead
}
func codewordDefaultOrder(sortBy string) string {
for _, col := range codewordCols {
if col.Key == sortBy {
return orderParam(col.DescFirst)
}
}
return ""
}
func (co *Corpus) buildCodewordViews() {
co.codewordViews = make(map[string]codewordView, 2*len(codewordCols))
for _, sc := range codewordCols {
for _, order := range []string{"asc", "desc"} {
rows := append([]codewordRow(nil), co.codewordTable...)
sortCodewordRows(rows, sc.Key, order == "desc")
heads := sortHeads("?", codewordCols, sc.Key, order == "desc")
co.codewordViews[sc.Key+"/"+order] = codewordView{sc.Key + "/" + order, rows, heads}
}
}
}
func sortCodewordRows(rows []codewordRow, sortBy string, desc bool) {
sort.SliceStable(rows, func(i, j int) bool {
var less, eq bool
switch sortBy {
case "token":
less, eq = rows[i].Token < rows[j].Token, rows[i].Token == rows[j].Token
case "docs":
less, eq = rows[i].Docs < rows[j].Docs, rows[i].Docs == rows[j].Docs
case "first", "last":
a, b := rows[i].First, rows[j].First
if sortBy == "last" {
a, b = rows[i].Last, rows[j].Last
}
if (a == "") != (b == "") {
return b == ""
}
less, eq = a < b, a == b
case "gloss":
a, b := rows[i].Gloss != nil, rows[j].Gloss != nil
less, eq = !a && b, a == b // missing before defined, ascending
default:
less, eq = rows[i].Count < rows[j].Count, rows[i].Count == rows[j].Count
}
if eq {
return rows[i].Token < rows[j].Token
}
if desc {
return !less
}
return less
})
}

162
corpus.go Normal file
View file

@ -0,0 +1,162 @@
package main
import (
"html/template"
"net/http"
"sync/atomic"
)
// Corpus is everything a load derives from the data and project directories: the
// documents, every index and table built from them, and the page bodies
// rendered ahead of time. One is built per load and never changed after it
// goes live, so handlers read it without locks.
type Corpus struct {
// api.go
facetsJSON []byte
facetsETag string
// articles.go
articles []Article
articleIndex map[string]*Article // key -> article
articleByLink map[string]*Article // link -> article
articleGroups []articleYearGroup
articlesTitled int
articlesBooks int // of articlesTitled
// assemblies.go
assemblies []*Assembly
assemblyIndex map[string]*Assembly
assembliesByDoc map[string][]*Assembly
// browse.go
// homeStats holds the corpus-derived widgets on the home page - the year
// histograms and top-N facet lists.
//
// By default every figure on the home page describes the REPORTED
// collection; homeStatsSIDtoday (/?sidtoday=1) adds the SIDtoday archive. FOIA
// material can postdate the June 2013 disclosures (a 2014 "year written"
// bar on a Snowden archive reads as impossible at a glance), and the
// SIDtoday archive is +2,100 issues of an internal newsletter that swamps
// the lists and the leak-era shape the charts exist to show.
// Every bar and list entry links to its /browse/{category}/{value} page
// scoped to that collection, so the number shown is the number of
// documents the page lists.
homeStats, homeStatsSIDtoday homeFigures
randomPools map[string][]int
collectionDocs map[string][]*Document
sidtodayDocs []*Document
// everyDoc is every document as pointers, built once.
everyDoc []*Document
archiveTimeline []archiveYear
categoryRowHTML map[string][]template.HTML // table key -> each entry's row, as Entries has them
azRowHTML map[string]template.HTML
categoryRowSize map[string]int
docRowHTML map[string]template.HTML
docRowMarkedHTML map[string]template.HTML // rows with the "reported" badge, for ReportedIssue documents
// categoryTables holds every category's table under its slug.
categoryTables map[string]*categoryTable
// seriesBySize is the series index of the SIDtoday tab.
seriesBySize map[bool][]categoryEntry
// categoryValueCounts is how many values each category's list holds.
categoryValueCounts map[bool]map[string]int
// categoryDocs is the posting list behind every /browse/{category}/{value}
// page: slug -> folded value -> the documents holding it, in corpus order.
categoryDocs map[string]map[string][]int32
// categoryNames maps each category's index keys to the spelling the lists
// and counts show: the first one in corpus order.
categoryNames map[string]map[string]string
// categoryTypeCounts splits each category value's count by document type
// (REPORTED / SIDTODAY / FOIA): the list pages' breakdown columns. They
// count type, not collection.
categoryTypeCounts map[string]map[string]*collCounts
// changelog.go
changelog []ChangeEntry
// codewords.go
codewords []codewordEntry
codewordTable []codewordRow
codewordViews map[string]codewordView
// data.go
archive Archive
serialPrev map[string]string
serialNext map[string]string
seriesIndex map[string]*seriesGroup
// explore.go
exploreTables map[string][]exploreTable
classPatterns struct {
Strips []clsYearRow
Sparse []clsYearRow
Cols []facetEntry // crossCols, each linked to its page
HeatRows []heatRow
}
// explore_ids.go
idRows []idRow
idChecksums string
// glossary.go
glossary []GlossaryEntry
glossaryIndex map[string]*GlossaryEntry
glossaryGroups []glossaryGroup
// main.go
prerendered map[string]template.HTML
// outlets.go
outletTables map[string]outletTable
// portions.go
portionStats portionSummary
// search.go
tfidf TFIDFIndex
// docIndex is populated by loadData() in data.go after archive.Docs is fully
// built. Provides O(1) doc lookup by ID.
docIndex map[string]*Document
// seo.go
sitemapPaths []string
canonicalSitemap []byte
// sitemap.go
sitemapParts [][]byte
}
// live is the corpus requests are served from. It is replaced whole, never
// modified: a load builds a new Corpus and stores it here when it is done.
var live atomic.Pointer[Corpus]
func init() { live.Store(&Corpus{}) }
// corpus returns the live corpus. Handlers do not call it. bind hands each
// request the corpus that was live when it arrived, and everything the
// request touches is reached through that one value.
func corpus() *Corpus { return live.Load() }
// loadData reads the corpus under dataDir into a new Corpus, pre-renders its
// big page bodies (from tmplSet, so parse the templates first).
func loadData() {
co := &Corpus{}
co.load()
co.prerenderPages()
live.Store(co)
}
// bind adapts a handler to the mux. the request is served from the corpus
// that is live when it arrives, and from that one throughout.
func bind(h func(*Corpus, http.ResponseWriter, *http.Request)) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) { h(live.Load(), w, r) }
}

1470
data.go Normal file

File diff suppressed because it is too large Load diff

377
explore.go Normal file
View file

@ -0,0 +1,377 @@
package main
import (
"log"
"net/http"
"sort"
"strconv"
"strings"
)
// The Explore section (/explore).
type exploreCol struct {
Header string
Key string
Ratio bool
}
type exploreMetric struct {
Slug string
Title string
Blurb string
Cols []exploreCol
Vals func(d *Document) ([]float64, bool)
}
const exploreRows = 200 // rows shown per page ("top 200 of N")
var exploreMetrics = []exploreMetric{
{
Slug: "programs",
Title: "Most programs mentioned",
Blurb: "Documents referencing the most surveillance programs.",
Cols: []exploreCol{
{Header: "Programs", Key: "programs"},
{Header: "Pages", Key: "pages"},
{Header: "Programs / page", Key: "ratio", Ratio: true},
},
Vals: func(d *Document) ([]float64, bool) {
n := len(d.DocMeta.Programs)
if n == 0 {
return nil, false
}
pages, ratio := float64(d.Pages), -1.0
if d.Pages > 0 {
ratio = float64(n) / float64(d.Pages)
}
return []float64{float64(n), pages, ratio}, true
},
},
{
Slug: "people",
Title: "Most people mentioned",
Blurb: "Documents whose metadata names the most individuals.",
Cols: []exploreCol{{Header: "People", Key: "people"}},
Vals: func(d *Document) ([]float64, bool) {
n := len(d.DocMeta.People)
return []float64{float64(n)}, n > 0
},
},
{
Slug: "countries",
Title: "Most countries mentioned",
Blurb: "Documents spanning the most countries.",
Cols: []exploreCol{{Header: "Countries", Key: "countries"}},
Vals: func(d *Document) ([]float64, bool) {
n := len(d.DocMeta.Countries)
return []float64{float64(n)}, n > 0
},
},
{
Slug: "pages",
Title: "Longest documents",
Blurb: "Documents with the most pages.",
Cols: []exploreCol{{Header: "Pages", Key: "pages"}},
Vals: func(d *Document) ([]float64, bool) {
return []float64{float64(d.Pages)}, d.Pages > 0
},
},
}
func exploreMetricBySlug(slug string) *exploreMetric {
for i := range exploreMetrics {
if exploreMetrics[i].Slug == slug {
return &exploreMetrics[i]
}
}
return nil
}
func (co *Corpus) handleExploreIndex(w http.ResponseWriter, r *http.Request) {
render(w, r, &explorePage{
Metrics: exploreMetrics,
HasAssemblies: len(co.assemblies) > 0,
})
}
type exploreRow struct {
Rank int
Doc Document
Vals []string
}
type exploreTable struct {
Rows []exploreRow
Total int
}
func exploreSet(slug string, foia bool) string {
if foia {
return slug
}
return slug + ":nofoia"
}
func (co *Corpus) buildExploreTables() {
co.exploreTables = make(map[string][]exploreTable, 2*len(exploreMetrics))
for mi := range exploreMetrics {
m := &exploreMetrics[mi]
if _, dup := co.exploreTables[m.Slug]; dup {
log.Fatalf("explore: duplicate metric slug %q", m.Slug)
}
for _, foia := range []bool{true, false} {
co.exploreTables[exploreSet(m.Slug, foia)] = co.buildExploreTable(m, foia)
}
}
}
func (co *Corpus) buildExploreTable(m *exploreMetric, foia bool) []exploreTable {
type scored struct {
doc *Document
vals []float64
}
var rows []scored
for i := range co.archive.Docs {
d := &co.archive.Docs[i]
if !foia && d.Collection == collectionFOIA {
continue
}
if vals, ok := m.Vals(d); ok {
rows = append(rows, scored{d, vals})
}
}
tables := make([]exploreTable, len(m.Cols))
for sortIdx := range m.Cols {
sorted := append([]scored(nil), rows...)
sort.SliceStable(sorted, func(i, j int) bool {
if a, b := sorted[i].vals[sortIdx], sorted[j].vals[sortIdx]; a != b {
return a > b
}
if a, b := sorted[i].vals[0], sorted[j].vals[0]; a != b {
return a > b
}
return sorted[i].doc.Title < sorted[j].doc.Title
})
if len(sorted) > exploreRows {
sorted = sorted[:exploreRows]
}
out := make([]exploreRow, len(sorted))
for i, rw := range sorted {
vals := make([]string, len(rw.vals))
for k, v := range rw.vals {
switch {
case v < 0:
vals[k] = "-"
case m.Cols[k].Ratio:
vals[k] = trimRatio(v)
default:
vals[k] = strconv.Itoa(int(v))
}
}
out[i] = exploreRow{Rank: i + 1, Doc: *rw.doc, Vals: vals}
}
tables[sortIdx] = exploreTable{Rows: out, Total: len(rows)}
}
return tables
}
func exploreKey(set string, sortIdx int) string {
return "explore:" + set + ":" + strconv.Itoa(sortIdx)
}
func (co *Corpus) handleExploreMetric(w http.ResponseWriter, r *http.Request) {
m := exploreMetricBySlug(r.PathValue("metric"))
if m == nil {
notFound(w, r)
return
}
sortIdx := 0
sortBy := r.URL.Query().Get("sort")
for i, c := range m.Cols {
if c.Key == sortBy {
sortIdx = i
break
}
}
set := exploreSet(m.Slug, readPrefs(r).IncludeFOIA)
t := co.exploreTables[set][sortIdx]
cols := []sortCol{{Label: "#", Class: "num-col"}, {Label: "Document"}, {Label: "Date"}}
for _, c := range m.Cols {
cols = append(cols, sortCol{Key: c.Key, Label: c.Header, Class: "num-col", DescFirst: true, Fixed: true})
}
render(w, r, &exploreTablePage{
Metric: m,
Rows: t.Rows,
RowsHTML: co.preHTML(exploreKey(set, sortIdx)),
Total: t.Total,
Shown: len(t.Rows),
Heads: sortHeads("?", cols, m.Cols[sortIdx].Key, true),
})
}
func trimRatio(v float64) string {
s := strconv.FormatFloat(v, 'f', 2, 64)
s = strings.TrimRight(s, "0")
return strings.TrimRight(s, ".")
}
// ---------------------------------------------------------------------------
// /explore/classification
type clsSeg struct {
Value string
Count int
Pct int
URL string
Class string // seg-0..seg-3, one per stripLevels entry
}
type clsYearRow struct {
Year string
URL string // the year's page, over every collection as the counts are
Total int
Segs []clsSeg
}
type heatCell struct {
Count int
URL string
}
type heatRow struct {
Name string
URL string
Cells []heatCell
}
var crossCols = []string{"TOP SECRET", "SECRET", "CONFIDENTIAL", "UNCLASSIFIED", "SI", "COMINT", "STRAP 1", "STRAP 2", "TK"}
var stripLevels = []string{"TOP SECRET", "SECRET", "CONFIDENTIAL", "UNCLASSIFIED"}
const stripMinMarks = 5
func (co *Corpus) buildClassPatterns() {
top := crossCols
rank := make(map[string]int, len(top))
for i, v := range top {
rank[valueKey(v)] = i
}
levelRank := make(map[string]int, len(stripLevels))
for i, v := range stripLevels {
levelRank[valueKey(v)] = i
}
yearCounts := map[string][]int{}
agCounts := map[string][]int{}
agName := map[string]string{}
for i := range co.archive.Docs {
d := &co.archive.Docs[i]
year := docYear(d)
agency := d.DocMeta.Creator
if n, ok := agName[valueKey(agency)]; ok {
agency = n
} else {
agName[valueKey(agency)] = agency
}
for _, cv := range d.DocMeta.Classification {
if lvl, isLevel := levelRank[valueKey(cv)]; isLevel && year != "" {
yc := yearCounts[year]
if yc == nil {
yc = make([]int, len(stripLevels))
yearCounts[year] = yc
}
yc[lvl]++
}
idx, known := rank[valueKey(cv)]
if known && agency != "" {
ac := agCounts[agency]
if ac == nil {
ac = make([]int, len(top))
agCounts[agency] = ac
}
ac[idx]++
}
}
}
years := make([]string, 0, len(yearCounts))
for y := range yearCounts {
years = append(years, y)
}
sort.Strings(years)
strips := make([]clsYearRow, 0, len(years))
var sparse []clsYearRow
for _, y := range years {
yc := yearCounts[y]
total := 0
for _, n := range yc {
total += n
}
row := clsYearRow{Year: y, URL: wholeArchive.link(categoryValueURL("docdate", y)), Total: total}
if total < stripMinMarks {
sparse = append(sparse, row)
continue
}
for i, n := range yc {
if n == 0 {
continue
}
row.Segs = append(row.Segs, clsSeg{
Value: stripLevels[i],
Count: n,
Pct: barPct(n, total, 1),
URL: wholeArchive.link(dateRangeSearchURL("CL", exactClause(stripLevels[i]), [2]string{"date_doc", y})),
Class: "seg-" + strconv.Itoa(i),
})
}
strips = append(strips, row)
}
mass := make(map[string]int, len(agCounts))
for a, cs := range agCounts {
for _, n := range cs {
mass[a] += n
}
}
ags := rankCounts(mass, -1)
rows := make([]heatRow, 0, len(ags))
for _, a := range ags {
hr := heatRow{Name: a, URL: wholeArchive.link(categoryValueURL("agency", a))}
for i, n := range agCounts[a] {
cell := heatCell{Count: n}
if n > 0 {
cell.URL = wholeArchive.link(searchURL([]searchClause{
{Field: "CR", Value: exactClause(a)},
{Field: "CL", Value: exactClause(top[i])},
}))
}
hr.Cells = append(hr.Cells, cell)
}
rows = append(rows, hr)
}
cols := make([]facetEntry, len(top))
for i, v := range top {
cols[i] = facetEntry{Name: v, URL: wholeArchive.link(categoryValueURL("classification", v))}
}
co.classPatterns.Strips = strips
co.classPatterns.Sparse = sparse
co.classPatterns.Cols = cols
co.classPatterns.HeatRows = rows
}
func (co *Corpus) handleClassPatterns(w http.ResponseWriter, r *http.Request) {
render(w, r, &classPatternsPage{
Cols: co.classPatterns.Cols,
Levels: stripLevels,
Strips: co.classPatterns.Strips,
Sparse: co.classPatterns.Sparse,
MinMarks: stripMinMarks,
HeatRows: co.classPatterns.HeatRows,
})
}

114
explore_ids.go Normal file
View file

@ -0,0 +1,114 @@
// explore_ids.go - /explore/ids: every document's identifiers in one table.
//
// /explore/ids.txt is the same checksums in sha256sum(1) format, so a mirror
// or a download verifies with "sha256sum -c" in the corpus root.
package main
import (
"fmt"
"net/http"
"path/filepath"
"sort"
"strings"
)
type idVersionRow struct {
Slug string
PDFFile string
PDFURL string
Size string
Pages int
SHA string
}
type idRow struct {
ID string
Collection string
Pages int
PDFFile string
PDFURL string
Size string
SHA string
Aka []AkaEntry
Versions []idVersionRow
}
func humanSize(n int64) string {
switch {
case n <= 0:
return ""
case n < 1<<10:
return fmt.Sprintf("%d B", n)
case n < 1<<20:
return fmt.Sprintf("%.0f KB", float64(n)/(1<<10))
case n < 1<<30:
return fmt.Sprintf("%.1f MB", float64(n)/(1<<20))
}
return fmt.Sprintf("%.2f GB", float64(n)/(1<<30))
}
func (co *Corpus) buildIDRows() {
co.idRows = make([]idRow, 0, len(co.archive.Docs))
var sums strings.Builder
for i := range co.archive.Docs {
d := &co.archive.Docs[i]
row := idRow{
ID: d.ID,
Collection: d.Collection,
Pages: d.Pages,
PDFFile: d.PDFFile,
PDFURL: filePath(d.ID, d.PDFFile),
Size: humanSize(d.PDFSize),
SHA: strings.TrimSpace(d.PDFSha256),
Aka: d.AkaID,
}
for _, v := range d.Versions {
row.Versions = append(row.Versions, idVersionRow{
Slug: v.Slug,
PDFFile: v.PDFFile,
PDFURL: versionFilePath(d.ID, v.Slug, v.PDFFile),
Size: humanSize(v.PDFSize),
Pages: len(v.Images),
SHA: v.SHA,
})
}
co.idRows = append(co.idRows, row)
rel, err := filepath.Rel(dataDir, d.Dir)
if err != nil {
rel = d.ID
}
if row.SHA != "" {
sums.WriteString(row.SHA + " " + filepath.ToSlash(filepath.Join(rel, d.PDFFile)) + "\n")
}
for _, v := range d.Versions {
if v.SHA != "" {
sums.WriteString(v.SHA + " " + filepath.ToSlash(filepath.Join(rel, "versions", v.Slug, v.PDFFile)) + "\n")
}
}
}
sort.Slice(co.idRows, func(i, j int) bool {
a, b := strings.ToLower(co.idRows[i].ID), strings.ToLower(co.idRows[j].ID)
if a != b {
return a < b
}
return co.idRows[i].ID < co.idRows[j].ID
})
co.idChecksums = sums.String()
}
func (co *Corpus) handleExploreIDs(w http.ResponseWriter, r *http.Request) {
render(w, r, &idsPage{
Rows: co.idRows,
RowsHTML: co.preHTML("ids"),
Shown: len(co.idRows),
})
}
func (co *Corpus) handleExploreIDsText(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
w.Header().Set("Cache-Control", "public, max-age=3600")
w.Write([]byte(co.idChecksums))
}

135
glossary.go Normal file
View file

@ -0,0 +1,135 @@
// glossary.go - codename/term glossary loaded from data/glossary.json.
package main
import (
"encoding/json"
"html/template"
"log"
"net/http"
"os"
"sort"
"strings"
)
type GlossaryEntry struct {
Term string `json:"term"`
Aka []string `json:"aka,omitempty"`
Definition string `json:"definition"`
See []string `json:"see,omitempty"`
SeeLinks []seeLink `json:"-"`
searchLink string
}
type seeLink struct{ Name, Term string }
func (e GlossaryEntry) DefinitionHTML() template.HTML {
return template.HTML(e.Definition)
}
func (e GlossaryEntry) SearchURL() string {
if e.searchLink != "" {
return e.searchLink
}
return e.buildSearchURL()
}
func (e GlossaryEntry) buildSearchURL() string {
clauses := []searchClause{{Conj: "and", Field: "TE", Value: searchClauseValue(e.Term)}}
for _, a := range e.Aka {
if strings.TrimSpace(a) != "" {
clauses = append(clauses, searchClause{Conj: "or", Field: "TE", Value: searchClauseValue(a)})
}
}
return searchURL(clauses)
}
func searchClauseValue(v string) string {
v = strings.TrimSpace(v)
if len(splitWords(v)) == 1 && len(splitWords(v)[0]) == len(v) {
return v
}
return `"` + v + `"`
}
func (co *Corpus) glossaryLookup(term string) *GlossaryEntry {
return co.glossaryIndex[strings.ToLower(term)]
}
func (co *Corpus) loadGlossary() {
p := projectPath("glossary.json")
b, err := os.ReadFile(p)
if err != nil {
log.Printf("Note: %s not found - glossary page will be empty.", p)
return
}
if err := json.Unmarshal(b, &co.glossary); err != nil {
warnf("%s is malformed: %v - glossary disabled.", p, err)
co.glossary = nil
return
}
co.glossaryIndex = map[string]*GlossaryEntry{}
sort.SliceStable(co.glossary, func(i, j int) bool {
return normFoldString(co.glossary[i].Term) < normFoldString(co.glossary[j].Term)
})
for i := range co.glossary {
e := &co.glossary[i]
e.searchLink = e.buildSearchURL()
addIndex := func(key string) {
low := strings.ToLower(key)
if prev := co.glossaryIndex[low]; prev != nil && prev != e {
warnf("glossary term/alias %q appears under both %q and %q - the later entry wins", key, prev.Term, e.Term)
}
co.glossaryIndex[low] = e
}
addIndex(e.Term)
for _, a := range e.Aka {
addIndex(a)
}
}
var unknown []string
for i := range co.glossary {
e := &co.glossary[i]
e.SeeLinks = nil
for _, name := range e.See {
l := seeLink{Name: name}
if t := co.glossaryLookup(name); t != nil {
l.Term = t.Term
} else {
unknown = append(unknown, name)
}
e.SeeLinks = append(e.SeeLinks, l)
}
}
if len(unknown) > 0 {
warnf("glossary: %d see-also names match no term or alias, shown unlinked: %s",
len(unknown), strings.Join(firstN(unknown, 8), ", "))
}
co.buildGlossaryGroups()
log.Printf("Glossary loaded: %d terms", len(co.glossary))
}
type glossaryGroup struct {
Letter string
Entries []GlossaryEntry
}
func (co *Corpus) buildGlossaryGroups() {
co.glossaryGroups = nil
for _, e := range co.glossary {
letter := firstLetter(e.Term)
if len(co.glossaryGroups) == 0 || co.glossaryGroups[len(co.glossaryGroups)-1].Letter != letter {
co.glossaryGroups = append(co.glossaryGroups, glossaryGroup{Letter: letter})
}
co.glossaryGroups[len(co.glossaryGroups)-1].Entries = append(co.glossaryGroups[len(co.glossaryGroups)-1].Entries, e)
}
}
func (co *Corpus) handleGlossary(w http.ResponseWriter, r *http.Request) {
render(w, r, &glossaryPage{
Groups: co.glossaryGroups,
GroupsHTML: co.preHTML("glossary"),
Total: len(co.glossary),
})
}

3
go.mod Normal file
View file

@ -0,0 +1,3 @@
module snowchive
go 1.22

194
history.go Normal file
View file

@ -0,0 +1,194 @@
// history.go - opt-in, cookie-local browsing history: recently viewed
// documents and recent searches. Both are OFF by default (see Prefs).
package main
import (
"net/http"
"net/url"
"strings"
)
const (
recentDocsCookie = "recent_docs"
recentSearchesCookie = "recent_searches"
recentDocsMax = 20
recentSearchesMax = 15
)
func readIDListCookie(r *http.Request, name string, max int) []string {
c, err := r.Cookie(name)
if err != nil || c.Value == "" {
return nil
}
var out []string
for _, id := range strings.Split(c.Value, "|") {
if id != "" {
out = append(out, id)
if len(out) == max {
break
}
}
}
return out
}
func pushFront(list []string, id string, max int) []string {
out := make([]string, 0, max)
out = append(out, id)
for _, x := range list {
if len(out) == max {
break
}
if x != id {
out = append(out, x)
}
}
return out
}
func pushRecentDoc(w http.ResponseWriter, r *http.Request, id string) {
ids := pushFront(readIDListCookie(r, recentDocsCookie, recentDocsMax), id, recentDocsMax)
writeListCookie(w, recentDocsCookie, ids)
}
type recentSearch struct {
Label string
URL string
QS string
}
func recordRecentSearch(w http.ResponseWriter, r *http.Request, qs string) []string {
if len(qs) > searchQSMax {
return nil
}
entry := url.QueryEscape(qs)
entries := readIDListCookie(r, recentSearchesCookie, recentSearchesMax)
entries = pushFront(entries, entry, recentSearchesMax)
return writeListCookie(w, recentSearchesCookie, entries)
}
func (co *Corpus) readRecentSearches(r *http.Request) []recentSearch {
return co.parseSearchEntries(readIDListCookie(r, recentSearchesCookie, recentSearchesMax))
}
func (co *Corpus) parseSearchEntries(entries []string) []recentSearch {
var out []recentSearch
for _, e := range entries {
qs, err := url.QueryUnescape(e)
if err != nil {
continue
}
v, err := url.ParseQuery(qs)
if err != nil {
continue
}
if label, ok := co.searchLabel(v); ok {
out = append(out, recentSearch{Label: label, URL: "/search?" + qs, QS: qs})
}
}
return out
}
const searchQSMax = 500
func (co *Corpus) savableSearch(v url.Values) bool {
if len(v.Encode()) > searchQSMax {
return false
}
_, ok := co.searchLabel(v)
return ok
}
func (co *Corpus) searchLabel(v url.Values) (label string, ok bool) {
var parts []string
fields := v["fqf"]
conjs := v["fqc"]
for i, term := range v["fqv"] {
if term == "" {
continue
}
label := term
if i < len(fields) && fields[i] != "" && fields[i] != "TE" {
code := fields[i]
if f, ok := fieldByCode[code]; ok {
label = term + " [" + f.Label + "]"
} else {
label = term + " [" + code + "]"
}
}
if i < len(conjs) && conjs[i] == "not" {
label = "-" + label
}
parts = append(parts, label)
}
var notes, filters []string
if colls := v["coll"]; len(colls) > 0 && len(colls) < len(validCollections) {
filters = append(filters, colls...)
}
if types := v["type"]; len(types) > 0 && (len(co.archive.Types) == 0 || len(types) < len(co.archive.Types)) {
filters = append(filters, types...)
}
if len(filters) > 0 {
notes = append(notes, "("+strings.Join(filters, ", ")+")")
}
if n := dateRangeNote("doc", v.Get("date_doc_start"), v.Get("date_doc_end")); n != "" {
notes = append(notes, n)
}
if n := dateRangeNote("pub", v.Get("date_pub_start"), v.Get("date_pub_end")); n != "" {
notes = append(notes, n)
}
switch {
case len(parts) == 0 && len(notes) == 0:
return "", false
case len(parts) == 0:
parts = []string{"all documents"}
}
return strings.Join(append(parts, notes...), " · "), true
}
func dateRangeNote(kind, start, end string) string {
switch {
case start == "" && end == "":
return ""
case start == end || (start != "" && end == padDateEnd(start)):
// date:2013 arrives as start "2013", end "2013-12-31": one year
return "(" + kind + " " + start + ")"
case start == "":
return "(" + kind + " to " + end + ")"
case end == "":
return "(" + kind + " from " + start + ")"
default:
return "(" + kind + " " + start + " to " + end + ")"
}
}
func handleHistoryClear(w http.ResponseWriter, r *http.Request) {
if !readPOSTForm(w, r) {
return
}
switch r.FormValue("which") {
case "docs":
delCookie(w, recentDocsCookie)
case "searches":
delCookie(w, recentSearchesCookie)
default:
delCookie(w, recentDocsCookie)
delCookie(w, recentSearchesCookie)
}
http.Redirect(w, r, safeBackPath(r.FormValue("back")), http.StatusSeeOther)
}
// safeBackPath only allows local absolute paths as redirect targets.
func safeBackPath(p string) string {
if !strings.HasPrefix(p, "/") || strings.HasPrefix(p, "//") {
return "/"
}
for i := 0; i < len(p); i++ {
if p[i] == '\\' || p[i] < 0x20 {
return "/"
}
}
return p
}

773
main.go Normal file
View file

@ -0,0 +1,773 @@
package main
import (
"bytes"
"context"
"crypto/sha256"
"embed"
"encoding/hex"
"encoding/json"
"flag"
"fmt"
"html/template"
"io/fs"
"log"
"maps"
"net"
"net/http"
"net/url"
"os"
"os/signal"
"path"
"path/filepath"
"slices"
"strconv"
"strings"
"syscall"
"time"
)
// devMode is set by the -dev flag. In dev mode templates and static files
// are read from ./templates and ./static on every request (so edits show
// without a rebuild), caching is disabled and error details are shown in
// responses. Never run production with -dev: the error detail leaks
// filesystem paths and template internals.
var devMode bool
// The templates and static assets are compiled into the binary, so a
// deployment is one executable plus the data and project directories - nothing depends
// on the working directory the process happens to start in. -dev swaps in
// the on-disk copies (see initAssets).
//
//go:embed templates static
var embeddedAssets embed.FS
var (
templateFS fs.FS // templates/*.html
staticFS fs.FS // static/*
)
// staticHashes holds a content hash per embedded asset, computed once at
// startup. It is both the assets' ETag (embed.FS files have a zero modtime,
// so net/http sends no Last-Modified and an unvalidated asset would be
// re-downloaded on every view) and their version stamp: base.html links
// them as ?v=<hash> (the asset func), and handleStatic caches exactly that
// URL forever. A changed file gets a new hash and so a new URL, whatever
// the build was stamped with.
var staticHashes map[string]string
func buildStaticHashes() {
staticHashes = map[string]string{}
err := fs.WalkDir(staticFS, ".", func(p string, d fs.DirEntry, err error) error {
if err != nil || d.IsDir() {
return err
}
b, err := fs.ReadFile(staticFS, p)
if err != nil {
return err
}
sum := sha256.Sum256(b)
staticHashes[p] = hex.EncodeToString(sum[:8])
return nil
})
if err != nil {
log.Fatalf("embedded static: %v", err)
}
}
func initAssets() {
if devMode {
templateFS = os.DirFS("templates")
staticFS = os.DirFS("static")
staticHashes = nil
return
}
var err error
if templateFS, err = fs.Sub(embeddedAssets, "templates"); err != nil {
log.Fatalf("embedded templates: %v", err)
}
if staticFS, err = fs.Sub(embeddedAssets, "static"); err != nil {
log.Fatalf("embedded static: %v", err)
}
buildStaticHashes()
}
type sortOption struct{ Key, Label string }
var sortOptions = func() []sortOption {
opts := []sortOption{{"relevance", "Relevance"}}
for _, s := range docSorts {
if s.InBar {
opts = append(opts, sortOption{s.Key, s.Label})
}
}
return opts
}()
func sortLabel(key string) string {
switch key {
case "relevance":
return "Relevance"
case "":
return "Title"
}
if s := docSortByKey[key]; s != nil {
return s.Label
}
return strings.ToUpper(key[:1]) + key[1:]
}
var (
// tmplSet holds one parsed template set PER PAGE (base + partials +
// page).
tmplSet map[string]*template.Template
funcMap = template.FuncMap{
"buildInfo": func() map[string]string {
return map[string]string{"Commit": buildCommit}
},
// articleKey is the key of the article page for a publication's
// link. Every linked publication of a loaded document has one
// (buildArticles), so the key needs no lookup - and no corpus.
"articleKey": func(link string) string {
if link == "" {
return ""
}
return articleKeyFor(link)
},
"asset": func(name string) string {
if h, ok := staticHashes[name]; ok {
return "/static/" + name + "?v=" + h
}
return "/static/" + name
},
// wholeArchiveQS is the scope that names every collection, for the
// links of a FOIA release on a page rendered once for all readers.
"wholeArchiveQS": func() string { return wholeArchive.suffix() },
"facetLinks": facetLinks,
"add": func(a, b int) int { return a + b },
"eqFold": sameValue,
"sortOptions": func() []sortOption { return sortOptions },
"sortLabel": sortLabel,
"agencyName": agencyName,
"dict": func(values ...interface{}) (map[string]interface{}, error) {
if len(values)%2 != 0 {
return nil, fmt.Errorf("dict: odd number of arguments (%d)", len(values))
}
dict := make(map[string]interface{}, len(values)/2)
for i := 0; i < len(values); i += 2 {
key, ok := values[i].(string)
if !ok {
return nil, fmt.Errorf("dict: argument %d is not a string key (%T)", i, values[i])
}
dict[key] = values[i+1]
}
return dict, nil
},
// pathEscape percent-encodes a value for use as a single URL path
// segment (so values like "TOP SECRET//SI" don't split the path).
"pathEscape": func(s string) template.URL {
return template.URL(url.PathEscape(s))
},
"azID": azID,
"pages": func(current, total int) []int {
var out []int
last := 0
for i := 1; i <= total; i++ {
if i == 1 || i == total || (i >= current-2 && i <= current+2) {
if last != 0 && i-last > 1 {
out = append(out, 0)
}
out = append(out, i)
last = i
}
}
return out
},
}
)
// Build identity, stamped at build time so mirrors are comparable at a
// glance. Shown in the footer next to the data load time:
//
// go build -ldflags "-X main.buildCommit=$(git rev-parse --short HEAD)"
var buildCommit = "dev"
// accelPrefix is set by -accel-files: the internal nginx location that maps
// to the data directory. With it, handleFiles still decides whether a file is
// served, but answers with an X-Accel-Redirect and nginx sends the bytes
// (sendfile, ranges, slow readers) without them passing through this process.
// Only set it behind nginx: any other client gets an empty 200.
var accelPrefix string
// accelExpires is set by -accel-expires: how long the proxy may keep a page
// this server marks public, sent as X-Accel-Expires (which nginx obeys ahead
// of Cache-Control and strips from the response). Browsers still get the
// short max-age. The corpus is read once at startup, so such a page cannot
// change before the next restart; clear the proxy's cache after one.
var accelExpires string
// apiEnabled is cleared by -no-api: public instances that don't want bulk
// scraping can turn the JSON API off.
var apiEnabled = true
func main() {
rebuildRelated := flag.Bool("rebuild-related", false, "Rebuild related docs index and exit")
flag.BoolVar(&devMode, "dev", false, "Dev mode: serve templates/ and static/ from the working directory and re-parse templates per request, no immutable static cache, verbose errors")
baseFlag := flag.String("base", os.Getenv("SNOWCHIVE_BASE"),
"Canonical base URL (e.g. https://archive.example.org) for the absolute URLs in robots.txt and the sitemap. "+
"Without it those URLs are built from each request's Host header, which is fine only behind a proxy that pins Host. "+
"Set this on anything internet-facing. Defaults to $SNOWCHIVE_BASE.")
listen := flag.String("listen", envOr("SNOWCHIVE_LISTEN", "127.0.0.1:8080"),
"TCP address to listen on. Defaults to $SNOWCHIVE_LISTEN, then 127.0.0.1:8080.")
root := flag.String("root", envOr("SNOWCHIVE_ROOT", "."),
"Archive directory holding data/ (the document corpus, one directory per document) and project/ "+
"(the files about the archive as a whole: glossary.json, assemblies.json, changelog.json and related.json; only -rebuild-related writes there). "+
"Defaults to $SNOWCHIVE_ROOT, then the working directory.")
check := flag.Bool("check", false,
"Load the corpus, print the warnings and the tally, then exit - non-zero when anything was reported. ")
noAPI := flag.Bool("no-api", envBool("SNOWCHIVE_NO_API"),
"Disable the read-only JSON API: every /api/ path answers a JSON 403 and the UI stops linking to it. Defaults on when $SNOWCHIVE_NO_API is true (1, t, true).")
flag.StringVar(&accelPrefix, "accel-files", os.Getenv("SNOWCHIVE_ACCEL_FILES"),
"Internal nginx location that aliases the data directory (e.g. /_files/). Document files are then answered with an X-Accel-Redirect into it and sent by nginx. "+
"Only behind nginx. Defaults to $SNOWCHIVE_ACCEL_FILES.")
accelTTL := flag.Duration("accel-expires", envDuration("SNOWCHIVE_ACCEL_EXPIRES"),
"How long the proxy may cache a page served to a reader without cookies (e.g. 10m), sent as X-Accel-Expires; browsers keep the short max-age. "+
"Clear the proxy cache after a restart with new data. Defaults to $SNOWCHIVE_ACCEL_EXPIRES.")
flag.Parse()
setRoot(*root)
if accelPrefix != "" {
if !strings.HasPrefix(accelPrefix, "/") || !strings.HasSuffix(accelPrefix, "/") {
log.Fatalf("-accel-files: %q must begin and end with a slash", accelPrefix)
}
}
if *accelTTL >= time.Second {
accelExpires = strconv.Itoa(int(*accelTTL / time.Second))
}
if *baseFlag != "" {
b, err := parseCanonicalBase(*baseFlag)
if err != nil {
log.Fatalf("-base: %v", err)
}
canonicalBase = b
}
apiEnabled = !*noAPI
initAssets()
// Before the load: it pre-renders the big page bodies from these.
if !devMode {
if err := parseTemplates(); err != nil {
log.Fatalf("Fatal: Could not parse templates at startup: %v", err)
}
}
loadData()
co := corpus()
if *check {
if loadErrors > 0 || loadWarnings > 0 {
log.Printf("check failed: %d errors, %d warnings", loadErrors, loadWarnings)
os.Exit(1)
}
log.Printf("check passed: %d documents, nothing to report", len(co.archive.Docs))
return
}
if *rebuildRelated {
log.Println("Rebuilding related docs index...")
existing := make(map[string][]string)
if b, err := os.ReadFile(relatedDocsPath()); err == nil {
if err := json.Unmarshal(b, &existing); err != nil {
warnf("could not parse existing related.json: %v - starting fresh", err)
existing = make(map[string][]string)
} else {
log.Printf("Loaded existing related.json with %d entries", len(existing))
}
}
stale := 0
for id := range existing {
if co.unknownDoc(id) {
delete(existing, id)
stale++
}
}
var missing []Document
for _, d := range co.archive.Docs {
if ids, ok := existing[d.ID]; !ok || slices.ContainsFunc(ids, co.unknownDoc) {
missing = append(missing, d)
}
}
if len(missing) == 0 && stale == 0 {
log.Println("No new documents - related.json is up to date.")
return
}
log.Printf("Computing related docs for %d documents...", len(missing))
maps.Copy(existing, co.buildRelatedDocs(missing, co.archive.Docs))
if err := saveRelatedDocs(existing); err != nil {
log.Fatalf("Failed to save related docs: %v", err)
}
log.Printf("Saved related.json with %d entries (%d computed, %d stale dropped, other entries and manual edits preserved)",
len(existing), len(missing), stale)
return
}
handler := withBaseHeaders(newMux())
srv := &http.Server{
Addr: *listen,
Handler: handler,
ReadHeaderTimeout: 5 * time.Second,
ReadTimeout: 15 * time.Second,
WriteTimeout: writeTimeout, // downloads set their own: fileWriteBudget
IdleTimeout: 60 * time.Second,
MaxHeaderBytes: 64 << 10,
}
warnNonLoopback(*listen)
log.Printf("Archive Server listening on %s (data: %s, project: %s)", srv.Addr, dataDir, projectDir)
errCh := make(chan error, 1)
go func() { errCh <- srv.ListenAndServe() }()
sig := make(chan os.Signal, 1)
signal.Notify(sig, os.Interrupt, syscall.SIGTERM)
select {
case err := <-errCh:
log.Fatal(err)
case s := <-sig:
signal.Reset(os.Interrupt, syscall.SIGTERM)
log.Printf("%s received - draining connections (again to stop now)", s)
ctx, cancel := context.WithTimeout(context.Background(), writeTimeout)
defer cancel()
if err := srv.Shutdown(ctx); err != nil {
log.Printf("Shutdown: %v", err)
}
}
}
func (co *Corpus) prerenderPages() {
co.prerendered = map[string]template.HTML{}
co.docRowHTML = nil
if devMode {
return
}
exec := func(key, page, block string, data interface{}) {
t := tmplSet[page]
if t == nil {
return
}
var buf bytes.Buffer
if err := t.ExecuteTemplate(&buf, block, data); err != nil {
warnf("prerender %s/%s: %v - rendering it per request", page, block, err)
return
}
co.prerendered[key] = template.HTML(buf.String())
}
exec("ids", "ids.html", "id-rows", &idsPage{Rows: co.idRows})
exec("archive-timeline", "browse.html", "archive-timeline",
&browsePage{ArchiveYears: co.archiveTimeline})
exec("glossary", "glossary.html", "glossary-groups",
&glossaryPage{Groups: co.glossaryGroups})
exec("articles", "articles.html", "article-groups",
&articlesPage{Groups: co.articleGroups})
for key, v := range co.codewordViews {
exec("codewords:"+key, "codewords.html", "codeword-rows",
&codewordsPage{Rows: v.Rows})
}
for set, tables := range co.exploreTables {
for i, t := range tables {
exec(exploreKey(set, i), "explore_table.html", "explore-rows",
&exploreTablePage{Rows: t.Rows})
}
}
co.prerenderCategoryRows()
co.prerenderDocRows()
total := 0
for _, h := range co.prerendered {
total += len(h)
}
log.Printf("Pre-rendered %d page bodies (%d KB)", len(co.prerendered), total/1024)
}
func (co *Corpus) preHTML(key string) template.HTML { return co.prerendered[key] }
func newMux() *http.ServeMux {
mux := http.NewServeMux()
mux.HandleFunc("GET /{$}", bind((*Corpus).handleIndex))
mux.HandleFunc("GET /random", bind((*Corpus).handleRandom))
mux.HandleFunc("GET /glossary", bind((*Corpus).handleGlossary))
mux.HandleFunc("GET /preferences", handlePreferences)
mux.HandleFunc("POST /preferences/save", handlePreferencesSave)
mux.HandleFunc("GET /pinned", bind((*Corpus).handlePinnedPage))
mux.HandleFunc("GET /pinned/export", bind((*Corpus).handlePinnedExport))
mux.HandleFunc("POST /pin/{id}", bind((*Corpus).handlePinToggle))
mux.HandleFunc("GET /articles", bind((*Corpus).handleArticles))
mux.HandleFunc("GET /article/{key}", bind((*Corpus).handleArticleView))
mux.HandleFunc("GET /changelog", bind((*Corpus).handleChangelog))
mux.HandleFunc("POST /history/clear", handleHistoryClear)
mux.HandleFunc("POST /pinsearch", bind((*Corpus).handleSearchPinToggle))
mux.HandleFunc("GET /robots.txt", handleRobots)
mux.HandleFunc("GET /sitemap.xml", bind((*Corpus).handleSitemap))
mux.HandleFunc("/", notFound)
mux.HandleFunc("GET /browse", bind((*Corpus).handleBrowseRoot))
mux.HandleFunc("GET /browse/{category}", bind((*Corpus).handleCategoryList))
mux.HandleFunc("GET /explore", bind((*Corpus).handleExploreIndex))
mux.HandleFunc("GET /explore/{metric}", bind((*Corpus).handleExploreMetric))
mux.HandleFunc("GET /explore/terms", bind((*Corpus).handleCodewords))
mux.HandleFunc("GET /explore/ids", bind((*Corpus).handleExploreIDs))
mux.HandleFunc("GET /explore/ids.txt", bind((*Corpus).handleExploreIDsText))
mux.HandleFunc("GET /explore/classification", bind((*Corpus).handleClassPatterns))
mux.HandleFunc("GET /explore/portions", bind((*Corpus).handlePortions))
mux.HandleFunc("GET /explore/outlets", bind((*Corpus).handleOutlets))
mux.HandleFunc("GET /explore/assemblies", bind((*Corpus).handleAssemblies))
mux.HandleFunc("GET /explore/assemblies/{id}", bind((*Corpus).handleAssemblyView))
mux.HandleFunc("GET /browse/{category}/{value}", bind((*Corpus).handleCategoryFilter))
mux.HandleFunc("GET /doc/{id}", bind((*Corpus).handleDocView))
mux.HandleFunc("GET /doc/{id}/pages", bind((*Corpus).handleDocPages))
mux.HandleFunc("GET /doc/{id}/versions/{slug}", bind((*Corpus).handleVersionView))
mux.HandleFunc("GET /doc/{id}/versions/{slug}/pages", bind((*Corpus).handleVersionPages))
mux.HandleFunc("GET /search", bind((*Corpus).handleSearch))
mux.HandleFunc("GET /about", handleAbout)
mux.HandleFunc("GET /files/", bind((*Corpus).handleFiles))
mux.HandleFunc("GET /static/", handleStatic)
// JSON API (read-only) - see api.go for endpoint documentation.
if apiEnabled {
mux.HandleFunc("GET /api/docs", bind((*Corpus).handleAPIDocs))
mux.HandleFunc("GET /api/doc/{id}", bind((*Corpus).handleAPIDoc))
mux.HandleFunc("GET /api/search", bind((*Corpus).handleAPISearch))
mux.HandleFunc("GET /api/facets", bind((*Corpus).handleAPIFacets))
mux.HandleFunc("/api/", func(w http.ResponseWriter, r *http.Request) {
apiError(w, http.StatusNotFound, "unknown API endpoint")
})
} else {
mux.HandleFunc("/api/", func(w http.ResponseWriter, r *http.Request) {
apiError(w, http.StatusForbidden, "the JSON API is disabled on this instance")
})
}
return mux
}
// withBaseHeaders wraps a handler with the headers every response carries.
func withBaseHeaders(next http.Handler) http.Handler {
const csp = "default-src 'self'; img-src 'self'; script-src 'self'; " +
"style-src 'self' 'unsafe-inline'; object-src 'none'; base-uri 'none'; " +
"form-action 'self'; frame-ancestors 'self'"
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
h := w.Header()
h.Set("Content-Security-Policy", csp)
h.Set("X-Frame-Options", "SAMEORIGIN")
h.Set("X-Content-Type-Options", "nosniff")
h.Set("Referrer-Policy", "no-referrer")
switch r.Method {
case http.MethodGet, http.MethodHead, http.MethodPost:
default:
h.Set("Allow", "GET, HEAD, POST")
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
next.ServeHTTP(w, r)
})
}
func envOr(key, fallback string) string {
if v := os.Getenv(key); v != "" {
return v
}
return fallback
}
func envBool(key string) bool {
v := os.Getenv(key)
if v == "" {
return false
}
b, err := strconv.ParseBool(v)
if err != nil {
log.Fatalf("$%s=%q is not a boolean (use 1/0 or true/false)", key, v)
}
return b
}
func envDuration(key string) time.Duration {
v := os.Getenv(key)
if v == "" {
return 0
}
d, err := time.ParseDuration(v)
if err != nil {
log.Fatalf("$%s=%q is not a duration (use e.g. 90s, 10m)", key, v)
}
return d
}
// accelPath is the X-Accel-Redirect target for a file: its path relative to
// the data directory under accelPrefix, percent-encoded because nginx decodes
// the header as a URI.
func accelPath(file string) (string, bool) {
rel, err := filepath.Rel(dataDir, file)
if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
return "", false
}
u := url.URL{Path: accelPrefix + filepath.ToSlash(rel)}
return u.EscapedPath(), true
}
// handleFiles serves the files the loader indexed for a document, and no
// other file its directory may hold (Document.servedFile).
func (co *Corpus) handleFiles(w http.ResponseWriter, r *http.Request) {
rest := strings.TrimPrefix(r.URL.Path, "/files/")
id, sub, ok := strings.Cut(rest, "/")
if !ok || sub == "" {
http.NotFound(w, r)
return
}
doc := co.docIndex[id]
if doc == nil {
http.NotFound(w, r)
return
}
sub = path.Clean("/" + sub)
rel := strings.TrimPrefix(sub, "/")
if strings.Contains(sub, "..") || !doc.servedFile(rel) {
http.NotFound(w, r)
return
}
cacheIfFile(w, os.DirFS(doc.Dir), rel, "public, max-age=86400")
if isTranscript(rel) {
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
}
file := filepath.Join(doc.Dir, filepath.FromSlash(sub))
if accelPrefix != "" {
if target, ok := accelPath(file); ok {
w.Header().Set("X-Accel-Redirect", target)
return
}
}
_ = http.NewResponseController(w).SetWriteDeadline(time.Now().Add(fileWriteBudget))
http.ServeFile(w, r, file)
}
const writeTimeout = 30 * time.Second
// fileWriteBudget is how long a single download may take in total. A 123 MB
// PDF over a bad mobile link (~35 KB/s) fits in an hour, so in practice this
// bound is only reached by a reader that has gone away.
var fileWriteBudget = time.Hour
func cacheIfFile(w http.ResponseWriter, fsys fs.FS, name, policy string) {
if fi, err := fs.Stat(fsys, name); err == nil && fi.Mode().IsRegular() {
w.Header().Set("Cache-Control", policy)
}
}
func handleStatic(w http.ResponseWriter, r *http.Request) {
if path.Ext(r.URL.Path) == "" {
http.NotFound(w, r)
return
}
rel := strings.TrimPrefix(path.Clean("/"+strings.TrimPrefix(r.URL.Path, "/static/")), "/")
if h, ok := staticHashes[rel]; ok {
if r.URL.Query().Get("v") == h {
w.Header().Set("Cache-Control", "public, max-age=31536000, immutable")
}
w.Header().Set("ETag", `"`+h+`"`)
}
http.StripPrefix("/static/", http.FileServerFS(staticFS)).ServeHTTP(w, r)
}
func navSection(path string) string {
switch {
case path == "/":
return "home"
case path == "/browse" || strings.HasPrefix(path, "/browse/"):
return "browse"
case path == "/search":
return "search"
case path == "/articles" || strings.HasPrefix(path, "/article/"):
return "articles"
case path == "/glossary":
return "glossary"
case path == "/about":
return "about"
case path == "/pinned":
return "pinned"
case path == "/preferences":
return "preferences"
}
return ""
}
func render(w http.ResponseWriter, r *http.Request, data pageData) {
renderCode(w, r, data, http.StatusOK)
}
func crossSitePOST(r *http.Request) bool {
switch r.Header.Get("Sec-Fetch-Site") {
case "same-origin", "none":
return false
case "cross-site", "same-site":
return true
}
if o := r.Header.Get("Origin"); o != "" && o != "null" {
if u, err := url.Parse(o); err == nil && !originIsHost(u, r.Host) {
return true
}
}
return false
}
func originIsHost(origin *url.URL, host string) bool {
def := map[string]string{"http": "80", "https": "443"}[strings.ToLower(origin.Scheme)]
op := origin.Port()
if op == "" {
op = def
}
h, hp, err := net.SplitHostPort(host)
if err != nil { // no port
h, hp = strings.Trim(host, "[]"), def
}
return strings.EqualFold(origin.Hostname(), h) && op == hp
}
func noStore(w http.ResponseWriter) {
w.Header().Set("Cache-Control", "private, no-store")
}
func readPOSTForm(w http.ResponseWriter, r *http.Request) bool {
if crossSitePOST(r) {
http.Error(w, "cross-site request rejected", http.StatusForbidden)
return false
}
r.Body = http.MaxBytesReader(w, r.Body, 1<<16)
if err := r.ParseForm(); err != nil {
http.Error(w, "bad form", http.StatusBadRequest)
return false
}
return true
}
func crossSiteFetch(r *http.Request) bool {
switch r.Header.Get("Sec-Fetch-Site") {
case "cross-site", "same-site":
return true
}
return false
}
func renderCode(w http.ResponseWriter, r *http.Request, data pageData, code int) {
name := data.templateName()
if w.Header().Get("Cache-Control") == "" {
if (r.Method == http.MethodGet || r.Method == http.MethodHead) &&
(code == http.StatusOK || code == http.StatusNotFound) &&
r.Header.Get("Cookie") == "" {
w.Header().Set("Cache-Control", "public, max-age=10")
w.Header().Set("Vary", "Cookie")
if accelExpires != "" {
w.Header().Set("X-Accel-Expires", accelExpires)
}
} else {
w.Header().Set("Cache-Control", "private, no-cache")
}
}
b := data.base()
b.Nav = navSection(r.URL.Path)
b.APIEnabled = apiEnabled
// A handler that read the reader's preferences passes them on
// (pageWith); they are only read here for one that had no use for them.
if !b.hasPrefs {
b.Prefs = readPrefs(r)
}
b.Theme = b.Prefs.ThemeAttr()
b.CardPagePreview = b.Prefs.CardPagePreview
var t *template.Template
var err error
if devMode {
t, err = parsePage(name)
if err != nil {
log.Printf("Template Parse Error (%s): %v", name, err)
serverError(w, err)
return
}
} else {
t = tmplSet[name]
if t == nil {
log.Printf("Template not found: %s", name)
serverError(w, fmt.Errorf("template not found: %s", name))
return
}
}
var buf bytes.Buffer
if err := t.ExecuteTemplate(&buf, name, data); err != nil {
log.Printf("Template Exec Error (%s): %v", name, err)
serverError(w, err)
return
}
if code != http.StatusOK {
w.WriteHeader(code)
}
w.Write(bytes.TrimSpace(buf.Bytes()))
w.Write([]byte("\n"))
}
func notFound(w http.ResponseWriter, r *http.Request) {
renderCode(w, r, &notFoundPage{Path: r.URL.Path}, http.StatusNotFound)
}
func parsePage(name string) (*template.Template, error) {
return template.New(name).Funcs(funcMap).ParseFS(templateFS, "base.html", "partials.html", name)
}
func parseTemplates() error {
pages, err := fs.Glob(templateFS, "*.html")
if err != nil {
return err
}
tmplSet = make(map[string]*template.Template)
for _, name := range pages {
if name == "base.html" || name == "partials.html" {
continue
}
t, err := parsePage(name)
if err != nil {
return fmt.Errorf("%s: %w", name, err)
}
tmplSet[name] = t
}
return nil
}
func serverError(w http.ResponseWriter, err error) {
w.Header().Set("Cache-Control", "no-store")
if devMode {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
}
func warnNonLoopback(addr string) {
host, _, err := net.SplitHostPort(addr)
if err != nil {
return
}
if host == "localhost" {
return
}
if ip := net.ParseIP(host); ip != nil && ip.IsLoopback() {
return
}
warnf("-listen %s is not loopback - anything that can route to this host reaches the archive directly, bypassing your proxy's caching, rate limits and logs.", addr)
}

591
mdparser.go Normal file
View file

@ -0,0 +1,591 @@
// mdparser.go - minimal dependency-free Markdown parser.
//
// Input trust: everything rendered through this parser is the archive's own
// curator-produced OCR never user-supplied input.
package main
import (
"fmt"
"html"
"regexp"
"strconv"
"strings"
"unicode"
"unicode/utf8"
)
// -- Inline patterns ----------------------------------------------------
var (
reInlineCode = regexp.MustCompile("`(.+?)`")
reStrikethrough = regexp.MustCompile(`~~(.+?)~~`)
reHardBreak = regexp.MustCompile(`(?i)&lt;br\s*/?&gt;`)
reEscape = regexp.MustCompile("\\\\([!-/:-@\\[-`|{}~])")
reOrderedItem = regexp.MustCompile(`^(\s*)(\d{1,9})\.(?:\s+(.*))?\s*$`)
reUnorderedItem = regexp.MustCompile(`^(\s*)[-*+](?:\s+(.*))?\s*$`)
reTableSep = regexp.MustCompile(`^\|[-| :]+\|$`)
)
// parseInline converts inline Markdown to HTML.
func parseInline(s string) string {
s = strings.ReplaceAll(s, "\x00", "")
s = strings.ReplaceAll(s, "\x01", "")
s = html.EscapeString(s)
// Backslash escapes (CommonMark): "\*" is a literal "*", not emphasis
// syntax. Pull escaped ASCII punctuation out to placeholders BEFORE the
// emphasis passes so the character can't be interpreted as a marker,
// then restore it at the end.
var escaped []string
if strings.IndexByte(s, '\\') >= 0 {
s = reEscape.ReplaceAllStringFunc(s, func(m string) string {
// m is "\X": EscapeString leaves the backslash alone. Keep
// the char after it.
ch := m[len(m)-1:]
escaped = append(escaped, ch)
return "\x01" + strconv.Itoa(len(escaped)-1) + "\x01"
})
}
// Extract inline code spans next and replace them with placeholders so
// the strikethrough/bold/italic passes can't reformat code content
// (e.g. `*x*` must render as literal *x*, not <em>x</em>). Escape
// placeholders that landed INSIDE a code span are put back as their
// original "\X" source text first: backslash escapes are not escapes
// within code spans (CommonMark), so `\*` renders as \*, not *. The
// escape pass still runs before this one on purpose - that ordering is
// what lets \` be a literal backtick that cannot OPEN a code span.
var codes []string
if strings.IndexByte(s, '`') >= 0 {
s = reInlineCode.ReplaceAllStringFunc(s, func(m string) string {
inner := reInlineCode.FindStringSubmatch(m)[1]
for i, ch := range escaped {
inner = strings.ReplaceAll(inner, "\x01"+strconv.Itoa(i)+"\x01", "\\"+ch)
}
codes = append(codes, "<code>"+inner+"</code>")
return "\x00" + strconv.Itoa(len(codes)-1) + "\x00"
})
}
// After the code-span pass, so `<br>` inside backticks stays literal.
if strings.Contains(s, "&lt;") {
s = reHardBreak.ReplaceAllString(s, "<br>")
}
if strings.Contains(s, "~~") {
s = reStrikethrough.ReplaceAllString(s, `<del>$1</del>`)
}
// ***x*** first, so it nests as <em><strong> rather than crossing.
s = emphasize(s, '*', 3, "<em><strong>", "</strong></em>")
s = emphasize(s, '*', 2, "<strong>", "</strong>")
s = emphasize(s, '_', 2, "<strong>", "</strong>")
s = emphasize(s, '*', 1, "<em>", "</em>")
s = emphasize(s, '_', 1, "<em>", "</em>")
// Restore code spans.
for i, code := range codes {
s = strings.Replace(s, "\x00"+strconv.Itoa(i)+"\x00", code, 1)
}
// Restore backslash-escaped characters as their literal selves.
// (Placeholders that sat inside a code span were already restored to
// their "\X" source form up in the code-extraction pass and no longer
// occur here; Replace on an absent placeholder is a no-op.)
for i, ch := range escaped {
s = strings.Replace(s, "\x01"+strconv.Itoa(i)+"\x01", ch, 1)
}
return s
}
// emphasize wraps the text between two runs of k delimiters c ('*' or '_')
// in open/close. It pairs them as CommonMark does: left to right, a run
// that can close gives its first k delimiters to the nearest unpaired
// opener before it, and a run that can open offers its last k to a later
// closer. Which runs can open or close follows CommonMark's flanking rules
// (delimiterRun).
func emphasize(s string, c byte, k int, open, close string) string {
var openers []int // offsets of unpaired openers, nearest last
tags := map[int]string{} // offset of k paired delimiters -> their tag
for i := 0; i < len(s); {
if s[i] != c {
i++
continue
}
end := i
for end < len(s) && s[end] == c {
end++
}
canOpen, canClose := delimiterRun(s, i, end, c)
n := end - i
if n >= k && canClose && len(openers) > 0 {
tags[openers[len(openers)-1]], tags[i] = open, close
openers = openers[:len(openers)-1]
n -= k
}
if n >= k && canOpen {
openers = append(openers, end-k)
}
i = end
}
if len(tags) == 0 {
return s
}
var b strings.Builder
for i := 0; i < len(s); i++ {
if t, ok := tags[i]; ok {
b.WriteString(t)
i += k - 1
} else {
b.WriteByte(s[i])
}
}
return b.String()
}
// delimiterRun reports whether the run of delimiters c at s[start:end]
// can open and can close emphasis, by CommonMark's flanking rules, which
// look at the whole run.
func delimiterRun(s string, start, end int, c byte) (canOpen, canClose bool) {
prev, next := ' ', ' '
if start > 0 {
prev, _ = utf8.DecodeLastRuneInString(s[:start])
}
if end < len(s) {
next, _ = utf8.DecodeRuneInString(s[end:])
}
left := !unicode.IsSpace(next) &&
(!isPunct(next) || unicode.IsSpace(prev) || isPunct(prev))
right := !unicode.IsSpace(prev) &&
(!isPunct(prev) || unicode.IsSpace(next) || isPunct(next))
if c == '_' {
return left && (!right || isPunct(prev)), right && (!left || isPunct(next))
}
return left, right
}
// isPunct is CommonMark's punctuation (Unicode P and S). The \x00/\x01
// placeholders count too: they stand for a code span (opened by a
// backtick) or a backslash-escaped punctuation character.
func isPunct(r rune) bool {
return r == 0 || r == 1 || unicode.IsPunct(r) || unicode.IsSymbol(r)
}
// -- Block-level parser --------------------------------------------
type parser struct {
lines []string
pos int
out strings.Builder
open []listLevel // the lists parseList is inside, outermost first
// numbered holds the last number of the ordered list at each indent,
// for as long as list lines keep coming (see endsItem).
numbered map[int]int
// top is the source's highest heading level and last the level of the
// heading emitted last (see parseHeading).
top, last int
}
// listLevel is one open list: the indent of its items and their kind.
type listLevel struct {
indent int
ordered bool
}
func newParser(src string) *parser {
src = strings.ReplaceAll(src, "\r\n", "\n")
src = strings.ReplaceAll(src, "\r", "\n")
p := &parser{lines: strings.Split(src, "\n"), top: 7, last: 1}
fence := ""
for _, line := range p.lines {
switch {
case fence != "":
if strings.HasPrefix(line, fence) {
fence = ""
}
case isFencedCodeStart(line):
fence = line[:3]
case isHeading(line):
p.top = min(p.top, headingLevel(line))
}
}
return p
}
func (p *parser) peek() (string, bool) {
if p.pos >= len(p.lines) {
return "", false
}
return p.lines[p.pos], true
}
func (p *parser) next() (string, bool) {
l, ok := p.peek()
if ok {
p.pos++
}
return l, ok
}
func (p *parser) emit(s string) { p.out.WriteString(s) }
// parse processes all lines and returns HTML.
func (p *parser) parse() string {
for p.pos < len(p.lines) {
line, _ := p.peek()
if !isListItem(line) {
p.numbered = nil // the run of list lines is over
}
switch {
case isFencedCodeStart(line):
p.parseFencedCode()
case isHR(line):
p.next()
p.emit("<hr>\n")
case isHeading(line):
p.parseHeading()
case isTableRow(line):
p.parseTable()
case isListItem(line):
p.parseList(reOrderedItem.MatchString(line), indentOf(line))
case strings.TrimSpace(line) == "":
p.next()
default:
p.parseParagraph()
}
}
return p.out.String()
}
// -- Heading --------------------------------------------------
// headingLevel returns the heading level of an ATX heading line - one to
// six '#' followed by whitespace or the end of the line - or 0 when the
// line is not a heading.
func headingLevel(s string) int {
n := 0
for n < len(s) && s[n] == '#' {
n++
}
if n == 0 || n > 6 || (n < len(s) && s[n] != ' ' && s[n] != '\t') {
return 0
}
return n
}
func isHeading(s string) bool { return headingLevel(s) > 0 }
// parseHeading emits a heading under the page's own <h1>: the source's top
// level becomes <h2> and no heading skips a level. The md-hN class keeps the
// size of the level the source wrote.
func (p *parser) parseHeading() {
line, _ := p.next()
level := headingLevel(line)
content := strings.TrimSpace(line[level:])
tag := min(level-p.top+2, p.last+1, 6)
p.last = tag
p.emit(fmt.Sprintf("<h%d class=\"md-h%d\">%s</h%d>\n", tag, level, parseInline(content), tag))
}
// -- Horizontal rule -------------------------------------------------
// isHR reports whether s is a thematic break: three or more of the same
// '-', '*' or '_', optionally with spaces or tabs between them.
func isHR(s string) bool {
t := strings.TrimSpace(s)
if t == "" || (t[0] != '-' && t[0] != '*' && t[0] != '_') {
return false
}
n := 0
for i := 0; i < len(t); i++ {
switch t[i] {
case t[0]:
n++
case ' ', '\t':
default:
return false
}
}
return n >= 3
}
// Fenced code block ---------------------------------------------------------
func isFencedCodeStart(s string) bool {
return strings.HasPrefix(s, "```") || strings.HasPrefix(s, "~~~")
}
func (p *parser) parseFencedCode() {
opener, _ := p.next()
fence := opener[:3]
var body strings.Builder
for {
line, ok := p.next()
if !ok || strings.HasPrefix(line, fence) {
break
}
body.WriteString(html.EscapeString(line))
body.WriteByte('\n')
}
p.emit("<pre><code>" + body.String() + "</code></pre>\n")
}
// -- Lists (unified, handles nesting) -----------------------------------------
// indentOf returns the number of leading whitespace characters in s.
func indentOf(s string) int {
return len(s) - len(strings.TrimLeft(s, " \t"))
}
// isListItem returns true if the line is a ul or ol item at any indent.
// A thematic break wins over a bullet ("* * *" is a rule, not an item).
func isListItem(s string) bool {
// A list marker is the first non-space byte: rule out the rest (most
// lines) before any regex runs.
t := strings.TrimLeft(s, "\t\n\f\r ")
if t == "" || (t[0] != '-' && t[0] != '*' && t[0] != '+' && (t[0] < '0' || t[0] > '9')) {
return false
}
return !isHR(s) && (reUnorderedItem.MatchString(s) || reOrderedItem.MatchString(s))
}
// itemContent strips the list marker and returns the item text (possibly
// empty for a bare marker like "6.").
func itemContent(s string) string {
if sub := reUnorderedItem.FindStringSubmatch(s); sub != nil {
return sub[2]
}
if sub := reOrderedItem.FindStringSubmatch(s); sub != nil {
return sub[3]
}
return strings.TrimSpace(s)
}
// orderedNum returns the source number of an ordered-list line ("7." -> 7)
// and true, or false if the line isn't an ordered item. "0." is a real
// number, so it cannot double as the "not ordered" answer.
func orderedNum(s string) (int, bool) {
if sub := reOrderedItem.FindStringSubmatch(s); sub != nil {
n, _ := strconv.Atoi(sub[2]) // at most nine digits: cannot fail
return n, true
}
return 0, false
}
// startsList reports whether list-item line s may open a new list on the
// line after text, where it would otherwise continue that text.
func startsList(s string) bool {
if itemContent(s) == "" {
return false
}
n, ok := orderedNum(s)
return !ok || n == 1
}
// endsItem reports whether line s, met while collecting the lazily
// continued text of a list item, ends that item rather than continuing it.
func (p *parser) endsItem(s string) bool {
if isHR(s) || isHeading(s) || isFencedCodeStart(s) || isTableRow(s) {
return true
}
if !isListItem(s) {
return false
}
if startsList(s) {
return true
}
indent, ordered := indentOf(s), reOrderedItem.MatchString(s)
for _, l := range p.open {
if l.indent == indent && l.ordered == ordered {
return true
}
}
if n, ok := orderedNum(s); ok {
for _, last := range p.numbered {
if n > last && n <= last+2 {
return true
}
}
}
return false
}
// parseList recursively parses a ul or ol at the given indent level.
func (p *parser) parseList(ordered bool, minIndent int) {
tag := "ul"
if ordered {
tag = "ol"
}
p.emit("<" + tag + ">\n")
p.open = append(p.open, listLevel{minIndent, ordered})
defer func() { p.open = p.open[:len(p.open)-1] }()
expected := 0 // running ordered-list counter, for value="N" decisions
for {
line, ok := p.peek()
if !ok || strings.TrimSpace(line) == "" || !isListItem(line) {
break
}
indent := indentOf(line)
if indent < minIndent {
// Belongs to a parent list - stop.
break
}
if indent == minIndent && reOrderedItem.MatchString(line) != ordered {
// Bullets after numbers, or numbers after bullets: a new
// list (as in CommonMark). Kept in this one, a number would
// lose itself to a bullet and a bullet would gain a number.
break
}
if indent > minIndent {
// Deeper indent mid-list without a parent <li> - treat as nested.
p.parseList(reOrderedItem.MatchString(line), indent)
continue
}
p.next()
// Item body: the marker line plus any lazily-continued lines after it.
var body []string
if c := itemContent(line); c != "" {
body = append(body, c)
}
for {
cont, ok := p.peek()
if !ok || strings.TrimSpace(cont) == "" || p.endsItem(cont) {
break
}
p.next()
body = append(body, strings.TrimSpace(cont))
}
liOpen := "<li>"
if ordered {
n, _ := orderedNum(line)
expected++
if n != expected {
liOpen = "<li value=\"" + strconv.Itoa(n) + "\">"
expected = n
}
if p.numbered == nil {
p.numbered = map[int]int{}
}
p.numbered[minIndent] = n
}
var parts []string
for _, l := range body {
parts = append(parts, parseInline(l))
}
text := strings.Join(parts, "<br>\n")
// Peek: if next line is deeper, nest a list inside this <li> -
// as many as follow, since bullets turning into numbers down
// there start a second nested list, which belongs here too.
next, hasNext := p.peek()
if hasNext && isListItem(next) && indentOf(next) > indent {
p.emit(liOpen + text + "\n")
for hasNext && isListItem(next) && indentOf(next) > indent {
p.parseList(reOrderedItem.MatchString(next), indentOf(next))
next, hasNext = p.peek()
}
p.emit("</li>\n")
} else {
p.emit(liOpen + text + "</li>\n")
}
}
p.emit("</" + tag + ">\n")
}
// -- Table ---------------------------------------------------------------------
func isTableRow(s string) bool {
t := strings.TrimSpace(s)
return len(t) >= 3 && t[0] == '|' && t[len(t)-1] == '|' && !strings.Contains(t, "\n")
}
// splitCells splits a pipe-delimited row into trimmed cell strings.
func splitCells(row string) []string {
row = strings.TrimPrefix(strings.TrimSpace(row), "|")
if strings.HasSuffix(row, "|") && !strings.HasSuffix(row, `\|`) {
row = row[:len(row)-1]
}
var cells []string
var cell strings.Builder
for i := 0; i < len(row); i++ {
switch {
case row[i] == '\\' && i+1 < len(row) && row[i+1] == '|':
cell.WriteByte('|')
i++
case row[i] == '|':
cells = append(cells, strings.TrimSpace(cell.String()))
cell.Reset()
default:
cell.WriteByte(row[i])
}
}
return append(cells, strings.TrimSpace(cell.String()))
}
func (p *parser) parseTable() {
headerLine, _ := p.next()
headers := splitCells(headerLine)
// Consume separator row (|---|---|) if present.
if sep, ok := p.peek(); ok && reTableSep.MatchString(strings.TrimSpace(sep)) {
p.next()
}
p.emit("<div class=\"table-scroll\">\n<table>\n<thead>\n<tr>\n")
for _, h := range headers {
p.emit("<th>" + parseInline(h) + "</th>\n")
}
p.emit("</tr>\n</thead>\n<tbody>\n")
for {
line, ok := p.peek()
if !ok || !isTableRow(line) {
break
}
p.next()
p.emit("<tr>\n")
for _, cell := range splitCells(line) {
p.emit("<td>" + parseInline(cell) + "</td>\n")
}
p.emit("</tr>\n")
}
p.emit("</tbody>\n</table>\n</div>\n")
}
// -- Paragraph -----------------------------------------------------------------
func (p *parser) parseParagraph() {
var lines []string
for {
line, ok := p.peek()
if !ok || strings.TrimSpace(line) == "" {
break
}
if isHeading(line) || isFencedCodeStart(line) || isHR(line) ||
(isListItem(line) && startsList(line)) || isTableRow(line) {
break
}
p.next()
lines = append(lines, strings.TrimSpace(line))
}
if len(lines) > 0 {
var parts []string
for _, l := range lines {
parts = append(parts, parseInline(l))
}
p.emit("<p>" + strings.Join(parts, "<br>\n") + "</p>\n")
}
}
// Render converts Markdown src to an HTML string.
func Render(src string) string {
return newParser(src).parse()
}

132
outlets.go Normal file
View file

@ -0,0 +1,132 @@
// outlets.go - /explore/outlets: publication activity per media outlet and
// year.
package main
import (
"net/http"
"sort"
)
type outletCell struct {
Docs int
Articles int
URL string
}
type outletRow struct {
Name string
URL string
Total int
Cells []outletCell
}
type outletTable struct {
Years []string
YearLinks []facetEntry
Rows []outletRow
Totals []int
}
func (co *Corpus) buildOutletTables() {
co.outletTables = map[string]outletTable{
"": co.buildOutletTable(""),
collectionNews: co.buildOutletTable(collectionNews),
}
}
func (co *Corpus) buildOutletTable(coll string) outletTable {
type key struct{ outlet, year string }
docs := map[key]int{}
stories := map[key]int{}
totals := map[string]int{}
yearSet := map[string]bool{}
display := map[string]string{}
canon := func(name string) string {
k := valueKey(name)
if d, ok := display[k]; ok {
return d
}
display[k] = name
return name
}
for i := range co.archive.Docs {
d := &co.archive.Docs[i]
if coll != "" && d.Collection != coll {
continue
}
for _, pub := range facetValues(d, "PU") {
totals[canon(pub)]++
}
// A document counts for a publisher in the years of THAT publisher's
// own entries: another outlet's earlier publication does not make it
// this outlet's document of that year (publishedBy, the cell's page).
counted := map[key]bool{}
for _, p := range d.Publication {
if p.ReportingOnly || p.Publisher == "" || len(p.DatePublished) < 4 {
continue
}
k := key{canon(p.Publisher), p.DatePublished[:4]}
if !counted[k] {
counted[k] = true
docs[k]++
yearSet[k.year] = true
}
}
}
for i := range co.articles {
a := &co.articles[i]
if a.Title == "" || a.Publisher == "" || len(a.Date) < 4 {
continue
}
// A story counts in a collection's table when one of its documents
// is in that collection.
if coll != "" && !a.inCollection(co, coll) {
continue
}
stories[key{canon(a.Publisher), a.Date[:4]}]++
}
years := make([]string, 0, len(yearSet))
for y := range yearSet {
years = append(years, y)
}
sort.Strings(years)
names := rankCounts(totals, -1)
scope := wholeArchive
if coll != "" {
scope = scopeOf(map[string]bool{coll: true})
}
t := outletTable{Years: years, Totals: make([]int, len(years))}
for _, y := range years {
t.YearLinks = append(t.YearLinks, facetEntry{Name: y, URL: scope.link(categoryValueURL("published", y))})
}
for _, name := range names {
row := outletRow{Name: name, URL: scope.link(categoryValueURL("publisher", name)), Total: totals[name]}
for yi, y := range years {
k := key{name, y}
cell := outletCell{Docs: docs[k], Articles: stories[k]}
if cell.Docs > 0 {
cell.URL = scope.link(publisherYearURL(name, y))
}
t.Totals[yi] += cell.Docs
row.Cells = append(row.Cells, cell)
}
t.Rows = append(t.Rows, row)
}
return t
}
func (co *Corpus) handleOutlets(w http.ResponseWriter, r *http.Request) {
coll := ""
if r.URL.Query().Get("coll") == collectionNews {
coll = collectionNews
}
render(w, r, &outletsPage{
Table: co.outletTables[coll],
NewsOnly: coll == collectionNews,
})
}

393
pages.go Normal file
View file

@ -0,0 +1,393 @@
package main
import (
"html/template"
"net/url"
)
// pageData is a page struct, it embeds Base and names its template.
type pageData interface {
base() *Base
templateName() string
}
// Base is what base.html reads. renderCode fills in everything but the
// category boxes.
type Base struct {
Nav string
APIEnabled bool
Theme string
CardPagePreview bool
Prefs Prefs
hasPrefs bool
ShowCatBoxes bool
CatCounts map[string]int
ActiveCategory string
}
func (b *Base) base() *Base { return b }
func pageWith(prefs Prefs) Base { return Base{Prefs: prefs, hasPrefs: true} }
type cardView struct {
Docs []*Document
ShowThumbs bool
CardPagePreview bool
HideClassification bool
HideJournalists bool
HidePeople bool
HidePrograms bool
HideSummary bool
FOIAQS string // see Prefs.foiaLinkQS
ShowUnpin bool
ActiveCategory string
ActiveValue string
MarkReported bool // the "reported" badge on individually reported SIDtoday issues
}
// listView is a page of documents as cards or as a table
type listView struct {
Docs []*Document
Total int
Page int
TotalPages int
View string
Sort string
Order string
DefaultSort string
BaseURL string
SortURL string
TableHeads []sortHead
RowsHTML template.HTML // the table view's rows, pre-rendered
Cards cardView
// MarkReported badges the individually reported SIDtoday issues, which
// only the SIDtoday tab tells apart from the bulk release.
MarkReported bool
}
// docTableRow is a row of the document table
type docTableRow struct {
*Document
MarkReported bool
}
func (lv listView) TableRows() []docTableRow {
rows := make([]docTableRow, len(lv.Docs))
for i, d := range lv.Docs {
rows[i] = docTableRow{d, lv.MarkReported}
}
return rows
}
type viewer struct {
Images []string
Sizes []imageSize
ImageBase string
Content template.HTML
Version *DocVersion
}
func (v viewer) ImageSize(i int) template.HTMLAttr {
if i < len(v.Sizes) {
return v.Sizes[i].Attr()
}
return ""
}
type notFoundPage struct {
Base
Path string
}
type aboutPage struct{ Base }
type indexPage struct {
Base
homeFigures
TrackRecent bool
RecentDocs []*Document
SnowdenTotal int
ArchiveTotal int
FOIATotal int
IncludeFOIA bool
ReportedTotal int
WithSIDtoday bool // the figures include the SIDtoday archive (/?sidtoday=1)
}
// browsePage is /browse
type browsePage struct {
Base
listView
Collection string
CollectionCounts map[string]int
SIDTodayCount int
SelectedTypes map[string]bool
AllTypes map[string]int
Params url.Values
SeriesList []categoryEntry
ArchiveYears []archiveYear
TimelineHTML template.HTML
}
// categoryPage is /browse/{category}
type categoryPage struct {
Base
Title string
Blurb string
Heads []sortHead
RowsHTML template.HTML
Entries []categoryEntry
CollCols bool
ShowCollCols bool
ShowAZ bool
Letters []string
SeeAlso []seeAlsoLink
}
// listPage is /browse/{category}/{value}
type listPage struct {
Base
listView
Title string
CatLabel string
Blurb string
SearchLink string
SelectedTypes map[string]bool
AllTypes map[string]int
Params url.Values
GlossaryEntry *GlossaryEntry
Refinements []searchFacet
Breakdowns []bdGroup
Timeline []timelineBar
PubTimeline []timelineBar
YearScope string
YearScopeAllURL string
MonthNav []monthNavEntry
MonthNavYear string
MonthNavYearURL string
MonthNavYearActive bool
MonthlessCount int
}
type docPage struct {
Base
viewer
Doc *Document
BrowseQS string
OGImageURL string
DocAbsURL string
Embeds docEmbeds
SeriesGroup *seriesGroup
Assemblies []*Assembly
SerialPrev *Document
SerialNext *Document
PublishedTogether []*Document
PublishedShown int
PublishedMore []*Article
PublishedCards cardView
RelatedDocs []*Document
RelatedCards cardView
Pinned bool
}
type versionPage struct {
Base
viewer
Doc *Document
Publications []Publication
PDFURL string
}
// pagesPage is the page-image grid of a document or of one of its versions.
type pagesPage struct {
Base
viewer
Doc *Document
TextOnly bool
}
type articlesPage struct {
Base
Groups []articleYearGroup
GroupsHTML template.HTML
Total int
Books int
}
type articlePage struct {
Base
Article Article
Total int
BrowseQS string
PublisherURL string
Cards cardView
}
type assembliesPage struct {
Base
Assemblies []*Assembly
TextOnly bool
}
type assemblyPage struct {
Base
Assembly *Assembly
TextOnly bool
}
type changelogPage struct {
Base
Entries []ChangeEntry
Total int
}
type codewordsPage struct {
Base
Rows []codewordRow
RowsHTML template.HTML
Total int
Shown int
Heads []sortHead
}
type explorePage struct {
Base
Metrics []exploreMetric
HasAssemblies bool
}
type exploreTablePage struct {
Base
Metric *exploreMetric
Rows []exploreRow
RowsHTML template.HTML
Total int
Shown int
Heads []sortHead
}
type classPatternsPage struct {
Base
Cols []facetEntry
Levels []string
Strips []clsYearRow
Sparse []clsYearRow
MinMarks int
HeatRows []heatRow
}
type idsPage struct {
Base
Rows []idRow
RowsHTML template.HTML
Shown int
}
type glossaryPage struct {
Base
Groups []glossaryGroup
GroupsHTML template.HTML
Total int
}
type outletsPage struct {
Base
Table outletTable
NewsOnly bool
}
type portionsPage struct {
Base
Title string
Stats portionSummary
RowLimit int
}
type pinnedPage struct {
Base
Total int
Searches []recentSearch
Full bool // a pin was refused: the cookie is full
Cards cardView
}
type preferencesPage struct {
Base
MaxCards int
Saved bool
Reset bool
}
type searchPage struct {
Base
FieldOptions []searchField
Facets []searchFacet
Queries []string
Fields []string
Conjs []string
Results []searchResult
SearchTypeCounts map[string]int
SelTypeSet map[string]bool
FormParams url.Values
Savable bool
ShowThumbs bool
FOIAQS string
Total int
Page int
TotalPages int
RecentSearches []recentSearch
SelColl map[string]bool
CollFiltered bool
CanonicalQS string
SearchPinned bool
Searched bool
QueryLabel string
DateDocStart string
DateDocEnd string
DatePubStart string
DatePubEnd string
Sort string
Order string
SearchBaseURL string
DateHist []timelineBar
SnippetsCapped bool
SnippetLimit int
OverClauses int
ClauseLimit int
OverWords int
WordLimit int
}
func (*notFoundPage) templateName() string { return "404.html" }
func (*aboutPage) templateName() string { return "about.html" }
func (*indexPage) templateName() string { return "index.html" }
func (*browsePage) templateName() string { return "browse.html" }
func (*categoryPage) templateName() string { return "category.html" }
func (*listPage) templateName() string { return "list.html" }
func (*docPage) templateName() string { return "doc.html" }
func (*versionPage) templateName() string { return "version.html" }
func (*pagesPage) templateName() string { return "pages.html" }
func (*articlesPage) templateName() string { return "articles.html" }
func (*articlePage) templateName() string { return "article.html" }
func (*assembliesPage) templateName() string { return "assemblies.html" }
func (*assemblyPage) templateName() string { return "assembly.html" }
func (*changelogPage) templateName() string { return "changelog.html" }
func (*codewordsPage) templateName() string { return "codewords.html" }
func (*explorePage) templateName() string { return "explore.html" }
func (*exploreTablePage) templateName() string { return "explore_table.html" }
func (*classPatternsPage) templateName() string { return "classpatterns.html" }
func (*idsPage) templateName() string { return "ids.html" }
func (*glossaryPage) templateName() string { return "glossary.html" }
func (*outletsPage) templateName() string { return "outlets.html" }
func (*portionsPage) templateName() string { return "portions.html" }
func (*pinnedPage) templateName() string { return "pinned.html" }
func (*preferencesPage) templateName() string { return "preferences.html" }
func (*searchPage) templateName() string { return "search.html" }
var pageTypes = []pageData{
&notFoundPage{}, &aboutPage{}, &indexPage{}, &browsePage{}, &categoryPage{}, &listPage{},
&docPage{}, &versionPage{}, &pagesPage{}, &articlesPage{}, &articlePage{}, &assembliesPage{},
&assemblyPage{}, &changelogPage{}, &codewordsPage{}, &explorePage{}, &exploreTablePage{},
&classPatternsPage{}, &idsPage{}, &glossaryPage{}, &outletsPage{}, &portionsPage{},
&pinnedPage{}, &preferencesPage{}, &searchPage{},
}

145
pins.go Normal file
View file

@ -0,0 +1,145 @@
// pins.go - pinned documents (a reading list). Cookie-local.
package main
import (
"net/http"
"net/url"
"slices"
)
const (
pinnedCookie = "pinned_docs"
pinnedMax = 200
pinnedSearchesCookie = "pinned_searches"
pinnedSearchesMax = 30
)
func (co *Corpus) readPinned(r *http.Request) []string {
var out []string
for _, id := range readIDListCookie(r, pinnedCookie, pinnedMax) {
if co.docIndex[id] != nil {
out = append(out, id)
}
}
return out
}
func toggleEntry(list []string, e string, max int) []string {
var next []string
removed := false
for _, x := range list {
if x == e {
removed = true
continue
}
next = append(next, x)
}
if !removed && len(next) < max {
next = append(next, e)
}
return next
}
func isPinned(r *http.Request, id string) bool {
return slices.Contains(readIDListCookie(r, pinnedCookie, pinnedMax), id)
}
func (co *Corpus) handlePinToggle(w http.ResponseWriter, r *http.Request) {
if !readPOSTForm(w, r) {
return
}
id := r.PathValue("id")
if co.docIndex[id] == nil {
notFound(w, r)
return
}
pinned := co.readPinned(r)
stored := writeListCookie(w, pinnedCookie, toggleEntry(pinned, id, pinnedMax))
if pinRefused(pinned, stored, id) {
http.Redirect(w, r, "/pinned?full=1", http.StatusSeeOther)
return
}
back := safeBackPath(r.FormValue("back"))
if back == "/" {
back = docPath(id)
}
http.Redirect(w, r, back, http.StatusSeeOther)
}
func readPinnedSearchEntries(r *http.Request) []string {
return readIDListCookie(r, pinnedSearchesCookie, pinnedSearchesMax)
}
func isPinnedSearch(r *http.Request, canonicalQS string) bool {
return slices.Contains(readPinnedSearchEntries(r), url.QueryEscape(canonicalQS))
}
func (co *Corpus) handleSearchPinToggle(w http.ResponseWriter, r *http.Request) {
if !readPOSTForm(w, r) {
return
}
qs := r.FormValue("qs")
if qs == "" {
http.Error(w, "bad query", http.StatusBadRequest)
return
}
v, err := url.ParseQuery(qs)
if err != nil {
http.Error(w, "bad query", http.StatusBadRequest)
return
}
entry := url.QueryEscape(qs)
pinned := readPinnedSearchEntries(r)
if !co.savableSearch(v) && !slices.Contains(pinned, entry) {
http.Error(w, "this search cannot be pinned: it has no terms or filters, or is too long", http.StatusBadRequest)
return
}
stored := writeListCookie(w, pinnedSearchesCookie, toggleEntry(pinned, entry, pinnedSearchesMax))
if pinRefused(pinned, stored, entry) {
http.Redirect(w, r, "/pinned?full=1", http.StatusSeeOther)
return
}
http.Redirect(w, r, safeBackPath(r.FormValue("back")), http.StatusSeeOther)
}
func pinRefused(before, stored []string, e string) bool {
return !slices.Contains(before, e) && !slices.Contains(stored, e)
}
func (co *Corpus) handlePinnedPage(w http.ResponseWriter, r *http.Request) {
prefs := readPrefs(r)
var docs []*Document
for _, id := range co.readPinned(r) {
docs = append(docs, co.docIndex[id])
}
searches := co.parseSearchEntries(readPinnedSearchEntries(r))
data := &pinnedPage{
Base: pageWith(prefs),
Total: len(docs),
Searches: searches,
Full: r.URL.Query().Get("full") != "",
Cards: newCardView(prefs, docs),
}
data.Cards.ShowUnpin = true
noStore(w)
render(w, r, data)
}
func (co *Corpus) handlePinnedExport(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Cache-Control", "private, no-store")
w.Header().Set("Content-Disposition", `attachment; filename="snowchive-pinned.json"`)
ids := co.readPinned(r)
out := make([]apiDocSummary, 0, len(ids))
for _, id := range ids {
out = append(out, docSummary(*co.docIndex[id]))
}
writeJSON(w, http.StatusOK, map[string]interface{}{
"count": len(out),
"docs": out,
})
}

230
portions.go Normal file
View file

@ -0,0 +1,230 @@
// portions.go - the portion markings inside the transcripts.
package main
import (
"html/template"
"log"
"net/http"
"regexp"
"strings"
)
// A marking opens a line, optionally behind a bullet, a heading hash or a
// list number: "(U)", "* (S//SI) ...", "## (U//FOUO) Heading".
var rePortion = regexp.MustCompile(`^[ \t]{0,6}(?:[-*+][ \t]+|#{1,6}[ \t]+|\d+[.)][ \t]+)?\(([^)\n]{1,42})\)`)
var portionLevels = map[string]string{
"U": "U", "UNCLASSIFIED": "U", "FOUO": "U",
"C": "C", "CONFIDENTIAL": "C",
"S": "S", "SECRET": "S",
"TS": "TS", "TOP SECRET": "TS",
}
var portionControls = map[string]bool{
"SI": true, "TK": true, "GAMMA": true, "SI-GAMMA": true, "STLW": true, "TSP": true,
"HCS": true, "KLONDIKE": true, "ECI": true, "SI-ECI": true, "COMINT": true,
"FOUO": true, "NF": true, "NOFORN": true, "OC": true, "ORCON": true,
"PROPIN": true, "PR": true, "RELIDO": true, "RSEN": true, "FISA": true,
"IMCON": true, "DSEN": true, "REL": true, "FVEY": true, "NATO": true,
"X1": true,
}
var portionLevelNames = []string{"U", "C", "S", "TS"}
var portionLevelLabels = map[string]string{
"U": "UNCLASSIFIED", "C": "CONFIDENTIAL", "S": "SECRET", "TS": "TOP SECRET",
}
// reSlashRun folds "/", "//" and "///" to one separator.
var reSlashRun = regexp.MustCompile(`[ \t]*/+[ \t]*`)
// portionMark is one marking, canonicalised.
type portionMark struct {
Raw string // as the transcript writes it, without the parentheses
Canonical string // "TS//SI//NF"
Level string // "TS"
Controls []string // ["SI", "NF"], in the order the marking writes them
OK bool // every part recognised
}
func parsePortion(raw string) portionMark {
m := portionMark{Raw: strings.TrimSpace(raw)}
written := strings.Split(reSlashRun.ReplaceAllString(strings.Join(strings.Fields(m.Raw), " "), "//"), "//")
if lvl := strings.TrimSpace(written[0]); lvl == "" || lvl != strings.ToUpper(lvl) {
return m
}
parts := make([]string, len(written))
for i := range written {
parts[i] = strings.ToUpper(strings.TrimSpace(written[i]))
}
lvl, isLevel := portionLevels[parts[0]]
if !isLevel {
return m
}
m.Level = lvl
if parts[0] == "FOUO" {
m.Controls = append(m.Controls, "FOUO")
}
known := true
for _, p := range parts[1:] {
if p == "" {
continue
}
m.Controls = append(m.Controls, p)
if !portionControlKnown(p) {
known = false
}
}
m.OK = known
m.Canonical = m.Level
if len(m.Controls) > 0 {
m.Canonical += "//" + strings.Join(m.Controls, "//")
}
return m
}
func portionControlKnown(p string) bool {
if portionControls[p] {
return true
}
if strings.HasPrefix(p, "REL") {
return true
}
if strings.HasPrefix(p, "ECI ") || strings.HasPrefix(p, "SI-ECI") {
return true
}
return false
}
func extractPortions(text string) []portionMark {
var out []portionMark
fenced := false
for _, line := range strings.Split(text, "\n") {
t := strings.TrimSpace(line)
if strings.HasPrefix(t, "```") || strings.HasPrefix(t, "~~~") {
fenced = !fenced
continue
}
if fenced || strings.HasPrefix(t, ">") {
continue
}
m := rePortion.FindStringSubmatch(line)
if m == nil {
continue
}
if p := parsePortion(m[1]); p.Level != "" {
out = append(out, p)
}
}
return out
}
// --- the corpus view -------------------------------------------------------
type portionRow struct {
Name string // the canonical marking, a level, or a control
Label string // the spelled-out level, where there is one
Marks int // occurrences across the corpus
Docs int // documents carrying it
SearchURL template.URL
}
type portionSummary struct {
Docs int // documents carrying at least one marking
Marks int // markings found
Distinct int // distinct canonical markings
Unrecog int // markings whose parts are not all known
UnrecogEg []portionRow
Levels []portionRow
Markings []portionRow
Controls []portionRow
}
const portionRows = 200
func (co *Corpus) buildPortions() {
co.portionStats = portionSummary{}
markMarks, lvlMarks, ctlMarks := map[string]int{}, map[string]int{}, map[string]int{}
docsBy := map[string]map[string]int{"PL": {}, "PM": {}, "PC": {}}
unrecogMarks := map[string]int{}
for i := range co.archive.Docs {
d := &co.archive.Docs[i]
d.PortionLevels, d.PortionMarkings, d.PortionControls = nil, nil, nil
marks := extractPortions(d.FullText)
if len(marks) == 0 {
continue
}
co.portionStats.Docs++
co.portionStats.Marks += len(marks)
for _, m := range marks {
if !m.OK {
co.portionStats.Unrecog++
unrecogMarks[m.Raw]++
continue
}
markMarks[m.Canonical]++
lvlMarks[m.Level]++
d.PortionMarkings = append(d.PortionMarkings, m.Canonical)
d.PortionLevels = append(d.PortionLevels, m.Level)
for _, c := range m.Controls {
ctlMarks[c]++
d.PortionControls = append(d.PortionControls, c)
}
}
d.PortionLevels = normalizeList(d.PortionLevels)
d.PortionMarkings = normalizeList(d.PortionMarkings)
d.PortionControls = normalizeList(d.PortionControls)
for code, docs := range docsBy {
for _, v := range facetValues(d, code) {
docs[v]++
}
}
}
co.portionStats.Distinct = len(markMarks)
for _, l := range portionLevelNames {
if lvlMarks[l] == 0 {
continue
}
co.portionStats.Levels = append(co.portionStats.Levels, portionRow{
Name: l, Label: portionLevelLabels[l], Marks: lvlMarks[l], Docs: docsBy["PL"][l],
SearchURL: portionSearchURL("PL", l),
})
}
co.portionStats.Markings = portionRowsOf(markMarks, docsBy["PM"], portionRows)
for i := range co.portionStats.Markings {
r := &co.portionStats.Markings[i]
r.SearchURL = portionSearchURL("PM", r.Name)
}
co.portionStats.Controls = portionRowsOf(ctlMarks, docsBy["PC"], portionRows)
for i := range co.portionStats.Controls {
r := &co.portionStats.Controls[i]
r.SearchURL = portionSearchURL("PC", r.Name)
}
co.portionStats.UnrecogEg = portionRowsOf(unrecogMarks, nil, 25)
log.Printf("Portion markings: %d in %d documents, %d distinct (%d unrecognised)",
co.portionStats.Marks, co.portionStats.Docs, co.portionStats.Distinct, co.portionStats.Unrecog)
}
func portionSearchURL(code, value string) template.URL {
return template.URL(wholeArchive.link(searchURL([]searchClause{{Field: code, Value: exactClause(value)}})))
}
func portionRowsOf(marks, docs map[string]int, n int) []portionRow {
names := rankCounts(marks, n)
out := make([]portionRow, len(names))
for i, k := range names {
out[i] = portionRow{Name: k, Marks: marks[k], Docs: docs[k]}
}
return out
}
func (co *Corpus) handlePortions(w http.ResponseWriter, r *http.Request) {
render(w, r, &portionsPage{
Title: "Portion markings",
Stats: co.portionStats,
RowLimit: portionRows,
})
}

262
prefs.go Normal file
View file

@ -0,0 +1,262 @@
// prefs.go - per-browser UI preferences stored in cookies, plus shared
// cookie helpers used by the pins and history features.
package main
import (
"net/http"
"slices"
"strconv"
"strings"
)
// setCookie / delCookie are the single place cookie attributes are decided.
func setCookie(w http.ResponseWriter, name, val string) {
http.SetCookie(w, &http.Cookie{
Name: name,
Value: val,
Path: "/",
MaxAge: 365 * 24 * 3600,
SameSite: http.SameSiteLaxMode,
HttpOnly: true,
})
}
func delCookie(w http.ResponseWriter, name string) {
http.SetCookie(w, &http.Cookie{
Name: name,
Value: "",
Path: "/",
MaxAge: -1,
SameSite: http.SameSiteLaxMode,
HttpOnly: true,
})
}
const cookieValueMax = 3800
func writeListCookie(w http.ResponseWriter, name string, entries []string) []string {
n, size := 0, 0
for _, e := range entries {
add := len(e)
if n > 0 {
add++ // the "|" separator
}
if size+add > cookieValueMax {
break
}
size += add
n++
}
entries = entries[:n]
if len(entries) == 0 {
delCookie(w, name)
return entries
}
setCookie(w, name, strings.Join(entries, "|"))
return entries
}
type Prefs struct {
DocView string // "images" (default), "split", "text"
Theme string // "auto" (default), "light", "dark"
PerPage int // docs per page (default 20); 0 means "all", which cardsPerPage bounds
ShowClassification bool // classification/dissemination marking on document cards (default true)
ShowJournalists bool // journalist names on document cards (default true)
ShowPeople bool // people line on document cards (default true)
ShowPrograms bool // program names on document cards (default true)
ShowSummary bool // short summary (description) on document cards (default true)
ShowThumbs bool // thumbnails on search results & cards (default true)
ShowCollCols bool // category tables: per-collection count columns always visible (default false, toggle on the page)
TextOnly bool // suppress thumbnails and page images (default false)
CardPagePreview bool // desktop: full page preview on card-thumb hover (default false, opt-in)
ScrollDoc bool // doc view: cap pane height and scroll (default false)
InfoSidebar bool // doc view: metadata beside the document instead of above (default false); cookie is "above"/"beside", not a 0/1 flag
ProgramsAsList bool // doc view: programs one-per-line (default false, comma-separated)
TagsAsList bool // doc view: tags one-per-line (default false, comma-separated)
OpenInfo bool // doc view: "More info" expanded by default (default false)
CollapsePubTogether bool // doc view: collapse "published in the same report" (default false)
CollapseRelated bool // doc view: collapse related documents (default false)
TrackRecent bool // record recently viewed documents (default false)
TrackSearches bool // record recent searches (default false)
IncludeFOIA bool // FOIA releases in search, category totals, /random and related documents when the URL names no collection (default false, opt-in)
}
func (p Prefs) thumbsOn() bool { return p.ShowThumbs && !p.TextOnly }
const maxCardsPerPage = 500
func (p Prefs) cardsPerPage() int {
if p.PerPage <= 0 || p.PerPage > maxCardsPerPage {
return maxCardsPerPage
}
return p.PerPage
}
func newCardView(p Prefs, docs []*Document) cardView {
return cardView{
Docs: docs,
ShowThumbs: p.thumbsOn(),
CardPagePreview: p.CardPagePreview,
HideClassification: !p.ShowClassification,
HideJournalists: !p.ShowJournalists,
HidePeople: !p.ShowPeople,
HidePrograms: !p.ShowPrograms,
HideSummary: !p.ShowSummary,
FOIAQS: p.foiaLinkQS(),
}
}
func (p Prefs) foiaLinkQS() string {
return pageScope(nil, p, false).including(collectionFOIA).suffix()
}
var boolPrefs = []struct {
cookie string
form string
invert bool
field func(*Prefs) *bool
}{
{"pref_hide_classification", "hide_classification", true, func(p *Prefs) *bool { return &p.ShowClassification }},
{"pref_hide_journalists", "hide_journalists", true, func(p *Prefs) *bool { return &p.ShowJournalists }},
{"pref_hide_people", "hide_people", true, func(p *Prefs) *bool { return &p.ShowPeople }},
{"pref_hide_programs", "hide_programs", true, func(p *Prefs) *bool { return &p.ShowPrograms }},
{"pref_hide_summary", "hide_summary", true, func(p *Prefs) *bool { return &p.ShowSummary }},
{"pref_hide_thumbs", "hide_thumbs", true, func(p *Prefs) *bool { return &p.ShowThumbs }},
{"pref_show_coll_cols", "show_coll_cols", false, func(p *Prefs) *bool { return &p.ShowCollCols }},
{"pref_text_only", "text_only", false, func(p *Prefs) *bool { return &p.TextOnly }},
{"pref_page_preview", "page_preview", false, func(p *Prefs) *bool { return &p.CardPagePreview }},
{"pref_scroll_doc", "scroll_doc", false, func(p *Prefs) *bool { return &p.ScrollDoc }},
{"pref_programs_list", "programs_list", false, func(p *Prefs) *bool { return &p.ProgramsAsList }},
{"pref_tags_list", "tags_list", false, func(p *Prefs) *bool { return &p.TagsAsList }},
{"pref_open_info", "open_info", false, func(p *Prefs) *bool { return &p.OpenInfo }},
{"pref_collapse_pub", "collapse_pub", false, func(p *Prefs) *bool { return &p.CollapsePubTogether }},
{"pref_collapse_related", "collapse_related", false, func(p *Prefs) *bool { return &p.CollapseRelated }},
{"pref_track_recent", "track_recent", false, func(p *Prefs) *bool { return &p.TrackRecent }},
{"pref_track_searches", "track_searches", false, func(p *Prefs) *bool { return &p.TrackSearches }},
{"pref_include_foia", "include_foia", false, func(p *Prefs) *bool { return &p.IncludeFOIA }},
}
var valuePrefs = []struct {
cookie string
form string
allowed []string
def string
apply func(*Prefs, string)
}{
{"pref_doc_view", "doc_view", []string{"images", "split", "text"}, "images",
func(p *Prefs, v string) { p.DocView = v }},
{"pref_theme", "theme", []string{"auto", "light", "dark"}, "auto",
func(p *Prefs, v string) { p.Theme = v }},
{"pref_per_page", "per_page", []string{"10", "20", "50", "100", "all"}, "20",
func(p *Prefs, v string) {
if v == "all" {
p.PerPage = 0
return
}
if n, err := strconv.Atoi(v); err == nil {
p.PerPage = n
}
}},
{"pref_info_position", "info_position", []string{"above", "beside"}, "above",
func(p *Prefs, v string) { p.InfoSidebar = v == "beside" }},
}
var prefCookies = func() []string {
var out []string
for _, vp := range valuePrefs {
out = append(out, vp.cookie)
}
for _, bp := range boolPrefs {
out = append(out, bp.cookie)
}
return append(out, recentDocsCookie, recentSearchesCookie)
}()
func (p Prefs) ThemeAttr() string {
if p.Theme == "auto" {
return ""
}
return p.Theme
}
func readPrefs(r *http.Request) Prefs {
var p Prefs
cookie := map[string]string{}
for _, c := range r.Cookies() {
cookie[c.Name] = c.Value
}
for i := range boolPrefs {
bp := &boolPrefs[i]
v := cookie[bp.cookie] == "1"
if bp.invert {
v = !v
}
*bp.field(&p) = v
}
for i := range valuePrefs {
vp := &valuePrefs[i]
v := cookie[vp.cookie]
if !slices.Contains(vp.allowed, v) {
v = vp.def
}
vp.apply(&p, v)
}
return p
}
func handlePreferences(w http.ResponseWriter, r *http.Request) {
noStore(w)
q := r.URL.Query()
render(w, r, &preferencesPage{
Base: pageWith(readPrefs(r)),
MaxCards: maxCardsPerPage,
Saved: q.Get("saved") == "1",
Reset: q.Get("reset") == "1",
})
}
func handlePreferencesSave(w http.ResponseWriter, r *http.Request) {
if !readPOSTForm(w, r) {
return
}
if r.FormValue("reset") == "1" {
for _, name := range prefCookies {
delCookie(w, name)
}
http.Redirect(w, r, "/preferences?reset=1", http.StatusSeeOther)
return
}
for _, vp := range valuePrefs {
v := r.FormValue(vp.form)
switch {
case v == vp.def:
delCookie(w, vp.cookie)
case slices.Contains(vp.allowed, v):
setCookie(w, vp.cookie, v)
}
}
for _, bp := range boolPrefs {
if r.FormValue(bp.form) == "1" {
setCookie(w, bp.cookie, "1")
} else {
delCookie(w, bp.cookie)
}
}
if r.FormValue("track_recent") != "1" {
delCookie(w, recentDocsCookie)
}
if r.FormValue("track_searches") != "1" {
delCookie(w, recentSearchesCookie)
}
http.Redirect(w, r, "/preferences?saved=1", http.StatusSeeOther)
}

2644
search.go Normal file

File diff suppressed because it is too large Load diff

233
seo.go Normal file
View file

@ -0,0 +1,233 @@
// seo.go - robots.txt, sitemap.xml, and the per-document metadata embeds.
package main
import (
"bytes"
"encoding/json"
"encoding/xml"
"fmt"
"html/template"
"net/http"
"net/url"
"strings"
)
var canonicalBase string
func requestBase(r *http.Request) string {
if canonicalBase != "" {
return canonicalBase
}
scheme := "http"
if r.TLS != nil || r.Header.Get("X-Forwarded-Proto") == "https" {
scheme = "https"
}
return scheme + "://" + r.Host
}
func handleRobots(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Cache-Control", "public, max-age=86400")
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
fmt.Fprintf(w, `User-agent: *
Allow: /
Disallow: /search
Disallow: /api/
Disallow: /random
Disallow: /pinned
Disallow: /preferences
Sitemap: %s/sitemap.xml
`, requestBase(r))
}
type sitemapURL struct {
Loc string `xml:"loc"`
}
type sitemapSet struct {
XMLName xml.Name `xml:"urlset"`
Xmlns string `xml:"xmlns,attr"`
URLs []sitemapURL `xml:"url"`
}
func (co *Corpus) buildSitemapPaths() {
paths := []string{
"/", "/browse", "/articles", "/glossary", "/changelog", "/about",
}
for _, c := range categories {
paths = append(paths, "/browse/"+c.Slug)
}
paths = append(paths, "/explore")
for _, m := range exploreMetrics {
paths = append(paths, "/explore/"+m.Slug)
}
paths = append(paths, "/explore/terms")
paths = append(paths, "/explore/classification")
paths = append(paths, "/explore/portions")
paths = append(paths, "/explore/outlets")
if len(co.assemblies) > 0 {
paths = append(paths, assembliesPath)
for _, a := range co.assemblies {
paths = append(paths, assemblyPath(a.ID))
}
}
for _, d := range co.archive.Docs {
paths = append(paths, docPath(d.ID))
}
for _, a := range co.articles {
if a.Title != "" {
paths = append(paths, "/article/"+a.Key)
}
}
co.sitemapPaths = paths
}
func (co *Corpus) marshalSitemap(base string) ([]byte, error) {
set := sitemapSet{Xmlns: "http://www.sitemaps.org/schemas/sitemap/0.9"}
set.URLs = make([]sitemapURL, len(co.sitemapPaths))
for i, p := range co.sitemapPaths {
set.URLs[i] = sitemapURL{Loc: base + p}
}
out, err := xml.MarshalIndent(set, "", " ")
if err != nil {
return nil, err
}
return append([]byte(xml.Header), out...), nil
}
func parseCanonicalBase(base string) (string, error) {
u, err := url.Parse(base)
if err != nil {
return "", fmt.Errorf("invalid base URL %q: %w", base, err)
}
if (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" || u.User != nil ||
(u.Path != "" && u.Path != "/") || u.RawQuery != "" || u.Fragment != "" {
return "", fmt.Errorf("base must be scheme://host with no path/query/fragment/userinfo, e.g. https://archive.example.org (got %q)", base)
}
return u.Scheme + "://" + u.Host, nil
}
const sitemapBaseMark = "{base url}"
func (co *Corpus) buildCanonicalSitemap() {
co.sitemapParts = nil
if canonicalBase == "" {
if b, err := co.marshalSitemap(sitemapBaseMark); err == nil {
co.sitemapParts = bytes.Split(b, []byte(sitemapBaseMark))
}
return
}
b, err := co.marshalSitemap(canonicalBase)
if err != nil {
errorf("sitemap: %v - building it per request instead", err)
return
}
co.canonicalSitemap = b
}
func (co *Corpus) handleSitemap(w http.ResponseWriter, r *http.Request) {
body := co.canonicalSitemap
var base bytes.Buffer
if body == nil && co.sitemapParts != nil {
xml.EscapeText(&base, []byte(requestBase(r)))
} else if body == nil {
var err error
body, err = co.marshalSitemap(requestBase(r))
if err != nil {
serverError(w, err)
return
}
}
w.Header().Set("Cache-Control", "public, max-age=3600")
w.Header().Set("Content-Type", "application/xml; charset=utf-8")
if body != nil {
w.Write(body)
return
}
for i, part := range co.sitemapParts {
if i > 0 {
w.Write(base.Bytes())
}
w.Write(part)
}
}
// docEmbeds is handed to doc.html as data["Embeds"].
type docEmbeds struct {
JSONLDTag template.HTML
COinS string
}
func buildDocEmbeds(doc *Document, absURL, pdfURL string) docEmbeds {
type entity struct {
Type string `json:"@type"`
Name string `json:"name"`
}
type media struct {
Type string `json:"@type"`
ContentURL string `json:"contentUrl"`
EncodingFormat string `json:"encodingFormat"`
}
keywords := strings.Join(append(append([]string{}, doc.DocMeta.Programs...), doc.Tags...), ", ")
ld := struct {
Context string `json:"@context"`
Type string `json:"@type"`
Name string `json:"name"`
URL string `json:"url"`
Identifier string `json:"identifier"`
Description string `json:"description,omitempty"`
DateCreated string `json:"dateCreated,omitempty"`
DatePublished string `json:"datePublished,omitempty"`
InLanguage string `json:"inLanguage"`
Keywords string `json:"keywords,omitempty"`
Author *entity `json:"author,omitempty"`
Publisher entity `json:"publisher"`
AssociatedMedia media `json:"associatedMedia"`
}{
Context: "https://schema.org",
Type: "DigitalDocument",
Name: doc.Title,
URL: absURL,
Identifier: absURL,
Description: doc.Description,
DateCreated: doc.DocMeta.DateCreated,
DatePublished: oldestPubDate(doc),
InLanguage: "en",
Keywords: keywords,
Publisher: entity{Type: "Organization", Name: "Snowden Archive"},
AssociatedMedia: media{
Type: "MediaObject",
ContentURL: pdfURL,
EncodingFormat: "application/pdf",
},
}
if doc.DocMeta.Creator != "" {
ld.Author = &entity{Type: "Organization", Name: doc.DocMeta.Creator}
}
var tag template.HTML
if b, err := json.Marshal(ld); err == nil {
tag = template.HTML(`<script type="application/ld+json">` + string(b) + `</script>`)
}
v := url.Values{}
v.Set("ctx_ver", "Z39.88-2004")
v.Set("rft_val_fmt", "info:ofi/fmt:kev:mtx:dc")
v.Set("rft.title", doc.Title)
if doc.DocMeta.Creator != "" {
v.Set("rft.creator", doc.DocMeta.Creator)
}
if doc.DocMeta.DateCreated != "" {
v.Set("rft.date", doc.DocMeta.DateCreated)
}
v.Set("rft.type", "Text")
v.Set("rft.format", "application/pdf")
v.Set("rft.identifier", absURL)
v.Set("rft.publisher", "Snowden Archive")
for _, p := range doc.DocMeta.Programs {
v.Add("rft.subject", p)
}
return docEmbeds{JSONLDTag: tag, COinS: v.Encode()}
}

415
static/main.js Normal file
View file

@ -0,0 +1,415 @@
// saved custom CSS; ?nocss=1 clears it
(function () {
try {
if (/[?&]nocss=1(?:&|$)/.test(location.search)) {
localStorage.removeItem('snowchive-custom-css');
return;
}
var css = localStorage.getItem('snowchive-custom-css') || '';
if (!css) return;
var el = document.createElement('style');
el.id = 'snowchive-custom-css';
el.textContent = css;
document.head.appendChild(el);
} catch (e) {}
})();
function snowchiveStorage() {
try { return window.localStorage || null; } catch (e) { return null; }
}
function isTyping(el) {
return !!el && (/^(INPUT|TEXTAREA|SELECT)$/.test(el.tagName) || el.isContentEditable);
}
// animated thumb rollover
(function () {
if (window.matchMedia && matchMedia('(prefers-reduced-motion: reduce)').matches) return;
const gifThumbs = Array.from(document.querySelectorAll('.thumb img[data-gif], .result-thumb img[data-gif]'));
gifThumbs.forEach(img => {
img.dataset.still = img.getAttribute('src');
const gif = img.getAttribute('data-gif');
img.addEventListener('mouseenter', () => img.setAttribute('src', gif));
img.addEventListener('mouseleave', () => img.setAttribute('src', img.dataset.still));
img.addEventListener('touchstart', () => {
gifThumbs.forEach(o => {
if (o !== img) o.setAttribute('src', o.dataset.still);
});
img.setAttribute('src', gif);
}, { passive: true });
});
})();
// card page-preview panel (opt-in, desktop)
(function () {
if (!window.matchMedia || !matchMedia('(hover: hover) and (pointer: fine)').matches) return;
if (!document.querySelector('[data-pp-id]')) return;
var panel = null, activeEl = null, pages = [], idx = 0, docId = '', timer = null, dragging = false, pinned = false;
var openTimer = null;
var WKEY = 'snowchive.pp.width', WMIN = 300, WDEFAULT = 460;
function imgURL(name) { return '/files/' + docId + '/images/' + name; }
function cancelClose() { if (timer) { clearTimeout(timer); timer = null; } }
function scheduleClose() { if (dragging || pinned) return; cancelClose(); timer = setTimeout(close_, 220); }
function maxW() { return Math.round(window.innerWidth * 0.9); }
function clampW(w) { return Math.max(WMIN, Math.min(w, maxW())); }
function loadW() { try { var v = parseInt(localStorage.getItem(WKEY), 10); if (v) return v; } catch (e) {} return WDEFAULT; }
function saveW(w) { try { localStorage.setItem(WKEY, String(w)); } catch (e) {} }
function applyW(w) { if (panel) panel.style.width = clampW(w) + 'px'; }
function build() {
panel = document.createElement('div');
panel.className = 'pp-panel';
panel.innerHTML =
'<div class="pp-resizer" title="Drag to resize (press s to pin first)"></div>' +
'<div class="pp-stage"><img class="pp-img" alt=""></div>' +
'<div class="pp-strip"></div>' +
'<div class="pp-foot">' +
'<span class="pp-counter"></span>' +
'<span class="pp-hint pp-hint-free">h / &larr; prev &middot; l / &rarr; next &middot; <b>s</b> pin to resize</span>' +
'<span class="pp-hint pp-hint-pinned">&#128204; pinned &middot; h / &larr; prev &middot; l / &rarr; next &middot; <b>esc</b> close</span>' +
'</div>';
document.body.appendChild(panel);
applyW(loadW());
initResize();
panel.addEventListener('mouseenter', cancelClose);
panel.addEventListener('mouseleave', scheduleClose);
}
function initResize() {
var resizer = panel.querySelector('.pp-resizer');
function onMove(e) { applyW(window.innerWidth - e.clientX); }
function onUp() {
dragging = false;
resizer.classList.remove('pp-dragging');
document.body.style.userSelect = '';
document.removeEventListener('mousemove', onMove);
document.removeEventListener('mouseup', onUp);
saveW(parseInt(panel.style.width, 10));
}
resizer.addEventListener('mousedown', function (e) {
e.preventDefault();
dragging = true;
cancelClose();
resizer.classList.add('pp-dragging');
document.body.style.userSelect = 'none';
document.addEventListener('mousemove', onMove);
document.addEventListener('mouseup', onUp);
});
}
function go(n) {
if (!pages.length) return;
idx = n < 0 ? 0 : (n >= pages.length ? pages.length - 1 : n);
panel.querySelector('.pp-img').src = imgURL(pages[idx]);
panel.querySelector('.pp-counter').textContent = (idx + 1) + ' / ' + pages.length;
var strip = panel.querySelector('.pp-strip'), i;
for (i = 0; i < strip.children.length; i++) {
strip.children[i].classList.toggle('pp-cur', i === idx);
}
var cur = strip.children[idx];
if (cur) {
var cr = cur.getBoundingClientRect(), sr = strip.getBoundingClientRect();
strip.scrollLeft += (cr.left + cr.width / 2) - (sr.left + sr.width / 2);
}
}
function buildStrip() {
var strip = panel.querySelector('.pp-strip');
strip.textContent = '';
pages.forEach(function (name, i) {
var t = new Image();
t.className = 'pp-th';
t.loading = 'lazy';
t.src = imgURL(name);
t.alt = 'Page ' + (i + 1);
t.addEventListener('click', function () { go(i); });
strip.appendChild(t);
});
}
function open_(el) {
cancelClose();
if (el === activeEl) return;
var imgs = (el.getAttribute('data-pp-imgs') || '').split(' ').filter(Boolean);
if (!imgs.length) { close_(); return; }
activeEl = el;
docId = el.getAttribute('data-pp-id');
pages = imgs;
if (!panel) build();
buildStrip();
go(0);
panel.classList.add('pp-open');
}
function close_() {
cancelClose();
pinned = false;
if (panel) { panel.classList.remove('pp-open'); panel.classList.remove('pp-pinned'); }
activeEl = null;
}
document.addEventListener('mouseover', function (e) {
if (dragging || pinned) return;
var el = e.target.closest('[data-pp-id]');
if (!el) return;
if (el === activeEl) { cancelClose(); return; }
clearTimeout(openTimer);
openTimer = setTimeout(function () { open_(el); }, 150);
});
document.addEventListener('mouseout', function (e) {
var el = e.target.closest('[data-pp-id]');
if (!el) return;
var to = e.relatedTarget;
if (to && (el.contains(to) || (panel && panel.contains(to)))) return;
clearTimeout(openTimer);
scheduleClose();
});
document.addEventListener('keydown', function (e) {
if (!activeEl) return;
if (e.altKey || e.ctrlKey || e.metaKey) return;
if (isTyping(document.activeElement)) return;
switch (e.key) {
case 'h': case 'ArrowLeft': go(idx - 1); e.preventDefault(); break;
case 'l': case 'ArrowRight': go(idx + 1); e.preventDefault(); break;
case 'Home': if (pinned) { go(0); e.preventDefault(); } break;
case 'End': if (pinned) { go(pages.length - 1); e.preventDefault(); } break;
case 's': case 'S':
pinned = !pinned;
panel.classList.toggle('pp-pinned', pinned);
if (!pinned && !(panel.matches(':hover') || (activeEl && activeEl.matches(':hover')))) close_();
e.preventDefault();
break;
case 'Escape': close_(); break;
}
});
window.addEventListener('resize', function () { if (panel) applyW(parseInt(panel.style.width, 10) || WDEFAULT); });
})();
// lightbox keyboard and swipe navigation
(function () {
var total = document.querySelectorAll('.lightbox:not(.lb-alt)').length;
if (!total) return;
document.addEventListener('keydown', function (e) {
if (e.altKey || e.ctrlKey || e.metaKey) return;
if (isTyping(e.target)) return;
if (e.key === 'Escape' && /^#lb-a\d+-\d+$/.test(location.hash)) {
location.hash = '#lb-closed';
e.preventDefault();
return;
}
var m = /^#lb-(\d+)$/.exec(location.hash);
if (!m) return;
var n = parseInt(m[1], 10);
switch (e.key) {
case 'ArrowLeft': case 'h':
if (n > 1) location.hash = '#lb-' + (n - 1);
e.preventDefault();
break;
case 'ArrowRight': case 'l':
if (n < total) location.hash = '#lb-' + (n + 1);
e.preventDefault();
break;
case 'Escape':
location.hash = '#lb-closed';
e.preventDefault();
break;
}
});
window.addEventListener('hashchange', function (e) {
var was = /#lb-(\d+)$/.exec(e.oldURL), is = /#lb-(\d+)$/.exec(e.newURL);
if (is) {
var close = document.querySelector('#lb-' + is[1] + ' .lb-close');
if (close) close.focus({ preventScroll: true });
} else if (was) {
var thumbs = document.querySelectorAll('a[href="#lb-' + was[1] + '"]');
for (var i = 0; i < thumbs.length; i++) {
if (!thumbs[i].closest('.lightbox')) { thumbs[i].focus(); break; }
}
}
});
var tx = 0, ty = 0;
document.addEventListener('touchstart', function (e) {
if (e.touches.length > 1) { tx = NaN; return; }
tx = e.changedTouches[0].clientX;
ty = e.changedTouches[0].clientY;
}, { passive: true });
document.addEventListener('touchend', function (e) {
var m = /^#lb-(\d+)$/.exec(location.hash);
if (!m) return;
var dx = e.changedTouches[0].clientX - tx;
var dy = e.changedTouches[0].clientY - ty;
if (!(Math.abs(dx) >= 50) || Math.abs(dy) > Math.abs(dx)) return;
var n = parseInt(m[1], 10);
if (dx < 0 && n < total) location.hash = '#lb-' + (n + 1);
else if (dx > 0 && n > 1) location.hash = '#lb-' + (n - 1);
}, { passive: true });
})();
// private per-document notes
(function () {
var box = document.getElementById('note-box');
var localStorage = snowchiveStorage();
if (!box || !localStorage) return;
box.hidden = false;
var key = 'snowchive-note:' + box.dataset.docId;
var ta = document.getElementById('note-text');
var status = document.getElementById('note-status');
try {
var saved = localStorage.getItem(key);
if (saved) {
var obj = JSON.parse(saved);
ta.value = obj.text || '';
if (ta.value) { box.open = true; status.textContent = '(saved)'; }
}
} catch (e) {}
var t = null;
function save() {
clearTimeout(t);
t = null;
try {
if (ta.value.trim() === '') {
localStorage.removeItem(key);
status.textContent = '';
} else {
localStorage.setItem(key, JSON.stringify({
title: box.dataset.docTitle,
text: ta.value,
updated: new Date().toISOString()
}));
status.textContent = '(saved)';
}
} catch (e) { status.textContent = '(could not save)'; }
}
ta.addEventListener('input', function () {
status.textContent = '...';
clearTimeout(t);
t = setTimeout(save, 400);
});
function flush() { if (t !== null) save(); }
ta.addEventListener('change', flush);
window.addEventListener('pagehide', flush);
})();
// the Pinned page's note listing
(function () {
var box = document.getElementById('note-list-box');
var list = document.getElementById('note-list');
if (!box || !list) return;
var localStorage = snowchiveStorage();
if (!localStorage) return;
var PREFIX = 'snowchive-note:';
var items = [];
try {
for (var i = 0; i < localStorage.length; i++) {
var k = localStorage.key(i);
if (!k || k.indexOf(PREFIX) !== 0) continue;
try {
var obj = JSON.parse(localStorage.getItem(k));
items.push({ id: k.slice(PREFIX.length), title: obj.title || k.slice(PREFIX.length),
text: obj.text || '', updated: obj.updated || '' });
} catch (e) {}
}
} catch (e) {}
if (!items.length) return;
items.sort(function (a, b) { return a.updated < b.updated ? 1 : -1; });
box.hidden = false;
items.forEach(function (it) {
var li = document.createElement('li');
li.className = 'note-item';
var a = document.createElement('a');
a.href = '/doc/' + encodeURIComponent(it.id);
a.textContent = it.title;
li.appendChild(a);
if (it.updated) {
var d = document.createElement('span');
d.className = 'faint note-date';
d.textContent = ' ' + it.updated.slice(0, 10);
li.appendChild(d);
}
var del = document.createElement('button');
del.type = 'button';
del.className = 'link-btn unpin-btn';
del.textContent = 'delete';
del.addEventListener('click', function () {
if (!confirm('Delete this note?')) return;
try { localStorage.removeItem(PREFIX + it.id); } catch (e) { return; }
items.splice(items.indexOf(it), 1);
li.remove();
if (!list.children.length) box.hidden = true;
});
li.appendChild(del);
var p = document.createElement('div');
p.className = 'note-text';
p.textContent = it.text;
li.appendChild(p);
list.appendChild(li);
});
document.getElementById('note-export').addEventListener('click', function () {
var blob = new Blob([JSON.stringify(items, null, 2)], { type: 'application/json' });
var a = document.createElement('a');
a.href = URL.createObjectURL(blob);
a.download = 'snowchive-notes.json';
a.click();
// revoked later: Safari starts the download after click() returns
setTimeout(function () { URL.revokeObjectURL(a.href); }, 40000);
});
})();
// the Preferences page's custom-CSS editor
(function () {
var localStorage = snowchiveStorage();
if (!localStorage) return;
var fs = document.getElementById('custom-css-details');
var ta = document.getElementById('custom-css');
var status = document.getElementById('custom-css-status');
if (!fs || !ta || !status) return;
fs.hidden = false;
var KEY = 'snowchive-custom-css';
try {
ta.value = localStorage.getItem(KEY) || '';
} catch (e) {}
function apply() {
var existing = document.getElementById('snowchive-custom-css');
if (existing) existing.remove();
if (ta.value) {
var el = document.createElement('style');
el.id = 'snowchive-custom-css';
el.textContent = ta.value;
document.head.appendChild(el);
}
}
document.getElementById('custom-css-save').addEventListener('click', function () {
try {
if (ta.value) {
localStorage.setItem(KEY, ta.value);
} else {
localStorage.removeItem(KEY);
}
apply();
status.textContent = ta.value ? 'Saved and applied.' : 'Cleared - custom CSS is off.';
} catch (e) { status.textContent = 'Could not save.'; }
});
})();

2353
static/style.css Normal file

File diff suppressed because it is too large Load diff

14
templates/404.html Normal file
View file

@ -0,0 +1,14 @@
{{ define "title" }}Page not found - Snowden Archive{{ end }}
{{ define "content" }}
<div class="notfound">
<h1>404 - Page not found</h1>
<p class="muted">Nothing lives at <code>{{ .Path }}</code>. It may have been moved, removed, or mistyped.</p>
<form action="/search" method="GET" class="home-search-form" role="search">
<input type="text" name="q" class="home-search" placeholder="Search the archive... (Enter to search)" aria-label="Search the archive">
</form>
<p><a href="/">&larr; Home</a> &middot; <a href="/browse">Browse the archive</a></p>
</div>
{{ end }}
{{ template "base.html" . }}

544
templates/about.html Normal file
View file

@ -0,0 +1,544 @@
{{ define "title" }}About - Snowden Archive{{ end }}
{{ define "content" }}
<div class="about-page">
<h1>About this archive</h1>
<p>This is a searchable archive of all published documents from the 2013
Snowden disclosures. The archive includes documents published alongside
news reporting, The Intercept's bulk SIDtoday release, and FOIA material
(official releases from FOIA litigation and FISC and ODNI
declassifications).</p>
<p>Every document carries its original PDF, page images, extracted text,
and curated metadata - agencies, programs, countries, classification
markings, people, and the journalists and articles that published it.</p>
<p>Use <a href="/browse">Browse</a> to explore by category,
<a href="/search">Search</a> for fielded boolean queries across text and
metadata, <a href="/articles">Articles</a> to follow the disclosures story
by story, the <a href="/glossary">Glossary</a> to decode codenames and
acronyms, and <a href="/explore">Explore</a> to dig deeper into ranked
views and lists.</p>
<nav class="toc" aria-label="Table of contents">
<strong>Contents</strong>
<ul>
<li><a href="#privacy">Privacy</a></li>
<li><a href="#whats-in">What's in this archive?</a></li>
<li><a href="#search">How to search</a></li>
{{ if .APIEnabled }}<li><a href="#api">Downloading data as JSON</a>
<ul><li><a href="#metadata">Metadata structure</a></li></ul>
</li>{{ else }}<li><a href="#metadata">Metadata structure</a></li>{{ end }}
<li><a href="#download-archive">Downloading the archive</a>
<ul>
<li><a href="#downloading-all-data">Downloading all data</a></li>
<li><a href="#downloading-texts">Downloading only extracted texts and metadata</a></li>
</ul>
</li>
<li><a href="#own-copy">Can I run my own copy or mirror this?</a></li>
<li><a href="#contact">Contact</a></li>
<li><a href="#contributing">Contributing</a></li>
<li><a href="#thanks">Thank you</a></li>
</ul>
</nav>
<section id="privacy">
<h2>Privacy</h2>
<p>By default, the server keeps no state about its visitors. Pins and
history live only in your own browser's cookies, and history features
are off unless you opt in via <a href="/preferences">Preferences</a>.
Anonymous page views set no cookies at all. All core features on this
website work without JavaScript.</p>
<p>Note that you can't verify that this server doesn't log and analyze
your visits. Use Tor to hide your real IP address, or
<a href="#own-copy">set up</a> your own instance on your machine to
browse fully offline.</p>
</section>
<section id="whats-in">
<h2>What's in this archive?</h2>
<p>By default, two collections are shown as tabs on the
<a href="/browse">Browse</a> page: <strong>Reported</strong> documents
(leaked files as the outlets that broke the stories presented them) and
the <strong>SIDtoday Archive</strong> (The Intercept's bulk release of
the NSA's internal SIDtoday newsletters). The third collection,
<strong>FOIA</strong> (official releases from FOIA litigation and FISC
and ODNI declassifications), can be included via
<a href="/preferences">Preferences</a>.</p>
<p>The metadata and many of the extracted texts are curated by hand and
may contain errors or omissions, so treat them as a guide to the
documents rather than as established fact, and check the original
documents and reporting before relying on them.</p>
</section>
<section id="search">
<h2>How to search</h2>
<p>The <a href="/search">Search</a> page supports fielded boolean
queries. Restrict a term to a field (title, full text, programs,
countries, journalists, and so on), combine clauses with AND/OR, quote
<code>"text"</code> to find it anywhere, match exact whole words with
<code>=...</code> (<code>publisher:="The Guardian"</code> matches the
whole value), and use <code>*</code> wildcards (<code>tap*</code>,
<code>*apping</code>).</p>
<p>Ways to write a value, in any row or after any operator:</p>
<ul>
<li><code>cable tapping</code> finds both words anywhere, in their other forms too (<em>tapping</em> also finds <em>tapped</em>).</li>
<li><code>"apping"</code> finds that text anywhere, even inside a word, like <code>grep -i</code> (it finds, for example, <em>mapping</em> and <em>tapping</em>).</li>
<li><code>=tap</code> or <code>="cable tapping"</code> finds exactly those whole words, so <code>=tap</code> skips <em>taps</em> and <em>strap</em>. On a metadata field it matches one whole value, so <code>agency:=NSA</code> finds documents whose agency is NSA but not <em>NSA, GCHQ</em>.</li>
<li>The <code>*</code> wildcard finds part of a word: <code>tap*</code> finds words starting with <em>tap</em>, <code>*land</code> words ending in <em>land</em>, and <code>*keys*</code> words containing <em>keys</em>. A leading <code>*</code> needs at least three letters after it.</li>
<li><code>"upstream collection"~10</code> finds the words within 10 words of each other (in any order, common stopwords ignored).</li>
<li>A leading <code>-</code> excludes within its row: <code>xkeyscore -tempora</code> finds <em>xkeyscore</em> without <em>tempora</em>, and the row keeps that meaning under OR and NOT.</li>
</ul>
<p>The simple search box (on the home page, on the 404 page, and via
the URL parameter <code>?q=</code>) understands field operators:</p>
<ul>
<li><code>program:XKEYSCORE country:Germany cable</code> or <code>journalist:="Poitras, Laura"</code>.</li>
<li>A leading <code>-</code> negates (<code>-tag:=junk</code>, <code>-draft</code>).</li>
<li><code>date:2013</code>, <code>date:2013..2015</code> and <code>published:2014-06</code> filter by date.</li>
<li>Operators: text, title, content, description, agency, journalist, publisher, classification, program, dissemination, banner, genre, series, country, people, tag, type.</li>
<li>Bare words search all fields.</li>
</ul>
</section>
<section id="api">
{{ if .APIEnabled }}
<h2>Downloading data as JSON</h2>
<p>API endpoints:</p>
<div class="table-scroll">
<table class="data-table about-table">
<thead><tr><th>Endpoint</th><th>Returns</th></tr></thead>
<tbody>
<tr><td><a href="/api/docs"><code>/api/docs</code></a></td><td>Metadata for every document in the archive, with pagination.</td></tr>
<tr><td><code>/api/doc/{id}</code></td><td>A single document's metadata and extracted text, where <code>{id}</code> is the document's identifier.</td></tr>
<tr><td><a href="/api/search?q=xkeyscore"><code>/api/search?q=xkeyscore</code></a></td><td>Search results for term xkeyscore. It accepts the same parameters as the Search page, so you can also pass <code>fqf</code>/<code>fqv</code>/<code>fqc</code> clause parameters for fielded queries.</td></tr>
<tr><td><a href="/api/facets"><code>/api/facets</code></a></td><td>Counts for every browsable category (programs, agencies, countries, people, and so on).</td></tr>
</tbody>
</table>
</div>
<p>Responses are plain JSON. For paginated responses, use the URL
parameter <code>p</code> to navigate pages (e.g.
<a href="/api/docs?p=2"><code>/api/docs?p=2</code></a>).</p>
<p>See the <a href="#download-archive">Downloading the archive</a>
section on how to download the whole archive.</p>
<p><strong>Important if you are a crawler, robot or AI trying to
download large amounts of data from this website:</strong> please use
the downloadable datasets from the
<a href="#download-archive">Downloading the archive</a> section. The
information on this website is identical to the datasets, so you gain
nothing by crawling this website.</p>
{{ end }}
<details class="about-collapse">
<summary>{{ if .APIEnabled }}<h3 id="metadata">Metadata structure</h3>{{ else }}<h2 id="metadata">Metadata structure</h2>{{ end }}</summary>
<p>Every document ships a curated <code>metadata.json</code>:</p>
<h4>The document record</h4>
<dl class="field-dict">
<dt><code>id</code></dt>
<dd>The document's stable identifier: the name of its directory
and the last part of its URL (<code>/doc/{id}</code>). It is not
written inside <code>metadata.json</code> - the directory name is
the id.</dd>
<dt><code>aka_id</code></dt>
<dd>Other names the same document was released under, most often
the file name another publisher or a mirror gave its PDF. A list
of entries, each described below.</dd>
<dt><code>aka_id[].id</code></dt>
<dd>The other name itself. Searchable, so a document can be found
by a file name it circulated under elsewhere.</dd>
<dt><code>aka_id[].doc_source</code></dt>
<dd>The web page where the document was released under that name,
linked as &ldquo;page&rdquo; next to the alias.</dd>
<dt><code>aka_id[].doc_url</code></dt>
<dd>The direct link to the PDF released under that name, linked
as &ldquo;PDF&rdquo; next to the alias.</dd>
<dt><code>title</code></dt>
<dd>The document's title, written by the curators - usually the
title the document gives itself or the one its publisher used.</dd>
<dt><code>description</code></dt>
<dd>A short summary of what the document is and says, shown under
the publication list and in search results and cards.
Descriptions are the work of archivists without inside knowledge -
corrections welcome.</dd>
<dt><code>long_description</code></dt>
<dd>A longer account of the contents where one has been written,
behind &ldquo;Show more info&rdquo; on the document page; often
empty. A <code>description:</code> search looks in both
descriptions.</dd>
<dt><code>tags</code></dt>
<dd>Free-text subjects added by the curators. Unlike the other
lists they are not necessarily taken from the document's own
vocabulary, so they are the loosest thing to browse or search by.
</dd>
<dt><code>type</code></dt>
<dd>The kind of release: <code>REPORTED</code> (a leaked document
as an outlet published it), <code>SIDTODAY</code> (an article of
the NSA's internal SIDtoday newsletter from The Intercept's bulk
release) or <code>FOIA</code> (obtained through a public-records
request). When it is empty, a document filed in the SIDtoday or FOIA
folder takes that type and any other is <code>REPORTED</code>.</dd>
<dt><code>collection</code></dt>
<dd>Which of the three collections - the tabs on
<a href="/browse">Browse</a> - the document belongs to:
<code>news</code> (Reported), <code>sidtoday-archive</code> or
<code>foia</code>. Documents filed under the SIDtoday and FOIA
folders belong to those collections whatever this says; elsewhere
an empty value means Reported, or FOIA for a document of type
<code>FOIA</code>. FOIA releases are left out of
browsing and search unless you include them in
<a href="/preferences">Preferences</a>.</dd>
<dt><code>pages</code></dt>
<dd>The page count of the PDF. When it is missing, the number of
page images is used instead.</dd>
<dt><code>pdf_sha256</code></dt>
<dd>The SHA-256 of the PDF as archived.</dd>
<dt><code>archive_comment</code></dt>
<dd>A note from the curators about this archive's copy - how it
was assembled, where it was found, what differs between releases,
or a known problem with it.</dd>
</dl>
<h4>The document itself: <code>document</code></h4>
<dl class="field-dict">
<dt><code>document.creator</code></dt>
<dd>The agency that <em>wrote</em> the document, in its short form
(<code>NSA</code>, <code>GCHQ</code>, <code>FISC</code>,
<code>DOJ</code>, <code>CSE</code> ...) - not the agency it is
about and not the one that published it. Empty when the author is
not known.</dd>
<dt><code>document.date_created</code></dt>
<dd>When the document was written, as precise as the document
allows: <code>2013-03-20</code>, <code>2013-03</code>,
<code>2013</code>, or empty when unknown. Nothing is invented to
fill the gaps, and nothing is guessed from the publication date.
</dd>
<dt><code>document.banner</code></dt>
<dd>The marking line <strong>verbatim</strong> - what the
document says about itself, qualifiers and all. Intranet pages
often carry a ceiling banner (&ldquo;DYNAMIC PAGE - HIGHEST
POSSIBLE CLASSIFICATION IS ...&rdquo;) while their content is
portion-marked unclassified; the banner records the claim, not an
assessment. Empty means the artifact has no marking line - nothing
is reconstructed from portion marks.</dd>
<dt><code>document.classification</code></dt>
<dd>The curator's <strong>assessment</strong> of the content's
classification, as component values: the level
(<code>TOP SECRET</code>, <code>SECRET</code>,
<code>CONFIDENTIAL</code>, <code>UNCLASSIFIED</code>) and any
control systems (<code>COMINT</code>, <code>SI</code>,
<code>TK</code>, <code>STRAP 1</code> ...), judged from portion
marks. If a document contains multiple classifications, highest is
used here.</dd>
<dt><code>document.dissemination</code></dt>
<dd>The curator's assessment of the handling controls, as
component values (<code>FOUO</code>, <code>NOFORN</code>,
<code>ORCON</code>, <code>REL TO ...</code>). Assessed the same
way as the classification.</dd>
<dt><code>document.genre</code></dt>
<dd>The document's form, from a fixed vocabulary:
<code>slides</code>, <code>report</code>, <code>wiki</code>,
<code>manual</code>, <code>guide</code>,
<code>newsletter</code>, <code>other</code>.</dd>
<dt><code>document.series.name</code></dt>
<dd>For a document that prints a series roster (currently only in
SIDtoday): the series name verbatim, as the document prints it.
Documents with the same name are listed together on the document
page and under <a href="/browse/series">Series</a>.</dd>
<dt><code>document.series.part</code></dt>
<dd>This document's 1-based position in the printed roster. Order
comes from the artifact; parts of a series not yet in the archive
show as gaps in the numbering, never renumbered away.</dd>
<dt><code>document.programs</code></dt>
<dd>The codenames and programs the document names
(<code>XKEYSCORE</code>, <code>PRISM</code>, ...), as curated
values for browsing and search; the
<a href="/glossary">Glossary</a> explains many of them.</dd>
<dt><code>document.countries</code></dt>
<dd>The countries the document mentions, by their English names.
A mention, not a claim that the country was a target or a partner.
</dd>
<dt><code>document.people</code></dt>
<dd>The people who appear in the document: named in its text, or
recognised in its images - for example
<a href="/doc/gchq-mobile-networks-in-my-noc-world">Mobile Networks
in My NOC World</a> lists King Charles and Queen Camilla,
who appear only in a photograph on one of its slides. Usually
written &ldquo;Surname, Given name&rdquo;. A person appearing in
these documents implies nothing about them beyond that appearance.
The journalists who published the document are under
<code>publication[].reporters</code>, not here.</dd>
</dl>
<h4>Where it was published: <code>publication[]</code></h4>
<p>One entry per story or book that published (or reported) the
document. Browse-by-publication-date counts a document under its
<em>oldest</em> dated entry; a publication-date search matches
<em>any</em> entry's date - or, combined with a journalist or
publisher, that entry's date.</p>
<dl class="field-dict">
<dt><code>publication[].publisher</code></dt>
<dd>Who published it: a news outlet, a book's publisher, or for a
FOIA release the organization that obtained and released it.
Browsable under <a href="/browse/publisher">Publishers</a> - except
entries marked <code>reporting_only</code>.</dd>
<dt><code>publication[].title</code></dt>
<dd>The headline of the article, or the book's title. Shown on the
document page and names the story on
<a href="/articles">Articles</a>.</dd>
<dt><code>publication[].link</code></dt>
<dd>The URL of the article. It is the story's identity: documents
whose entries share a link were published in the same report and
are listed together on its article page.</dd>
<dt><code>publication[].date_published</code></dt>
<dd>When this publication appeared, in the same
<code>YYYY-MM-DD</code> / <code>YYYY-MM</code> /
<code>YYYY</code> forms as the document date.</dd>
<dt><code>publication[].reporters</code></dt>
<dd>The journalists bylined on the story, written &ldquo;Surname,
Given name&rdquo;. Browsable under
<a href="/browse/journalist">Journalists</a> and searchable with
<code>journalist:</code>.</dd>
<dt><code>publication[].contributors</code></dt>
<dd>People credited on the story without a byline - additional
reporting, research, technical analysis. Shown on the document
page and found by an all-fields search, but not listed among the
journalists.</dd>
<dt><code>publication[].note</code></dt>
<dd>Anything particular to this publication: that it showed only
some of the pages, published them in a different order, redacted
differently, or printed them in a book appendix.</dd>
<dt><code>publication[].version</code></dt>
<dd>When this publication released a different copy from the one
shown on the document page, the name of that copy in the
document's <code>versions/</code> folder; its entry then links to
&ldquo;view this copy&rdquo;. Without it, the entry published the
copy shown on the page.</dd>
<dt><code>publication[].doc_source</code></dt>
<dd>The web page where this publication presented the document
(the outlet's own document page, a DocumentCloud page), linked as
&ldquo;document page&rdquo;. Where a copy came from is recorded
here, on the publication that released it, not on the
document.</dd>
<dt><code>publication[].doc_url</code></dt>
<dd>The direct link to the PDF file this publication released,
linked as &ldquo;original PDF&rdquo;.</dd>
<dt><code>publication[].reporting_only</code></dt>
<dd><code>true</code> when the outlet reported the story without
publishing the document - a reporting partner. The entry keeps its
byline and its article, but the outlet is not counted as one of
the document's publishers and no copy is linked from it.</dd>
</dl>
<h4>Other copies: <code>versions</code></h4>
<p>Other published copies of a document (e.g. less redacted, with extra
pages, or a different scan) are kept in its <code>versions/</code>
folder, one sub-folder per copy, each with its own PDF, text and page
images. The <code>versions</code> object adds details about them,
keyed by the copy's folder name.</p>
<dl class="field-dict">
<dt><code>versions.{name}.changed_pages</code></dt>
<dd>The pages of that copy, by its own page numbers, that differ
from the copy shown on the document page. They are linked as
&ldquo;changed pages&rdquo; next to the copy.</dd>
<dt><code>versions.{name}.pdf_sha256</code></dt>
<dd>The SHA-256 of that copy's PDF.</dd>
</dl>
</details>
</section>
<section id="download-archive">
<h2>Downloading the archive</h2>
<p>Each release comes in two packages:</p>
<ul>
<li><strong>full</strong>, ~18 GB: all the PDFs, page images (the
PDF pages as JPEG files), extracted texts, and metadata,</li>
<li><strong>pdfs</strong>, ~3.5 GB: the same without the page
images.</li>
</ul>
<p>Both extract to <code>data/</code> and <code>project/</code>, ready
to <a href="#own-copy">run your own copy</a>. Each release also
includes the source of the server version it was published with, and
the last six releases stay online. For only the extracted texts and
metadata (~58 MB), see <a href="#downloading-texts">below</a>.</p>
<h3 id="downloading-all-data">Downloading all data</h3>
<h4>HTTP</h4>
<p><a href="https://data.snowchive.org/files/">https://data.snowchive.org/files/</a></p>
<details>
<summary>See a curl example</summary>
<pre>
# download the latest release:
rel=$(curl -fsS https://data.snowchive.org/files/latest.txt)
base="https://data.snowchive.org/files/$rel"
curl -f -C - -O "$base/snowchive-$rel-full.tar" # "pdfs" instead of "full" for no page images
# verify (optional):
curl -fsS -O "$base/SHA256SUMS" -O "$base/SHA256SUMS.gpg"
curl -fsS -O "https://libroot.org/public/libroot.asc"
# "gpg --show-keys libroot.asc" should show 3B054BC2E7B2CC825FF23E94108F97B8FC5B2EBC
gpg --import libroot.asc
gpg --verify SHA256SUMS.gpg SHA256SUMS
sha256sum --check --ignore-missing SHA256SUMS
# extract to ./snowchive-data/:
mkdir -p snowchive-data &amp;&amp; tar -xf "snowchive-$rel-full.tar" -C snowchive-data
</pre>
</details>
<h4>Torrent</h4>
<p>Each release folder on
<a href="https://data.snowchive.org/files/">https://data.snowchive.org/files/</a>
has a <code>.torrent</code> for both packages.</p>
<h3 id="downloading-texts">Downloading only extracted texts and metadata</h3>
<p>Repository for the extracted texts and metadata:
<a href="https://git.disroot.org/libroot/snowchive-meta">https://git.disroot.org/libroot/snowchive-meta</a></p>
<pre>
git clone https://git.disroot.org/libroot/snowchive-meta
cd snowchive-meta/data
# grep only the extracted texts:
grep "fiber optic" -ri --include=*.md
# grep only the metadata:
grep "fiber optic" -ri --include=*.json
</pre>
</section>
<section id="own-copy">
<h2>Can I run my own copy or mirror this?</h2>
<p>Yes. The server software behind this archive is free software with
no database and no external dependencies, designed to run offline or,
for example, as a Tor hidden service.</p>
<p>First, get a <a href="#downloading-all-data">release package</a>
and extract it: <strong>full</strong> for everything, or
<strong>pdfs</strong> for a site that works the same but shows no page
images.</p>
<p>Then get the server (Go required). Each release folder has the
source of the server version it was published with,
<code>snowchive-server-&lt;commit&gt;.tar.gz</code>, covered by the same
<code>SHA256SUMS</code> - the version known to work with that data:</p>
<pre>
tar -xzf snowchive-server-&lt;commit&gt;.tar.gz
cd snowchive-server-&lt;commit&gt;
go run . -root=/path/to/snowchive-data/ -listen=127.0.0.1:8081
</pre>
<p>Or clone the latest version from the
<a href="https://git.disroot.org/libroot/snowchive-server">snowchive-server</a>
repository:</p>
<pre>
git clone https://git.disroot.org/libroot/snowchive-server
cd snowchive-server
go run . -root=/path/to/snowchive-data/ -listen=127.0.0.1:8081
</pre>
<p>Recommended minimal setup for production instances:</p>
<pre>
go build -ldflags "-X main.buildCommit=$(git rev-parse --short HEAD)" -o snowchive .
./snowchive -root=/path/to/snowchive-data/ -listen=127.0.0.1:8081 -base=https://archive.example.com
</pre>
<p>System requirements: at least 300 MB of RAM (1 GB or more
recommended for production).</p>
<p>The maintainers of this project do not endorse any public Snowchive
instance other than the official snowchive.org instance (unless you
are a reputable organization that wants to host a public
instance).</p>
<p>See the
<a href="https://git.disroot.org/libroot/snowchive-server">snowchive-server README</a>
for the full documentation.</p>
</section>
<section id="contact">
<h2>Contact</h2>
<p>If the archive is missing a document, or some metadata is wrong,
please submit an issue on the project repository:
<a href="https://git.disroot.org/libroot/snowchive-meta/issues">https://git.disroot.org/libroot/snowchive-meta/issues</a></p>
<p>You can also report issues and errors or give feedback directly to
the project maintainers via email:
<a href="https://libroot.org/about">https://libroot.org/about</a></p>
<p>Please submit server-related issues to the Snowchive server
repository:
<a href="https://git.disroot.org/libroot/snowchive-server/issues">https://git.disroot.org/libroot/snowchive-server/issues</a></p>
</section>
<section id="contributing">
<h2>Contributing</h2>
<p>We welcome anyone who wants to contribute to the project, for
example by fixing errors in the metadata or in the extracted texts.
Please see the <a href="#contact">Contact</a> section above.</p>
<p><small>Running this project doesn't cost much, but if you wish, you
can send a donation to:</small></p>
<dl class="field-dict">
<dt><small>XMR</small></dt>
<dd><small><code style="word-break:break-all;">85gTBgYuLB4Mvo1ZCGo62126hBQQ678UeESTuSi3L9SECHXr6S7mXE28yAKvURgdJneGKqbPDcBydFfTkK7H5bvh2WXE4oT</code></small></dd>
<dt><small>BTC</small></dt>
<dd><small><code style="word-break:break-all;">bc1q5f5mc4pytajrn2xrp3lsftx64fw6hsdxxvs3ha</code></small></dd>
</dl>
</section>
<section id="thanks">
<h2>Thank you</h2>
<p>We want to thank</p>
<ul>
<li>all the journalists, reporters, and other people who broke the
stories or helped bring them to light,</li>
<li>other archives and the people behind them, such as
<ul>
<li>University of Toronto's and Canadian Journalists for
Free Expression's The Snowden Surveillance Archive (shut
down in mid-2024),</li>
<li>Courage Foundation's Snowden Revelations (shut down in
early 2022),</li>
</ul>
</li>
<li>organizations, resources and blogs such as
<ul>
<li><a href="https://www.eff.org/nsa-spying/nsadocs">EFF's NSA document collection</a>,</li>
<li><a href="https://www.aclu.org/nsa-documents-released-to-the-public-since-june-2013">ACLU's NSA document collection</a>,</li>
<li><a href="https://christopher-parsons.com/resources/the-sigint-summaries/">Christopher Parsons' SIGINT Summaries</a>,</li>
<li><a href="https://electrospaces.net">Electrospaces</a>,</li>
<li><a href="https://github.com/iamcryptoki/snowden-archive">iamcryptoki/snowden-archive</a>,</li>
<li><a href="https://cryptome.org">Cryptome</a>,</li>
</ul>
</li>
<li>and of course, Edward Snowden himself.</li>
</ul>
</section>
</div>
{{ end }}
{{ template "base.html" . }}

25
templates/article.html Normal file
View file

@ -0,0 +1,25 @@
{{ define "title" }}{{ with .Article.Title }}{{ . }}{{ else }}{{ if .Article.Book }}Book{{ else }}Article{{ end }} at {{ .Article.Host }}{{ end }} - Snowden Archive{{ end }}
{{ define "content" }}
<header class="list-header article-header">
<p class="muted article-kicker">{{ if .Article.Book }}Book{{ else }}Article{{ end }}</p>
<h1>{{ if .Article.Title }}{{ .Article.Title }}{{ else }}<span class="muted">[untitled]</span> {{ .Article.Host }}{{ end }}</h1>
<p class="article-byline muted">
{{ with .Article.Publisher }}{{ if $.PublisherURL }}<a href="{{ $.PublisherURL }}" class="meta-link-soft">{{ . }}</a>{{ else }}{{ . }}{{ end }}{{ end }}
{{ with .Article.Date }} &middot; {{ . }}{{ end }}
{{ if .Article.Journalists }} &middot;
{{ facetLinks (dict "Vals" .Article.Journalists "Cat" "journalist" "Class" "meta-link-soft" "QS" .BrowseQS) }}
{{ end }}
</p>
<p>
<a href="{{ .Article.Link }}" target="_blank" rel="noopener" class="doc-top-btn doc-pdf-link">{{ if .Article.Book }}About the book at{{ else }}Read the article at{{ end }} {{ .Article.Host }} &nearr;</a>
</p>
</header>
<h2>Documents published with this {{ if .Article.Book }}book{{ else }}article{{ end }} <small class="count-note">{{ .Total }}</small></h2>
{{ if .Cards.Docs }}
{{ template "doc-cards" .Cards }}
{{ else }}
<p class="muted">No documents found for this {{ if .Article.Book }}book{{ else }}article{{ end }}.</p>
{{ end }}
{{ end }}
{{ template "base.html" . }}

39
templates/articles.html Normal file
View file

@ -0,0 +1,39 @@
{{ define "title" }}Articles - Snowden Archive{{ end }}
{{ define "content" }}
<header class="list-header">
<h1>Articles
<small class="count-note">{{ .Total }} articles{{ with .Books }} and {{ . }} book{{ if ne . 1 }}s{{ end }}{{ end }}</small>
</h1>
</header>
<p class="muted tab-desc">
The news stories, and the few books, these documents were published with, newest first.
A document covered by several stories appears under each.
</p>
{{ if .Groups }}
{{ if .GroupsHTML }}{{ .GroupsHTML }}{{ else }}{{ template "article-groups" . }}{{ end }}{{ else }}
<p class="muted">No articles.</p>
{{ end }}
{{ end }}
{{ template "base.html" . }}
{{ define "article-groups" }}{{ range .Groups }}
<section class="article-year">
<h2>{{ .Year }} <small class="count-note">{{ len .Articles }}</small></h2>
<ul class="article-list">
{{ range .Articles }}
<li class="article-item">
<span class="article-date faint">{{ if .Date }}{{ .Date }}{{ else }}-{{ end }}</span>
<span class="article-main">
<a href="/article/{{ .Key }}" class="article-title">{{ if .Title }}{{ .Title }}{{ else }}<span class="muted">[untitled]</span> {{ .Host }}{{ end }}</a>
<span class="article-meta muted">
{{ if .Book }}<span class="article-kind">Book</span> &middot; {{ end }}{{ with .Publisher }}{{ . }}{{ end }}{{ if and .Publisher .Journalists }} &middot; {{ end }}{{ range $i, $j := .Journalists }}{{ if $i }}, {{ end }}{{ $j }}{{ end }}
&middot; {{ len .DocIDs }} doc{{ if ne (len .DocIDs) 1 }}s{{ end }}
</span>
</span>
</li>
{{ end }}
</ul>
</section>
{{ end }}
{{ end }}

32
templates/assemblies.html Normal file
View file

@ -0,0 +1,32 @@
{{ define "title" }}Assemblies - Snowden Archive{{ end }}
{{ define "content" }}
<header class="list-header">
<p class="crumb"><a href="/explore">&larr; Explore</a></p>
<h1>Assemblies
<small class="count-note">{{ len .Assemblies }}</small>
</h1>
</header>
<p class="muted tab-desc">
Sets of pages this archive put together from documents and copies published
separately, so they can be read in one place. An assembly is our own
arrangement, not a document that was leaked or published in this form.
</p>
{{ if .Assemblies }}
<ul class="assembly-list">
{{ range .Assemblies }}
<li>
{{ if not $.TextOnly }}<a href="/explore/assemblies/{{ .ID }}" class="assembly-thumb"><img src="{{ (index .Slides 0).Image }}"{{ (index .Slides 0).Size.Attr }} alt="" loading="lazy"></a>{{ end }}
<div>
<a href="/explore/assemblies/{{ .ID }}"><strong>{{ .Title }}</strong></a>
<div class="muted">{{ len .Slides }} slide{{ if ne (len .Slides) 1 }}s{{ end }} from {{ .Copies }} cop{{ if eq .Copies 1 }}y{{ else }}ies{{ end }}</div>
{{ with .Note }}<p class="assembly-note">{{ . }}</p>{{ end }}
</div>
</li>
{{ end }}
</ul>
{{ else }}
<p class="muted">No assemblies yet.</p>
{{ end }}
{{ end }}
{{ template "base.html" . }}

60
templates/assembly.html Normal file
View file

@ -0,0 +1,60 @@
{{ define "title" }}{{ .Assembly.Title }} - Snowden Archive{{ end }}
{{ define "assembly-credit" }}{{ with .Source }}{{ . }} &middot; {{ end }}<a href="{{ .Path }}">{{ .DocTitle }}</a>{{ with .Version }} (copy: {{ . }}){{ end }}, p. {{ .Page }}{{ end }}
{{ define "content" }}
{{ $a := .Assembly }}{{ $n := len $a.Slides }}
<header class="list-header">
<p class="crumb"><a href="/explore/assemblies">&larr; Assemblies</a></p>
<h1>{{ $a.Title }} <small class="count-note">{{ $n }} slide{{ if ne $n 1 }}s{{ end }}</small></h1>
<p class="muted">Put together by this archive from {{ $a.Copies }} separately published cop{{ if eq $a.Copies 1 }}y{{ else }}ies{{ end }}. This is our arrangement, not a document in its own right.</p>
{{ with $a.Note }}<p class="assembly-note">{{ . }}</p>{{ end }}
</header>
{{ if .TextOnly }}
<p class="muted textonly-note">Text-only mode is on - page images are not loaded. <a href="/preferences">Preferences</a></p>
<ul class="page-link-list">
{{ range $a.Slides }}
<li><a href="{{ .Image }}" target="_blank" rel="noopener">Slide {{ .Pos }}</a> <span class="muted">{{ template "assembly-credit" . }}</span>
{{- with .Note }} <span class="muted">- {{ . }}</span>{{ end }}
{{- range .Alts }}<br><a href="{{ .Image }}" target="_blank" rel="noopener">Other copy</a> <span class="muted">{{ template "assembly-credit" . }}</span>{{ end }}
</li>
{{ end }}
</ul>
{{ else }}
<ul class="page-sheet assembly-sheet">
{{ range $s := $a.Slides }}
<li>
<a href="#lb-{{ $s.Pos }}" title="Open slide {{ $s.Pos }} full screen">
<img src="{{ $s.Image }}"{{ $s.Size.Attr }} alt="Slide {{ $s.Pos }}" loading="lazy">
<span class="page-num faint">{{ $s.Pos }}</span>
</a>
<p class="assembly-cap">{{ template "assembly-credit" $s }}
{{- range $k, $alt := $s.Alts }}<br><a href="#lb-a{{ $s.Pos }}-{{ add $k 1 }}">Other copy{{ with $alt.Source }}: {{ . }}{{ end }}</a>{{ end }}
</p>
</li>
{{ end }}
</ul>
{{ range $i, $s := $a.Slides }}
<div class="lightbox" id="lb-{{ $s.Pos }}" aria-label="Slide {{ $s.Pos }} of {{ $n }}">
<a class="lb-backdrop" href="#lb-closed" tabindex="-1" aria-hidden="true"></a>
<a class="lb-close" href="#lb-closed" title="Close">&times;</a>
{{ if $i }}<a class="lb-prev" href="#lb-{{ $i }}" title="Previous slide">&lsaquo;</a>{{ end }}
{{ if lt $s.Pos $n }}<a class="lb-next" href="#lb-{{ add $s.Pos 1 }}" title="Next slide">&rsaquo;</a>{{ end }}
<img src="{{ $s.Image }}"{{ $s.Size.Attr }} alt="Slide {{ $s.Pos }} of {{ $n }}" loading="lazy" decoding="async">
<span class="lb-counter lb-caption">{{ $s.Pos }} / {{ $n }} &middot; {{ template "assembly-credit" $s }}{{ with $s.Note }} &middot; {{ . }}{{ end }}{{ range $k, $alt := $s.Alts }} &middot; <a href="#lb-a{{ $s.Pos }}-{{ add $k 1 }}">Other copy{{ with $alt.Source }}: {{ . }}{{ end }}</a>{{ end }}</span>
</div>
{{ range $k, $alt := $s.Alts }}
<div class="lightbox lb-alt" id="lb-a{{ $s.Pos }}-{{ add $k 1 }}" aria-label="Other copy of slide {{ $s.Pos }}">
<a class="lb-backdrop" href="#lb-closed" tabindex="-1" aria-hidden="true"></a>
<a class="lb-close" href="#lb-closed" title="Close">&times;</a>
<img src="{{ $alt.Image }}"{{ $alt.Size.Attr }} alt="Other copy of slide {{ $s.Pos }}" loading="lazy" decoding="async">
<span class="lb-counter lb-caption">Other copy of slide {{ $s.Pos }} &middot; {{ template "assembly-credit" $alt }} &middot; <a href="#lb-{{ $s.Pos }}">Back to slide {{ $s.Pos }}</a></span>
</div>
{{ end }}
{{ end }}
{{ end }}
{{ end }}
{{ template "base.html" . }}

35
templates/base.html Normal file
View file

@ -0,0 +1,35 @@
<!DOCTYPE html>
<html lang="en"{{ if .Theme }} data-theme="{{ .Theme }}"{{ end }}>
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>{{ block "title" . }}Snowden Archive{{ end }}</title>
<link rel="stylesheet" href="{{ asset "style.css" }}">
<script src="{{ asset "main.js" }}" defer></script>
{{ block "head" . }}{{ end }}
</head>
<body>
<a class="skip-link" href="#main">Skip to main content</a>
<nav class="site-nav">
<a href="/" class="{{ if eq .Nav "home" }}active{{ end }}">Home</a>
<a href="/browse" class="{{ if eq .Nav "browse" }}active{{ end }}">Browse</a>
<a href="/search" class="{{ if eq .Nav "search" }}active{{ end }}">Search</a>
<a href="/articles" class="{{ if eq .Nav "articles" }}active{{ end }}">Articles</a>
<a href="/glossary" class="{{ if eq .Nav "glossary" }}active{{ end }}">Glossary</a>
<a href="/random">Random</a>
<a href="/about" class="{{ if eq .Nav "about" }}active{{ end }}">About</a>
<a href="/pinned" class="nav-right{{ if eq .Nav "pinned" }} active{{ end }}">Pinned</a>
<a href="/preferences" class="{{ if eq .Nav "preferences" }}active{{ end }}">Preferences</a>
</nav>
{{ if .ShowCatBoxes }}{{ template "cat-boxes" . }}{{ end }}
<main id="main" tabindex="-1">
{{ block "content" . }}{{ end }}
</main>
<footer>
{{ $b := buildInfo }}
<p>Snowden Archive &middot; <a href="/changelog">changelog</a>
<span class="build-info faint">&middot; rev {{ $b.Commit }}</span>
</p>
</footer>
</body>
</html>

155
templates/browse.html Normal file
View file

@ -0,0 +1,155 @@
{{ define "title" }}{{ if eq .Collection "foia" }}FOIA{{ else if eq .Collection "sidtoday-archive" }}SIDtoday Archive{{ else }}Reported{{ end }} documents - Snowden Archive{{ end }}
{{ define "content" }}
<hr class="section-rule">
<nav class="collection-tabs" aria-label="Collections">
<a href="/browse"
class="collection-tab {{ if eq .Collection "news" }}active-tab{{ end }}"
{{ if eq .Collection "news" }}aria-current="page"{{ end }}>
Reported <span class="tab-count">{{ index .CollectionCounts "news" }}</span>
</a>
<a href="/browse?collection=sidtoday-archive"
class="collection-tab {{ if eq .Collection "sidtoday-archive" }}active-tab{{ end }}"
{{ if eq .Collection "sidtoday-archive" }}aria-current="page"{{ end }}>
SIDtoday Archive <span class="tab-count">{{ .SIDTodayCount }}</span>
</a>
{{ if or .Prefs.IncludeFOIA (eq .Collection "foia") }}
<a href="/browse?collection=foia"
class="collection-tab {{ if eq .Collection "foia" }}active-tab{{ end }}"
{{ if eq .Collection "foia" }}aria-current="page"{{ end }}>
FOIA <span class="tab-count">{{ index .CollectionCounts "foia" }}</span>
</a>
{{ end }}
</nav>
{{ if eq .Collection "sidtoday-archive" }}
<header class="tab-header">
<h1 class="tab-title">SIDtoday Archive <small>({{ .Total }} issues)</small></h1>
<p class="tab-desc">
All NSA SIDtoday newsletters in chronological order. This section
includes The Intercept's bulk release plus issues that were individually
reported by media (marked <span class="issue-reported">reported</span>; those
also appear in the <a href="/browse">Reported</a> collection).
</p>
{{ with .SeriesList }}
<details class="series-index">
<summary>Series <span class="count">{{ len . }}</span> <span class="muted pref-note">- recurring columns and multi-part runs</span></summary>
<ul class="series-index-list">
{{ range . }}<li><a href="{{ .URL }}">{{ .Name }}</a> <span class="count">{{ .Count }}</span></li>{{ end }}
</ul>
<p class="muted pref-note"><a href="/browse/series">Full series table</a> - sortable, with per-collection counts.</p>
</details>
{{ end }}
<div class="browse-controls{{ if eq .View "table" }} browse-controls-spaced{{ end }}">
<div class="view-toggle">
<a href="/browse?collection=sidtoday-archive" class="{{ if and (ne .View "card") (ne .View "table") }}active-toggle{{ end }}">Timeline</a>
<a href="/browse?collection=sidtoday-archive&amp;view=card" class="{{ if eq .View "card" }}active-toggle{{ end }}">Card View</a>
<a href="/browse?collection=sidtoday-archive&amp;view=table" class="{{ if eq .View "table" }}active-toggle{{ end }}">Table View</a>
</div>
</div>
</header>
{{ if eq .View "card" }}
<div class="sort-bar-spaced">{{ template "sort-bar" (dict "SortURL" .BaseURL "SortBy" .Sort "Order" .Order "ShowRelevance" false "DefaultSort" .DefaultSort) }}</div>
<h2 class="sr-only">Documents</h2>
{{ template "doc-cards" .Cards }}
{{ template "pagination" . }}
{{ else if eq .View "table" }}
{{ template "doc-table" . }}
{{ else }}
{{ if .ArchiveYears }}
{{ if .TimelineHTML }}{{ .TimelineHTML }}{{ else }}{{ template "archive-timeline" . }}{{ end }}{{ else }}
<p class="muted">No documents in this collection yet.</p>
{{ end }}
{{ end }}
{{ else }}
<header class="browse-header">
<h1 class="browse-title">
{{ if eq .Collection "foia" }}FOIA Documents{{ else }}Reported Documents{{ end }}
<small class="count-note">({{ .Total }} total)</small>
</h1>
{{ $cardURL := .SortURL }}
{{ $tableURL := print .SortURL "view=table" }}
{{ if .Sort }}
{{ $cardURL = print $cardURL "sort=" .Sort }}
{{ $tableURL = print $tableURL "&sort=" .Sort }}
{{ if .Order }}
{{ $cardURL = print $cardURL "&order=" .Order }}
{{ $tableURL = print $tableURL "&order=" .Order }}
{{ end }}
{{ end }}
<div class="browse-controls">
<div class="view-toggle">
<a href="{{ $cardURL }}" class="{{ if ne .View "table" }}active-toggle{{ end }}">Card View</a>
<a href="{{ $tableURL }}" class="{{ if eq .View "table" }}active-toggle{{ end }}">Table View</a>
</div>
</div>
</header>
<p class="tab-desc collection-desc">
{{ if eq .Collection "foia" }}
Documents released by government agencies under Freedom of Information Act
requests, rather than through the Snowden disclosures - obtained by
journalists, researchers, and transparency organizations and added here
for context.
{{ else }}
Documents published alongside news reporting on the Snowden disclosures.
{{ end }}
</p>
{{ if ne .View "table" }}
{{ template "sort-bar" (dict "SortURL" .SortURL "SortBy" .Sort "Order" .Order "ShowRelevance" false "DefaultSort" .DefaultSort) }}
{{ end }}
{{ if eq .Collection "news" }}
{{ template "type-filter" (dict
"Action" "/browse"
"Params" .Params
"AllTypes" .AllTypes
"SelectedTypes" .SelectedTypes
) }}
{{ end }}
{{ if eq .View "table" }}
{{ template "doc-table" . }}
{{ else }}
<h2 class="sr-only">Documents</h2>
{{ template "doc-cards" .Cards }}
{{ template "pagination" . }}
{{ end }}
{{ end }}
{{ end }}
{{ template "base.html" . }}
{{ define "archive-timeline" }}<nav class="year-nav" aria-label="Jump to year">
<span class="year-nav-label">Jump to year:</span>
{{ range .ArchiveYears }}<a href="#y{{ .Year }}">{{ .Year }}</a>{{ end }}
</nav>
{{ range .ArchiveYears }}
<section class="archive-year" id="y{{ .Year }}">
<h2>{{ .Year }} <small>({{ .Count }} issue{{ if ne .Count 1 }}s{{ end }})</small></h2>
{{ range .Months }}
<div class="archive-month">
{{ with .Name }}<h3>{{ . }}</h3>{{ end }}
<ul class="archive-issues">
{{ range .Docs }}
<li>
<span class="issue-date">{{ with .DocMeta.DateCreated }}{{ . }}{{ else }}-{{ end }}</span>
<a href="/doc/{{ .ID }}">{{ .Title }}</a>
<a href="/files/{{ .ID }}/{{ pathEscape .PDFFile }}" target="_blank" rel="noopener" class="doc-pdf" title="Open PDF">[pdf]</a>
{{ if .ReportedIssue }}<span class="issue-reported" title="Individually reported - also in the Reported collection">reported</span>{{ end }}
</li>
{{ end }}
</ul>
</div>
{{ end }}
</section>
{{ end }}
{{ end }}

59
templates/category.html Normal file
View file

@ -0,0 +1,59 @@
{{ define "title" }}Browse by {{ .Title }} - Snowden Archive{{ end }}
{{ define "content" }}
<h1>Browse by {{ .Title }}</h1>
{{ with .Blurb }}<p class="tab-desc cat-blurb">{{ . }}</p>{{ end }}
{{ if .SeeAlso }}
<p class="muted see-also">See also: {{ range $i, $l := .SeeAlso }}{{ if $i }} &middot; {{ end }}<a href="{{ $l.URL }}">{{ $l.Text }}</a>{{ end }}.</p>
{{ end }}
{{ if .ShowAZ }}
<nav class="az-nav" aria-label="Jump to letter">
<span class="az-label">Jump to:</span>
{{ range .Letters }}<a href="#{{ azID . }}">{{ . }}</a>{{ end }}
</nav>
{{ end }}
{{ if .CollCols }}
<input type="checkbox" id="coll-cols" class="coll-cols-checkbox"{{ if .ShowCollCols }} checked{{ end }}>
<label for="coll-cols" class="coll-cols-label">
<span class="collapsed-text">[+] Show per-type counts</span>
<span class="expanded-text">[&minus;] Hide per-type counts</span>
<span class="sr-only">Toggle the per-type count columns</span>
</label>
{{ end }}
<div class="table-scroll">
<table class="cat-table cat-table-colls">
<thead>
<tr>{{ range .Heads }}{{ template "sort-th" . }}{{ end }}</tr>
</thead>
<tbody>
{{ if .RowsHTML }}{{ .RowsHTML }}{{ else }}{{ template "category-rows" . }}{{ end }} </tbody>
</table>
</div>
{{ end }}
{{ template "base.html" . }}
{{ define "category-row" }}
<tr>
<td>
<a href="{{ .URL }}">
{{ .Name }}
</a>
{{- with .Note }} <small class="muted cat-note">{{ . }}</small>{{ end }}
</td>
<td class="num-col">{{ .Count }}</td>
<td class="num-col coll-col">{{ if .News }}{{ .News }}{{ else }}<span class="faint">-</span>{{ end }}</td>
<td class="num-col coll-col">{{ if .Archive }}{{ .Archive }}{{ else }}<span class="faint">-</span>{{ end }}</td>
{{- if not .NoFOIA }}
<td class="num-col coll-col">{{ if .FOIA }}{{ .FOIA }}{{ else }}<span class="faint">-</span>{{ end }}</td>
{{- end }}
</tr>
{{ end }}
{{ define "category-az-row" }}
<tr class="az-row"><th id="{{ azID . }}" colspan="5">{{ . }}</th></tr>
{{ end }}
{{ define "category-rows" }}{{ $prevL := "" }}{{ range .Entries }}{{ if $.ShowAZ }}{{ if ne .FirstLetter $prevL }}{{ template "category-az-row" .FirstLetter }}{{ $prevL = .FirstLetter }}{{ end }}{{ end }}{{ template "category-row" . }}{{ end }}{{ end }}

25
templates/changelog.html Normal file
View file

@ -0,0 +1,25 @@
{{ define "title" }}Changelog - Snowden Archive{{ end }}
{{ define "content" }}
<h1>Changelog
{{ if .Total }}<small class="count-note">{{ .Total }} entries</small>{{ end }}
</h1>
{{ if .Entries }}
<ul class="changelog-list">
{{ range .Entries }}
<li class="changelog-entry">
<span class="changelog-date faint">{{ if ge (len .Date) 10 }}{{ slice .Date 0 10 }}{{ else }}{{ .Date }}{{ end }}</span>
<span class="changelog-type changelog-type-{{ .Type }}">{{ .Type }}</span>
{{ if and .ID (ne .Type "removed") }}
<a href="{{ .DocURL }}">{{ .Headline }}</a>
{{ else }}
{{ .Headline }}
{{ end }}
</li>
{{ end }}
</ul>
{{ else }}
<p class="muted">No changes recorded.</p>
{{ end }}
{{ end }}
{{ template "base.html" . }}

View file

@ -0,0 +1,50 @@
{{ define "title" }}Classification patterns - Snowden Archive{{ end }}
{{ define "content" }}
<header class="list-header">
<p class="crumb"><a href="/explore">&larr; Explore</a></p>
<h1>Classification patterns</h1>
</header>
<p class="muted">Two views of the marking metadata. Every number counts the
documents carrying that exact classification component. A document carrying
several markings counts once per marking.</p>
<h2>Mix by document year</h2>
<div class="cls-legend">
{{ range $i, $v := .Levels }}<span class="cls-key"><span class="cls-swatch seg-{{ $i }}"></span>{{ $v }}</span>{{ end }}
</div>
{{ range .Strips }}
<div class="cls-row">
<a class="cls-year" href="{{ .URL }}">{{ .Year }}</a>
<span class="cls-strip">
{{ range .Segs }}
<a class="cls-seg {{ .Class }}" href="{{ .URL }}" style="width: {{ .Pct }}%" title="{{ .Value }}: {{ .Count }}"></a>
{{ end }}
</span>
<span class="cls-total faint">{{ .Total }}</span>
</div>
{{ end }}
{{ with .Sparse }}<p class="muted"><small>Not shown, with fewer than {{ $.MinMarks }} markings each:
{{ range $i, $y := . }}{{ if $i }}, {{ end }}<a href="{{ $y.URL }}">{{ $y.Year }}</a> ({{ $y.Total }}){{ end }}.</small></p>{{ end }}
<h2>Agency &times; classification</h2>
<div class="table-scroll">
<table class="data-table heat-table">
<thead>
<tr><th>Agency</th>{{ range .Cols }}<th><a href="{{ .URL }}">{{ .Name }}</a></th>{{ end }}</tr>
</thead>
<tbody>
{{ range .HeatRows }}
<tr>
<td><a href="{{ .URL }}">{{ .Name }}</a></td>
{{ range .Cells }}
<td class="heat-cell">{{ if .URL }}<a href="{{ .URL }}">{{ .Count }}</a>{{ else }}<span class="faint">-</span>{{ end }}</td>
{{ end }}
</tr>
{{ end }}
</tbody>
</table>
</div>
{{ end }}
{{ template "base.html" . }}

38
templates/codewords.html Normal file
View file

@ -0,0 +1,38 @@
{{ define "title" }}Acronyms & codewords - Snowden Archive{{ end }}
{{ define "content" }}
<header class="list-header">
<p class="crumb"><a href="/explore">&larr; Explore</a></p>
<h1>Acronyms &amp; codewords</h1>
</header>
<p class="muted">ALL-CAPS tokens (3+ letters) extracted automatically from titles and
transcripts - acronyms, program names, compartments, org designators. This is a
text statistic, not curated metadata. A token links to a search for the word in any
letter case, across the metadata too, so the search can find more documents than the
capitals counted here.</p>
<div class="table-scroll">
<table class="data-table codeword-table">
<thead>
<tr>{{ range .Heads }}{{ template "sort-th" . }}{{ end }}</tr>
</thead>
<tbody>
{{ if .RowsHTML }}{{ .RowsHTML }}{{ else }}{{ template "codeword-rows" . }}{{ end }} </tbody>
</table>
</div>
{{ end }}
{{ template "base.html" . }}
{{ define "codeword-rows" }} {{ range .Rows }}
<tr>
<td><a href="{{ .SearchURL }}">{{ .Token }}</a>{{ if .ProgURL }} <a class="faint cw-prog" href="{{ .ProgURL }}" title="Also a curated program - its page tracks metadata mentions, including descriptions">program &rarr;</a>{{ end }}</td>
<td>{{ .Count }}</td>
<td>{{ .Docs }}</td>
<td>{{ if .First }}<a href="{{ .FirstURL }}">{{ .First }}</a>{{ else }}<span class="faint">-</span>{{ end }}</td>
<td>{{ if .Last }}<a href="{{ .LastURL }}">{{ .Last }}</a>{{ else }}<span class="faint">-</span>{{ end }}</td>
<td>{{ if .Gloss }}<a href="{{ .GlossURL }}">defined</a>{{ else }}<span class="faint">-</span>{{ end }}</td>
</tr>
{{ end }}
{{ end }}

272
templates/doc.html Normal file
View file

@ -0,0 +1,272 @@
{{ define "title" }}{{ .Doc.Title }} - Snowden Archive{{ end }}
{{ define "head" }}
<meta name="description" content="{{ .Doc.Description }}">
<link rel="schema.DC" href="http://purl.org/dc/elements/1.1/">
<meta name="DC.title" content="{{ .Doc.Title }}">
{{ if .Doc.DocMeta.Creator }}<meta name="DC.creator" content="{{ .Doc.DocMeta.Creator }}">{{ end }}
{{ if .Doc.DocMeta.DateCreated }}<meta name="DC.date" content="{{ .Doc.DocMeta.DateCreated }}">{{ end }}
{{ if .Doc.Description }}<meta name="DC.description" content="{{ .Doc.Description }}">{{ end }}
{{ range .Doc.DocMeta.Programs }}<meta name="DC.subject" content="{{ . }}">{{ end }}
{{ range .Doc.Tags }}<meta name="DC.subject" content="{{ . }}">{{ end }}
<meta name="DC.type" content="Text">
<meta name="DC.format" content="application/pdf">
<meta name="DC.identifier" content="{{ .DocAbsURL }}">
<meta name="DC.language" content="en">
<meta name="DC.publisher" content="Snowden Archive">
<meta property="og:type" content="article">
<meta property="og:site_name" content="Snowden Archive">
<meta property="og:title" content="{{ .Doc.Title }}">
{{ if .Doc.Description }}<meta property="og:description" content="{{ .Doc.Description }}">{{ end }}
<meta property="og:url" content="{{ .DocAbsURL }}">
{{ with .OGImageURL }}<meta property="og:image" content="{{ . }}">{{ end }}
<meta name="twitter:card" content="summary">
{{ if .APIEnabled }}<link rel="alternate" type="application/json" href="/api/doc/{{ .Doc.ID }}">{{ end }}
{{ .Embeds.JSONLDTag }}
{{ end }}
{{ define "content" }}
{{ $sidebar := .Prefs.InfoSidebar }}
<div class="doc-layout{{ if $sidebar }} info-sidebar{{ end }}">
{{ if $sidebar }}
<input type="checkbox" id="info-collapse" class="info-collapse-checkbox">
{{ end }}
<header class="doc-header">
<div class="header-main">
<div>
<h1>{{ .Doc.Title }}</h1>
<a href="/files/{{ .Doc.ID }}/{{ pathEscape .Doc.PDFFile }}" target="_blank" rel="noopener"
class="doc-top-btn doc-pdf-link" title="Open original PDF">Open PDF</a>
{{ with .Doc.Versions }}<a href="#publications" class="doc-top-btn" title="Other published copies of this document: less redacted, extra pages or a different scan">+{{ len . }} other cop{{ if eq (len .) 1 }}y{{ else }}ies{{ end }}</a>{{ end }}
<form method="POST" action="/pin/{{ .Doc.ID }}" class="inline-form">
{{ $back := print "/doc/" .Doc.ID }}
<input type="hidden" name="back" value="{{ $back }}">
<button type="submit" class="doc-top-btn pin-btn">{{ if .Pinned }}&#9733; Unpin{{ else }}&#9734; Pin{{ end }}</button>
</form>
{{ if or .SerialPrev .SerialNext }}
<nav class="serial-nav" aria-label="SIDtoday issues">
{{ with .SerialPrev }}<a href="/doc/{{ .ID }}" class="serial-prev" title="{{ .Title }}">&larr; Previous issue{{ with .DocMeta.DateCreated }} ({{ . }}){{ end }}</a>{{ end }}
{{ with .SerialNext }}<a href="/doc/{{ .ID }}" class="serial-next" title="{{ .Title }}">Next issue{{ with .DocMeta.DateCreated }} ({{ . }}){{ end }} &rarr;</a>{{ end }}
</nav>
{{ end }}
<div class="doc-info">
<strong>Creating agency:</strong> {{ with .Doc.DocMeta.Creator }}<a href="/browse/agency/{{ pathEscape . }}{{ $.BrowseQS }}">{{ . }}</a>{{ with agencyName . }} <small class="muted">{{ . }}</small>{{ end }}{{ else }}-{{ end }}<br>
<strong>Document date:</strong> {{ with .Doc.DocMeta.DateCreated }}{{ . }}{{ else }}-{{ end }}<br>
{{ if or (eq .Doc.Type "FOIA") (eq .Doc.Type "SIDTODAY") }}<strong>Type:</strong> {{ template "type-badge" .Doc.Type }}<br>{{ end }}
{{ if .Doc.Pages }}<strong>Pages:</strong> <a href="/doc/{{ .Doc.ID }}/pages">{{ .Doc.Pages }}</a><br>{{ end }}
{{ with .Doc.DocMeta.Genre }}<strong>Genre:</strong> <a href="/browse/genre/{{ pathEscape . }}{{ $.BrowseQS }}">{{ . }}</a><br>{{ end }}
{{ with .Doc.DocMeta.Banner }}<strong>Banner:</strong> <a href="/browse/banner/{{ pathEscape . }}{{ $.BrowseQS }}" class="doc-banner-link banner">{{ . }}</a><br>{{ end }}
<strong>Classification: </strong>
{{- facetLinks (dict "Vals" .Doc.DocMeta.Classification "Cat" "classification" "Sep" "space" "Class" "tag classification" "QS" $.BrowseQS) -}}
<br>
<strong>Dissemination: </strong>
{{- facetLinks (dict "Vals" .Doc.DocMeta.Dissemination "Cat" "dissemination" "Class" "doc-banner-link dissemination" "QS" $.BrowseQS) -}}
</div>
{{ with .SeriesGroup }}
<div class="rule-top series-box">
<strong>Series:</strong>
<a href="/browse/series/{{ pathEscape .Name }}{{ $.BrowseQS }}">{{ .Name }}</a>
<ol class="series-list">
{{ range .Docs }}
<li{{ if gt .DocMeta.Series.Part 0 }} value="{{ .DocMeta.Series.Part }}"{{ end }}>
{{ if eq .ID $.Doc.ID }}<strong aria-current="page">{{ .Title }}</strong>
{{ else }}<a href="/doc/{{ .ID }}">{{ .Title }}</a>{{ end }}
</li>
{{ end }}
</ol>
</div>
{{ end }}
{{ with .Assemblies }}
<div class="rule-top series-box">
<strong>Part of {{ if gt (len .) 1 }}assemblies{{ else }}an assembly{{ end }}:</strong>
{{ range $i, $a := . }}{{ if $i }}, {{ end }}<a href="/explore/assemblies/{{ $a.ID }}">{{ $a.Title }}</a>{{ end }}
</div>
{{ end }}
<div class="rule-top pub-block" id="publications">
{{- range $i, $pub := .Doc.Publication }}
<div class="pub-entry{{ if $i }} pub-sep{{ end }}">
{{- if or $pub.Link $pub.Publisher $pub.DatePublished }}
<div class="pub-head">
{{- if $pub.Link }}<a href="{{ $pub.Link }}" target="_blank" rel="noopener" class="pub-original">{{ with $pub.Publisher }}{{ . }}{{ else }}Read the original{{ end }} &nearr;</a>
{{- else }}{{ with $pub.Publisher }}<span class="pub-outlet">{{ . }}</span>{{ end }}{{ end -}}
{{- with $pub.DatePublished }}<span class="pub-date muted">{{ . }}</span>{{ end -}}
</div>
{{- end -}}
{{- if $pub.Title -}}
{{ $ak := articleKey $pub.Link }}
<div class="pub-title">{{ if $ak }}<a href="/article/{{ $ak }}"><em>{{ $pub.Title }}</em></a>{{ else }}<em>{{ $pub.Title }}</em>{{ end }}</div>
{{- end -}}
{{- if or $pub.Journalists $pub.Contributors }}
<div class="pub-people muted">
{{- range $j, $name := $pub.Journalists -}}
{{- if $j }} <span class="doc-dot">&middot;</span> {{ end -}}
<a href="/browse/journalist/{{ pathEscape $name }}{{ $.BrowseQS }}" class="meta-link">{{ $name }}</a>
{{- end -}}
{{- range $j, $name := $pub.Contributors -}}
{{- if or $j $pub.Journalists }} <span class="doc-dot">&middot;</span> {{ end -}}
<span class="meta-link">{{ $name }}</span>
{{- end -}}
</div>
{{- end -}}
{{- if $pub.ReportingOnly }}
<div class="pub-reporting-only muted">Reporting partner - published this story, not the document</div>
{{- end -}}
{{- with $pub.Note }}
<div class="pub-note">{{ . }}</div>
{{- end -}}
{{- $copy := $.Doc.VersionFor $pub.Version -}}
{{- $shown := and (not $pub.ReportingOnly) (not $pub.Version) $.Doc.Versions -}}
{{- if and (not $pub.ReportingOnly) (or $copy $pub.DocSource $pub.DocURL $shown) }}
<div class="pub-copy muted">
{{- if $shown }}<span class="pub-shown">the copy shown on this page</span>{{ end -}}
{{- with $copy }}<a href="/doc/{{ $.Doc.ID }}/versions/{{ .Slug }}">view this copy &rarr;</a>{{ if .ChangedPages }}<span class="changed-pages">changed pages: {{ template "changed-page-links" . }}</span>{{ end }}{{ end -}}
{{- with $pub.DocSource }}<a href="{{ . }}" target="_blank" rel="noopener">document page &nearr;</a>{{ end -}}
{{- with $pub.DocURL }}<a href="{{ . }}" target="_blank" rel="noopener">original PDF &nearr;</a>{{ end -}}
</div>
{{- end -}}
</div>
{{- end }}
{{- with .Doc.UnlinkedVersions }}
<div class="pub-entry pub-sep">
<div class="pub-title">Other copies</div>
<div class="pub-copy muted">
{{- range . }}<span><a href="/doc/{{ $.Doc.ID }}/versions/{{ .Slug }}">{{ .Slug }} &rarr;</a>{{ if .ChangedPages }} (changed pages: {{ template "changed-page-links" . }}){{ end }}</span>{{ end -}}
</div>
</div>
{{- end }}
</div>
<div class="rule-top">
<p>{{ .Doc.Description }}</p>
<div class="meta-expand">
<input type="checkbox" id="meta-toggle" class="meta-checkbox"{{ if .Prefs.OpenInfo }} checked{{ end }}>
<label for="meta-toggle" class="meta-label">
<span class="collapsed-text">[+] Show more info</span>
<span class="expanded-text">[-] Show less info</span>
<span class="sr-only">Toggle extended document metadata</span>
</label>
<div class="meta-content">
<div class="meta-grid">
<div class="meta-wide long-desc-container">
<strong>Long Description:</strong>
<div class="long-description-text">
{{- with .Doc.LongDescription }}{{ . }}{{ else }}<em>No extended description available.</em>{{ end -}}
</div>
</div>
<div>
<strong>Programs:</strong><br>
{{- facetLinks (dict "Vals" .Doc.DocMeta.Programs "Cat" "program" "Class" "tag" "AsList" .Prefs.ProgramsAsList "QS" $.BrowseQS) -}}
{{- if not (and .Doc.DocMeta.Programs .Prefs.ProgramsAsList) }}<br>{{ end -}}
<br>
<strong>People mentioned:</strong><br>
{{- facetLinks (dict "Vals" .Doc.DocMeta.People "Cat" "people" "Class" "meta-link" "QS" $.BrowseQS) -}}
<br><br>
<strong>Countries:</strong><br>
{{- facetLinks (dict "Vals" .Doc.DocMeta.Countries "Cat" "country" "Class" "meta-link" "QS" $.BrowseQS) -}}
<br><br>
<strong>Tags:</strong><br>
{{- facetLinks (dict "Vals" .Doc.Tags "Cat" "tag" "Class" "meta-link" "AsList" .Prefs.TagsAsList "QS" $.BrowseQS) -}}
<br><br>
<strong>Identifier:</strong><br>
<code class="checksum">{{ .Doc.ID }}</code>
{{ with .Doc.AkaID }}
<br><strong>Also known as:</strong><br>
{{ range $i, $aka := . }}{{ if $i }}<br>{{ end }}<code class="checksum">{{ $aka.ID }}</code>{{ with $aka.DocSource }} <a href="{{ . }}" target="_blank" rel="noopener">page &nearr;</a>{{ end }}{{ with $aka.DocURL }} <a href="{{ . }}" target="_blank" rel="noopener">PDF &nearr;</a>{{ end }}{{ end }}
{{ end }}
{{ with .Doc.PDFSha256 }}
<br><br>
<strong>PDF SHA-256:</strong><br>
<code class="checksum">{{ . }}</code>
{{ end }}
</div>
</div>
</div>
</div>
<details class="note-box note-subtle" id="note-box" hidden data-doc-id="{{ .Doc.ID }}" data-doc-title="{{ .Doc.Title }}">
<summary>My notes <span class="note-status faint" id="note-status"></span></summary>
<textarea id="note-text" rows="4" placeholder="Private notes about this document - stored only in this browser (localStorage). See all your notes on the Pinned page."></textarea>
</details>
</div>
</div>
</div>
</header>
{{ if $sidebar }}
<label for="info-collapse" class="info-toggle-label" title="Show/hide document info">
<span class="sr-only">Toggle the document info sidebar</span>
</label>
{{ end }}
<div class="doc-container">
{{ template "doc-viewport" . }}
</div>
</div>{{/* /.doc-layout */}}
{{ if and .Images (not .Prefs.TextOnly) }}
{{ template "lightbox-overlays" . }}
{{ end }}
<div class="doc-extra">
{{ with .Doc.ArchiveComment }}
<div class="archive-note">
<strong>Archive note:</strong> {{ . }}
</div>
{{ end }}
<span class="Z3988" title="{{ .Embeds.COinS }}"></span>
</div>
{{ if or .PublishedTogether .RelatedDocs }}
<div class="related-section">
{{ if .PublishedTogether }}
<details class="collapse-section"{{ if not $.Prefs.CollapsePubTogether }} open{{ end }}>
<summary>
<span class="collapse-ind" aria-hidden="true"></span>
<h2>Published in the Same Report
<small>{{ if .PublishedMore }}({{ .PublishedShown }} of {{ len .PublishedTogether }} documents){{ else }}({{ len .PublishedTogether }} documents){{ end }}</small>
</h2>
</summary>
{{ template "doc-cards" .PublishedCards }}
{{ with .PublishedMore }}{{ $one := eq (len .) 1 }}
{{ range . }}
<p class="pub-more"><a class="see-all" href="/article/{{ .Key }}">See all {{ len .DocIDs }} documents published with {{ if $one }}this article{{ else if .Title }}&ldquo;{{ .Title }}&rdquo;{{ else }}the article at {{ .Host }}{{ end }} &rarr;</a></p>
{{ end }}
{{ end }}
</details>
{{ end }}
{{ if .RelatedDocs }}
<details class="collapse-section"{{ if not $.Prefs.CollapseRelated }} open{{ end }}>
<summary>
<span class="collapse-ind" aria-hidden="true"></span>
<h2>Related Documents</h2>
</summary>
{{ template "doc-cards" .RelatedCards }}
</details>
{{ end }}
</div>
{{ end }}
{{ end }}
{{ template "base.html" . }}

50
templates/explore.html Normal file
View file

@ -0,0 +1,50 @@
{{ define "title" }}Explore - Snowden Archive{{ end }}
{{ define "content" }}
<h1>Explore the archive</h1>
<p class="tab-desc">
Ranked views and lists across the whole archive, which the
<a href="/browse">/browse</a> doesn't include.
</p>
<ul class="explore-list">
{{ range .Metrics }}
<li>
<a href="/explore/{{ .Slug }}">{{ .Title }}</a>
<span class="muted">- {{ .Blurb }}</span>
</li>
{{ end }}
<li>
<a href="/explore/portions">Portion markings</a>
<span class="muted">- The marking on each classified portion, read out of the OCRs (levels, compartments, and how often).</span>
</li>
<li>
<a href="/explore/classification">Classification patterns</a>
<span class="muted">- The marking mix across document years, and an agency x classification heat table.</span>
</li>
<li>
<a href="/explore/outlets">Publishers by year</a>
<span class="muted">- When each publisher released documents, and how much (i.e. documents and stories per publisher and year).</span>
</li>
<li>
<a href="/explore/ids">Technical IDs</a>
<span class="muted">- Every document id, PDF filename, size and SHA-256 checksum.</span>
</li>
<li>
<a href="/browse/genre">Genres</a>
<span class="muted">- How many documents of each kind the archive holds (slides, reports, wikis, manuals, guides, newsletters).</span>
</li>
{{ if $.HasAssemblies }}
<li>
<a href="/explore/assemblies">Assemblies</a>
<span class="muted">- Sets of pages this archive put together from documents and copies published separately, so they can be read in one place.</span>
</li>
{{ end }}
<li>
<a href="/explore/terms">Acronyms &amp; codewords</a>
<span class="muted">- Every ALL-CAPS token across the corpus, auto-extracted (e.g. acronyms, program names, compartments, org designators).</span>
</li>
</ul>
{{ end }}
{{ template "base.html" . }}

View file

@ -0,0 +1,35 @@
{{ define "title" }}{{ .Metric.Title }} - Snowden Archive{{ end }}
{{ define "content" }}
<p class="crumb"><a href="/explore">&larr; Explore</a></p>
<h1>{{ .Metric.Title }}</h1>
<p class="tab-desc">{{ .Metric.Blurb }}</p>
<p class="muted">Top {{ .Shown }} of {{ .Total }} documents.</p>
<div class="table-scroll">
<table class="cat-table explore-table">
<thead>
<tr>{{ range .Heads }}{{ template "sort-th" . }}{{ end }}</tr>
</thead>
<tbody>
{{ if .RowsHTML }}{{ .RowsHTML }}{{ else }}{{ template "explore-rows" . }}{{ end }} </tbody>
</table>
</div>
{{ end }}
{{ template "base.html" . }}
{{ define "explore-rows" }} {{ range .Rows }}
<tr>
<td class="num-col faint">{{ .Rank }}</td>
<td>
<a href="/doc/{{ .Doc.ID }}">{{ .Doc.Title }}</a>
{{ template "type-badge" .Doc.Type }}
</td>
<td class="explore-date">{{ with .Doc.DocMeta.DateCreated }}{{ . }}{{ else }}<span class="faint">-</span>{{ end }}</td>
{{ range .Vals }}
<td class="num-col">{{ . }}</td>
{{ end }}
</tr>
{{ end }}
{{ end }}

36
templates/glossary.html Normal file
View file

@ -0,0 +1,36 @@
{{ define "title" }}Glossary - Snowden Archive{{ end }}
{{ define "content" }}
<h1>Glossary {{ if .Total }}<small class="muted">({{ .Total }} terms)</small>{{ end }}</h1>
{{ if .Groups }}
<nav class="letter-nav" aria-label="Jump to letter">
{{ range .Groups }}<a href="#g{{ .Letter }}">{{ .Letter }}</a>{{ end }}
</nav>
{{ if .GroupsHTML }}{{ .GroupsHTML }}{{ else }}{{ template "glossary-groups" . }}{{ end }}{{ else }}
<p class="muted">No glossary entries.</p>
{{ end }}
{{ end }}
{{ template "base.html" . }}
{{ define "glossary-groups" }}{{ range .Groups }}
<section class="glossary-group" id="g{{ .Letter }}">
<h2>{{ .Letter }}</h2>
<dl class="glossary-list">
{{ range .Entries }}
<dt id="term-{{ pathEscape .Term }}">
{{ .Term }}
{{ if .Aka }}<span class="muted glossary-aka">(a.k.a. {{ range $i, $a := .Aka }}{{ if $i }}, {{ end }}{{ $a }}{{ end }})</span>{{ end }}
<a class="glossary-search" href="{{ .SearchURL }}"{{ if .Aka }} title="Searches the term and its aliases"{{ end }}>search &rarr;</a>
</dt>
<dd>
{{ .DefinitionHTML }}
{{ if .SeeLinks }}<div class="glossary-see muted">See also:
{{ range $i, $s := .SeeLinks }}{{ if $i }}, {{ end }}{{ if $s.Term }}<a href="#term-{{ pathEscape $s.Term }}">{{ $s.Name }}</a>{{ else }}{{ $s.Name }}{{ end }}{{ end }}
</div>{{ end }}
</dd>
{{ end }}
</dl>
</section>
{{ end }}
{{ end }}

63
templates/ids.html Normal file
View file

@ -0,0 +1,63 @@
{{ define "title" }}Technical IDs - Snowden Archive{{ end }}
{{ define "content" }}
<header class="list-header">
<p class="crumb"><a href="/explore">&larr; Explore</a></p>
<h1>Technical IDs <small class="count-note">{{ .Shown }} document{{ if ne .Shown 1 }}s{{ end }}</small></h1>
</header>
<p class="muted">
<a href="/explore/ids.txt">Checksums as plain text</a>
<span class="faint">(<code>sha256sum -c</code> in the corpus root)</span>
</p>
<div class="table-scroll">
<table class="data-table id-table">
<colgroup>
<col class="c-id"><col class="c-coll"><col class="c-pages">
<col class="c-pdf"><col class="c-size"><col class="c-sha">
</colgroup>
<thead>
<tr>
<th>Document</th>
<th>Collection</th>
<th class="num">Pages</th>
<th>PDF</th>
<th class="num">Size</th>
<th>SHA-256</th>
</tr>
</thead>
<tbody>
{{ if .RowsHTML }}{{ .RowsHTML }}{{ else }}{{ template "id-rows" . }}{{ end }}
</tbody>
</table>
</div>
{{ end }}
{{ template "base.html" . }}
{{ define "id-rows" }}{{ range .Rows }}
<tr>
<td><a href="/doc/{{ .ID }}">{{ .ID }}</a>
{{- range .Aka }}<br><span class="faint">also published as</span> <code>{{ .ID }}</code>{{ end -}}
</td>
<td>{{ .Collection }}</td>
<td class="num">{{ if .Pages }}{{ .Pages }}{{ end }}</td>
<td><a href="{{ .PDFURL }}" target="_blank" rel="noopener"><code>{{ .PDFFile }}</code></a></td>
<td class="num">{{ .Size }}</td>
<td>{{ with .SHA }}<code class="checksum">{{ . }}</code>{{ else }}<span class="faint">-</span>{{ end }}</td>
</tr>
{{ $id := .ID }}
{{ range .Versions }}
<tr class="id-version">
<td><span class="faint">&#8627;</span> <a href="/doc/{{ $id }}/versions/{{ .Slug }}">versions/{{ .Slug }}</a></td>
<td class="faint">copy</td>
<td class="num">{{ if .Pages }}{{ .Pages }}{{ end }}</td>
<td><a href="{{ .PDFURL }}" target="_blank" rel="noopener"><code>{{ .PDFFile }}</code></a></td>
<td class="num">{{ .Size }}</td>
<td>{{ if .SHA }}<code class="checksum">{{ .SHA }}</code>{{ else }}<span class="faint">-</span>{{ end }}</td>
</tr>
{{ end }}
{{ else }}
<tr><td colspan="6" class="muted">No documents.</td></tr>
{{ end }}{{ end }}

150
templates/index.html Normal file
View file

@ -0,0 +1,150 @@
{{ define "content" }}
<h1>Snowden Archive</h1>
<p class="muted home-intro">
This archive holds {{ .SnowdenTotal }} published Snowden documents. {{ .ReportedTotal }} documents are
published with news reporting, and {{ .ArchiveTotal }} documents are the NSA newsletter SIDtoday
issues, from The Intercept's bulk release.{{ if .FOIATotal }} This archive also holds {{ .FOIATotal }}
FOIA documents related to the Snowden disclosures (FOIA litigation, FISC and ODNI declassifications),
which {{ if .IncludeFOIA }}you have included via <a href="/preferences">Preferences</a>{{ else }}are left out by default but can be included via <a href="/preferences">Preferences</a>{{ end }}.{{ end }}
</p>
<form action="/search" method="GET" class="home-search-form" role="search">
<input type="text" name="q" class="home-search" placeholder="Search the archive... (Enter to search)" aria-label="Search the archive">
</form>
<div class="home-actions">
<a class="home-action" href="/browse">Browse the archive &rarr;</a>
</div>
{{ if and .TrackRecent .RecentDocs }}
<section class="home-section">
<h2>Recently viewed
<button type="submit" form="clear-recent-docs" class="link-btn faint">clear</button>
</h2>
<form method="POST" action="/history/clear" id="clear-recent-docs" class="inline-form">
<input type="hidden" name="which" value="docs">
<input type="hidden" name="back" value="/">
</form>
<ul class="stat-list">
{{ range .RecentDocs }}
<li><a href="/doc/{{ .ID }}">{{ .Title }}</a> <span class="muted">{{ .DocMeta.Creator }}</span></li>
{{ end }}
</ul>
</section>
{{ end }}
<p class="home-stats-note muted">{{ if .WithSIDtoday }}The statistics below include SIDtoday archive documents (<a href="/">exclude them</a>).{{ else }}The statistics below exclude SIDtoday archive documents (<a href="/?sidtoday=1">include them</a>).{{ end }}</p>
{{ if .DocHist }}
<section class="home-section">
<h2>When the documents were written</h2>
<div class="date-hist">
{{ range .DocHist }}
<a class="hist-col" href="{{ .URL }}" title="{{ .Count }} documents dated {{ .Year }}">
<span class="hist-count">{{ .Count }}</span>
<span class="hist-track"><span class="hist-bar" style="height: {{ .Pct }}%"></span></span>
<span class="hist-year">{{ .Year }}</span>
</a>
{{ end }}
</div>
</section>
{{ end }}
{{ if .PubHist }}
<section class="home-section">
<h2>When they were published</h2>
<div class="date-hist">
{{ range .PubHist }}
<a class="hist-col" href="{{ .URL }}" title="{{ .Count }} documents first published in {{ .Year }}">
<span class="hist-count">{{ .Count }}</span>
<span class="hist-track"><span class="hist-bar" style="height: {{ .Pct }}%"></span></span>
<span class="hist-year">{{ .Year }}</span>
</a>
{{ end }}
</div>
</section>
{{ end }}
{{ if .TopPrograms }}
<section class="home-section">
<h2>Top programs <a class="see-all" href="/browse/program">all &rarr;</a></h2>
<div class="pill-list">
{{ range .TopPrograms }}
<a class="pill" href="{{ .URL }}">{{ .Name }} <span class="pill-count">{{ .Count }}</span></a>
{{ end }}
</div>
</section>
{{ end }}
<div class="home-grid">
{{ if .TopAgencies }}
<section class="home-section">
<h2>Agencies <a class="see-all" href="/browse/agency">all &rarr;</a></h2>
<ul class="stat-list">
{{ range .TopAgencies }}
<li><a href="{{ .URL }}">{{ .Name }}</a> <span class="muted">{{ .Count }}</span></li>
{{ end }}
</ul>
</section>
{{ end }}
{{ if .TopCountries }}
<section class="home-section">
<h2>Countries <a class="see-all" href="/browse/country">all &rarr;</a></h2>
<ul class="stat-list">
{{ range .TopCountries }}
<li><a href="{{ .URL }}">{{ .Name }}</a> <span class="muted">{{ .Count }}</span></li>
{{ end }}
</ul>
</section>
{{ end }}
{{ if .TopPeople }}
<section class="home-section">
<h2>People <a class="see-all" href="/browse/people">all &rarr;</a></h2>
<ul class="stat-list">
{{ range .TopPeople }}
<li><a href="{{ .URL }}">{{ .Name }}</a> <span class="muted">{{ .Count }}</span></li>
{{ end }}
</ul>
</section>
{{ end }}
{{ if .TopJournalists }}
<section class="home-section">
<h2>Journalists <a class="see-all" href="/browse/journalist">all &rarr;</a></h2>
<ul class="stat-list">
{{ range .TopJournalists }}
<li><a href="{{ .URL }}">{{ .Name }}</a> <span class="muted">{{ .Count }}</span></li>
{{ end }}
</ul>
</section>
{{ end }}
{{ if .TopPublishers }}
<section class="home-section">
<h2>Publishers <a class="see-all" href="/browse/publisher">all &rarr;</a></h2>
<ul class="stat-list">
{{ range .TopPublishers }}
<li><a href="{{ .URL }}">{{ .Name }}</a> <span class="muted">{{ .Count }}</span></li>
{{ end }}
</ul>
</section>
{{ end }}
</div>
{{ if .TopTags }}
<section class="home-section">
<h2>Top tags <a class="see-all" href="/browse/tag">all &rarr;</a></h2>
<div class="pill-list">
{{ range .TopTags }}
<a class="pill" href="{{ .URL }}">{{ .Name }} <span class="pill-count">{{ .Count }}</span></a>
{{ end }}
</div>
</section>
{{ end }}
<p class="explore-link">Digging deeper? <a href="/explore">Explore the archive</a> for ranked views and lists.
{{ end }}
{{ template "base.html" . }}

95
templates/list.html Normal file
View file

@ -0,0 +1,95 @@
{{ define "title" }}{{ with .CatLabel }}{{ . }}{{ else }}Category{{ end }}: {{ .Title }} - Snowden Archive{{ end }}
{{ define "content" }}
<header class="list-header">
<h1>{{ with .CatLabel }}{{ . }}{{ else }}Category{{ end }}: {{ .Title }}
{{- if eq .ActiveCategory "agency" }}{{ with agencyName .Title }} <small class="muted cat-note">{{ . }}</small>{{ end }}{{ end }}
<small class="count-note">{{ .Total }} document{{ if ne .Total 1 }}s{{ end }}</small>
</h1>
{{ if .SearchLink }}
<a href="{{ .SearchLink }}" class="faint search-link">Search documents matching &ldquo;{{ .Title }}&rdquo; &rarr;</a>
{{ end }}
{{ with .Blurb }}<p class="muted cat-blurb">{{ . }}</p>{{ end }}
{{ with .YearScope }}
<p class="muted coll-scope">Showing the documents {{ $.Title }} itself published in <strong>{{ . }}</strong> -
<a href="{{ $.YearScopeAllURL }}">all years</a>.</p>
{{ end }}
</header>
{{ if .MonthNav }}
<nav class="month-nav" aria-label="Months in {{ .MonthNavYear }}">
<a href="{{ .MonthNavYearURL }}"{{ if .MonthNavYearActive }} class="active"{{ end }}>All {{ .MonthNavYear }}</a>
{{ range .MonthNav }}<a href="{{ .URL }}"{{ if .Active }} class="active"{{ end }}>{{ .Label }} <span class="facet-count">{{ .Count }}</span></a>
{{ end }}
{{ if .MonthlessCount }}<span class="faint month-nav-note">+{{ .MonthlessCount }} in {{ .MonthNavYear }} with year-only dates</span>{{ end }}
</nav>
{{ end }}
{{ with .GlossaryEntry }}
<div class="glossary-box">
<strong>{{ .Term }}</strong>
{{ if .Aka }}<span class="muted glossary-aka">(a.k.a. {{ range $i, $a := .Aka }}{{ if $i }}, {{ end }}{{ $a }}{{ end }})</span>{{ end }}
- {{ .DefinitionHTML }}
<a href="/glossary#term-{{ pathEscape .Term }}" class="glossary-search">glossary &rarr;</a>
</div>
{{ end }}
{{ if or .Timeline .PubTimeline .Breakdowns }}
<details class="entity-timeline">
<summary>Graphs</summary>
{{ if .Timeline }}{{ template "timeline-hist" (dict "Bars" .Timeline "Label" "By document date" "Verb" "dated") }}{{ end }}
{{ if .PubTimeline }}{{ template "timeline-hist" (dict "Bars" .PubTimeline "Label" "By published date" "Verb" "published") }}{{ end }}
{{ range .Breakdowns }}
<div class="timeline-block">
<h2 class="timeline-label">By {{ .Label }}</h2>
<div class="breakdown-bars">
{{ range .Values }}
<a class="bd-row" href="{{ .URL }}" title="{{ .Count }} documents - open as a search">
<span class="bd-name">{{ .Value }}</span>
<span class="bd-track"><span class="bd-bar" style="width: {{ .Pct }}%"></span></span>
<span class="bd-count">{{ .Count }}</span>
</a>
{{ end }}
</div>
</div>
{{ end }}
</details>
{{ end }}
{{ if .Refinements }}
<details class="search-facets">
<summary>Refine <span class="faint">(combine with another filter, via search)</span></summary>
<div class="facets-body">
{{ range .Refinements }}
<div class="facet-group">
<span class="facet-label">{{ .Label }}</span>
<span class="facet-values">{{ range .Values }}<a class="facet-link" href="{{ .URL }}">{{ .Name }}<span class="facet-count">{{ .Count }}</span></a>{{ end }}</span>
</div>
{{ end }}
</div>
</details>
{{ end }}
<div class="list-controls">
{{ if ne .View "table" }}
{{ template "sort-bar" (dict "SortURL" .SortURL "SortBy" .Sort "Order" .Order "ShowRelevance" false "DefaultSort" .DefaultSort) }}
{{ end }}
{{ template "type-filter" (dict
"Action" (print "/browse/" .ActiveCategory "/" (pathEscape .Title))
"Params" .Params
"AllTypes" .AllTypes
"SelectedTypes" .SelectedTypes
) }}
</div>
{{ if eq .View "table" }}
{{ template "doc-table" . }}
{{ else }}
<h2 class="sr-only">Documents</h2>
{{ template "doc-cards" .Cards }}
{{ template "pagination" . }}
{{ end }}
{{ end }}
{{ template "base.html" . }}

54
templates/outlets.html Normal file
View file

@ -0,0 +1,54 @@
{{ define "title" }}Publishers by year - Snowden Archive{{ end }}
{{ define "content" }}
<header class="list-header">
<p class="crumb"><a href="/explore">&larr; Explore</a></p>
<h1>Publishers by year</h1>
</header>
<p class="muted">When each publisher released documents, and how much. The number of
distinct stories are after the slash.</p>
<p class="hist-toggles faint">Showing:
{{ if .NewsOnly }}<a href="/explore/outlets">all collections</a> &middot; <strong>reported only</strong>
{{ else }}<strong>all collections</strong> &middot; <a href="/explore/outlets?coll=news">reported only</a>{{ end }}
</p>
{{ with .Table }}
{{ if .Rows }}
<div class="table-scroll">
<table class="data-table heat-table outlet-table">
<thead>
<tr>
<th>Outlet</th>
{{ range .YearLinks }}<th class="num"><a href="{{ .URL }}">{{ .Name }}</a></th>{{ end }}
<th class="num-col">Total</th>
</tr>
</thead>
<tbody>
{{ range .Rows }}
<tr>
<td><a href="{{ .URL }}">{{ .Name }}</a></td>
{{ range .Cells }}
<td class="heat-cell">{{ if .Docs }}<a href="{{ .URL }}" title="{{ .Docs }} documents, {{ .Articles }} stories">{{ .Docs }}{{ if .Articles }}<small class="faint">/{{ .Articles }}</small>{{ end }}</a>{{ else }}<span class="faint">-</span>{{ end }}</td>
{{ end }}
<td class="num-col">{{ .Total }}</td>
</tr>
{{ end }}
</tbody>
<tfoot>
<tr>
<th>All outlets</th>
{{ range .Totals }}<th class="num-col">{{ if . }}{{ . }}{{ else }}<span class="faint">-</span>{{ end }}</th>{{ end }}
<th></th>
</tr>
</tfoot>
</table>
</div>
{{ else }}
<p class="muted">No dated publications in the corpus yet.</p>
{{ end }}
{{ end }}
{{ end }}
{{ template "base.html" . }}

40
templates/pages.html Normal file
View file

@ -0,0 +1,40 @@
{{ define "title" }}Pages - {{ .Doc.Title }}{{ with .Version }} ({{ .Slug }}){{ end }} - Snowden Archive{{ end }}
{{ define "content" }}
<header class="list-header">
<p class="crumb">{{ with .Version }}<a href="/doc/{{ $.Doc.ID }}/versions/{{ .Slug }}">&larr; {{ $.Doc.Title }} (copy: {{ .Slug }})</a>{{ else }}<a href="/doc/{{ .Doc.ID }}">&larr; {{ .Doc.Title }}</a>{{ end }}</p>
<h1>All pages <small class="count-note">{{ len .Images }} page{{ if ne (len .Images) 1 }}s{{ end }}</small></h1>
{{ with .Version }}{{ if .ChangedPages }}<p class="changed-pages-note">Pages that differ from the main document: {{ template "changed-page-links" . }}</p>{{ end }}{{ end }}
</header>
{{ if and .Images .TextOnly }}
<p class="muted textonly-note">Text-only mode is on - page images are not loaded. <a href="/preferences">Preferences</a></p>
<ul class="page-link-list">
{{ $id := .Doc.ID }}
{{ range $i, $img := .Images }}
<li><a href="{{ $.ImageBase }}{{ pathEscape $img }}" target="_blank" rel="noopener">Page {{ add $i 1 }}</a>{{ if and $.Version ($.Version.PageChanged (add $i 1)) }} <span class="muted">(differs from the main document)</span>{{ end }}</li>
{{ end }}
</ul>
{{ else if .Images }}
<ul class="page-sheet">
{{ $id := .Doc.ID }}
{{ range $i, $img := .Images }}
<li{{ if and $.Version ($.Version.PageChanged (add $i 1)) }} class="page-changed"{{ end }}>
<a href="#lb-{{ add $i 1 }}" title="Open page {{ add $i 1 }} full screen">
<img src="{{ $.ImageBase }}{{ pathEscape $img }}"{{ $.ImageSize $i }}
alt="Page {{ add $i 1 }}" loading="lazy">
<span class="page-num faint">{{ add $i 1 }}</span>{{ if and $.Version ($.Version.PageChanged (add $i 1)) }}<span class="page-changed-badge">differs</span>{{ end }}
</a>
</li>
{{ end }}
</ul>
{{ template "lightbox-overlays" . }}
{{ else }}
<p class="muted">This document has no page images.</p>
{{ end }}
{{ end }}
{{ template "base.html" . }}

412
templates/partials.html Normal file
View file

@ -0,0 +1,412 @@
{{ define "cat-boxes" }}
{{ $a := .ActiveCategory }}{{ if not $a }}{{ $a = "" }}{{ end }}
<nav id="cat-boxes" aria-label="Browse categories">
<a href="/browse/program" class="cat-box{{ if eq $a "program" }} cat-box-active{{ end }}"><strong>Programs</strong><span class="count">{{ index .CatCounts "program" }}</span></a>
<a href="/browse/journalist" class="cat-box{{ if eq $a "journalist" }} cat-box-active{{ end }}"><strong>Journalists</strong><span class="count">{{ index .CatCounts "journalist" }}</span></a>
<a href="/browse/agency" class="cat-box{{ if eq $a "agency" }} cat-box-active{{ end }}"><strong>Agencies</strong><span class="count">{{ index .CatCounts "agency" }}</span></a>
<a href="/browse/publisher" class="cat-box{{ if eq $a "publisher" }} cat-box-active{{ end }}"><strong>Publishers</strong><span class="count">{{ index .CatCounts "publisher" }}</span></a>
<a href="/browse/country" class="cat-box{{ if eq $a "country" }} cat-box-active{{ end }}"><strong>Countries</strong><span class="count">{{ index .CatCounts "country" }}</span></a>
<a href="/browse/people" class="cat-box{{ if eq $a "people" }} cat-box-active{{ end }}"><strong>People</strong><span class="count">{{ index .CatCounts "people" }}</span></a>
<a href="/browse/classification" class="cat-box{{ if eq $a "classification" }} cat-box-active{{ end }}"><strong>Classification</strong><span class="count">{{ index .CatCounts "classification" }}</span></a>
<a href="/browse/dissemination" class="cat-box{{ if eq $a "dissemination" }} cat-box-active{{ end }}"><strong>Dissemination</strong><span class="count">{{ index .CatCounts "dissemination" }}</span></a>
<a href="/browse/published" class="cat-box{{ if eq $a "published" }} cat-box-active{{ end }}"><strong>Pub Year</strong><span class="count">{{ index .CatCounts "published" }}</span></a>
<a href="/browse/docdate" class="cat-box{{ if eq $a "docdate" }} cat-box-active{{ end }}"><strong>Doc Date</strong><span class="count">{{ index .CatCounts "docdate" }}</span></a>
<a href="/browse/tag" class="cat-box{{ if eq $a "tag" }} cat-box-active{{ end }}"><strong>Tags</strong><span class="count">{{ index .CatCounts "tag" }}</span></a>
</nav>
{{ end }}
{{/* pub-publisher is a publication entry's outlet: a link to the story where
the entry has one, else to the outlet's page - which a partner that only
reported has none of. */}}
{{ define "pub-publisher" -}}
{{ if .Pub.Link }}<a href="{{ .Pub.Link }}" target="_blank" rel="noopener"{{ if or .LinkClass .On }} class="{{ .LinkClass }}{{ if and .LinkClass .On }} {{ end }}{{ if .On }}active{{ end }}"{{ end }}>{{ .Pub.Publisher }} &nearr;</a>
{{- else if .Pub.ReportingOnly }}{{ .Pub.Publisher }}
{{- else }}<a href="/browse/publisher/{{ pathEscape .Pub.Publisher }}{{ with .QS }}{{ . }}{{ end }}"{{ if or .PageClass .On }} class="{{ .PageClass }}{{ if and .PageClass .On }} {{ end }}{{ if .On }}active{{ end }}"{{ end }}>{{ .Pub.Publisher }}</a>{{ end }}
{{- end }}
{{ define "sort-th" }}{{ if .URL }}<th{{ with .Class }} class="{{ . }}"{{ end }}><a href="{{ .URL }}">{{ .Label }} {{ if .Active }}{{ if .Desc }}&#9660;{{ else }}&#9650;{{ end }}{{ else }}<span class="sort-hint">&#8597;</span>{{ end }}</a></th>{{ else }}<th{{ with .Class }} class="{{ . }}"{{ end }}>{{ .Label }}</th>{{ end }}{{ end }}
{{ define "doc-table" }}
<div class="table-scroll">
<table class="data-table">
<thead>
<tr>{{ range .TableHeads }}{{ template "sort-th" . }}{{ end }}</tr>
</thead>
<tbody>
{{ if .RowsHTML }}{{ .RowsHTML }}{{ else }}{{ range .TableRows }}{{ template "doc-table-row" . }}{{ end }}{{ end }} </tbody>
</table>
</div>
{{ end }}
{{/* The rows are rendered once for every reader, so a FOIA release's links
always name every collection: they must list the release itself. */}}
{{ define "doc-table-row" }}{{ $qs := "" }}{{ if eq .Collection "foia" }}{{ $qs = wholeArchiveQS }}{{ end }}
<tr>
<td>
<a href="/doc/{{ .ID }}"><strong>{{ .Title }}</strong></a>
[<a href="/files/{{ .ID }}/{{ pathEscape .PDFFile }}" target="_blank" rel="noopener">pdf</a>]
</td>
<td>
{{ if .DocMeta.Creator }}<a href="/browse/agency/{{ pathEscape .DocMeta.Creator }}{{ $qs }}">{{ .DocMeta.Creator }}</a>{{ else }}-{{ end }}
</td>
<td>
{{ with .DocMeta.DateCreated }}{{ . }}{{ else }}-{{ end }}
</td>
<td>
{{- $first := true -}}
{{- range .Publication -}}
{{- if .DatePublished -}}
{{- if not $first -}}, {{ end -}}
{{ .DatePublished }}
{{- $first = false -}}
{{- end -}}
{{- end -}}
{{- if $first -}}-{{- end -}}
</td>
<td>{{ facetLinks (dict "Vals" .DocMeta.Classification "Cat" "classification" "QS" $qs) }}</td>
<td>{{ facetLinks (dict "Vals" .DocMeta.Dissemination "Cat" "dissemination" "QS" $qs) }}</td>
<td>{{ facetLinks (dict "Vals" .DocMeta.Programs "Cat" "program" "QS" $qs) }}</td>
<td>{{ facetLinks (dict "Vals" .DocMeta.Countries "Cat" "country" "QS" $qs) }}</td>
<td>{{ facetLinks (dict "Vals" .DocMeta.People "Cat" "people" "QS" $qs) }}</td>
<td>{{ facetLinks (dict "Vals" .JournalistsDeduped "Cat" "journalist" "QS" $qs) }}</td>
<td>
{{- $first = true -}}
{{- range .Publication -}}
{{- if .Publisher -}}
{{- if not $first -}}, {{ end -}}
{{ template "pub-publisher" (dict "Pub" . "QS" $qs) }}
{{- $first = false -}}
{{- end -}}
{{- end -}}
{{- if $first -}}-{{- end -}}
</td>
<td>
{{ template "type-badge" .Type }}
{{ if and .MarkReported .ReportedIssue }}<span class="issue-reported" title="Individually reported - also in the Reported collection">reported</span>{{ end }}
</td>
</tr>
{{ end }}
{{ define "doc-cards" }}
<ul class="doc-grid">
{{ range .Docs }}
<li class="doc-card">
{{ if $.ShowThumbs }}
<div class="thumb-container"{{ if and $.CardPagePreview .Images }} data-pp-id="{{ .ID }}" data-pp-imgs="{{ range $i, $im := .Images }}{{ if $i }} {{ end }}{{ pathEscape $im }}{{ end }}"{{ end }}>
<a href="/doc/{{ .ID }}" class="thumb" tabindex="-1" aria-hidden="true">
{{ if .HasThumb }}
<img src="/files/{{ .ID }}/thumb.jpg"
{{ if .HasGifThumb }}data-gif="/files/{{ .ID }}/thumb.gif"{{ end }}
class="doc-img"
alt="" loading="lazy">
{{ end }}
</a>
</div>
{{ end }}
<div class="doc-body">
{{ $cat := $.ActiveCategory }}{{ $val := $.ActiveValue }}
{{/* A FOIA release's links name every collection for a reader who
leaves FOIA out: the pages they open must list the release. */}}
{{ $qs := "" }}{{ if eq .Collection "foia" }}{{ $qs = $.FOIAQS }}{{ end }}
<h3>
<a href="/doc/{{ .ID }}" class="doc-title">{{ if .Title }}{{ .Title }}{{ else }}<span class="doc-untitled">(untitled)</span>{{ end }}</a>
{{ template "type-badge" .Type }}
{{ if and $.MarkReported .ReportedIssue }}<span class="issue-reported" title="Individually reported - also in the Reported collection">reported</span>{{ end }}
<a href="/files/{{ .ID }}/{{ pathEscape .PDFFile }}" target="_blank" rel="noopener" class="doc-pdf" title="Open PDF">[pdf]</a>
{{ if $.ShowUnpin }}
<button type="submit" form="unpin-{{ .ID }}" class="link-btn unpin-btn" title="Remove from pinned">&#9733; unpin</button>
{{ end }}
</h3>
{{ if $.ShowUnpin }}
<form method="POST" action="/pin/{{ .ID }}" id="unpin-{{ .ID }}" class="inline-form">
<input type="hidden" name="back" value="/pinned">
</form>
{{ end }}
<div class="doc-content">
<div>
{{ if .DocMeta.Creator }}
<a href="/browse/agency/{{ pathEscape .DocMeta.Creator }}{{ $qs }}"
class="doc-creator{{ if and (eq $cat "agency") (eqFold .DocMeta.Creator $val) }} active{{ end }}">{{ .DocMeta.Creator }}</a>
{{ end }}
{{ if and .DocMeta.Creator .DocMeta.DateCreated }}
<span class="doc-dot">&middot;</span>
{{ end }}
{{ if .DocMeta.DateCreated }}
{{ if ge (len .DocMeta.DateCreated) 4 }}
{{ $year := slice .DocMeta.DateCreated 0 4 }}
<a href="/browse/docdate/{{ $year }}{{ $qs }}"
class="doc-created{{ if and (eq $cat "docdate") (eq $year $val) }} active{{ end }}">{{ .DocMeta.DateCreated }}</a>
{{ else }}
<span class="doc-created">{{ .DocMeta.DateCreated }}</span>
{{ end }}
{{ end }}
{{ if .Pages }}
{{ if or .DocMeta.Creator .DocMeta.DateCreated }}
<span class="doc-dot">&middot;</span>
{{ end }}
<span class="doc-pages">{{ .Pages }} p.</span>
{{ end }}
</div>
{{ if and .Description (not $.HideSummary) }}
<p class="card-summary">{{ .Description }}</p>
{{ end }}
{{ if and (or .DocMeta.Classification .DocMeta.Dissemination) (not $.HideClassification) }}
<div>
<span class="doc-classification">Marking:</span>
{{ with .DocMeta.Classification }}{{ facetLinks (dict "Vals" . "Cat" "classification" "Sep" "dot" "Active" (and (eq $cat "classification") $val) "QS" $qs) }}{{ end }}
{{ if and .DocMeta.Classification .DocMeta.Dissemination }}<span class="doc-dot">&middot;</span>{{ end }}
{{ with .DocMeta.Dissemination }}{{ facetLinks (dict "Vals" . "Cat" "dissemination" "Sep" "dot" "Active" (and (eq $cat "dissemination") $val) "QS" $qs) }}{{ end }}
</div>
{{ end }}
{{ if and .DocMeta.Programs (not $.HidePrograms) }}
<div>
<span class="doc-programs">Programs:</span>
{{ facetLinks (dict "Vals" .DocMeta.Programs "Cat" "program" "Sep" "dot" "Active" (and (eq $cat "program") $val) "QS" $qs) }}
</div>
{{ end }}
{{ if and .DocMeta.People (not $.HidePeople) }}
<div>
<span class="doc-people">People:</span>
{{ facetLinks (dict "Vals" .DocMeta.People "Cat" "people" "Sep" "dot" "Active" (and (eq $cat "people") $val) "QS" $qs) }}
</div>
{{ end }}
<div class="doc-publication">
{{ $shown := false }}
{{ range $pub := .Publication }}
{{ if or $pub.Title $pub.Link $pub.Publisher $pub.DatePublished }}
{{ if $shown }}<span class="doc-dot">&middot;</span>{{ end }}{{ $shown = true }}
{{ if $pub.Title }}
{{ $ak := articleKey $pub.Link }}
{{ if $ak }}<a href="/article/{{ $ak }}" class="pub-article-title">&ldquo;{{ $pub.Title }}&rdquo;</a>{{ else }}<span class="pub-article-title">&ldquo;{{ $pub.Title }}&rdquo;</span>{{ end }}
-
{{ end }}
{{ if or $pub.Link $pub.Publisher }}
{{ template "pub-publisher" (dict "Pub" $pub "QS" $qs "LinkClass" "pub-link" "PageClass" "pub-source" "On" (and (eq $cat "publisher") (eqFold $pub.Publisher $val))) }}
{{ end }}
{{ if $pub.DatePublished }}
<span class="doc-published">({{ $pub.DatePublished }})</span>
{{ end }}
{{ end }}
{{ end }}
{{ $journos := .JournalistsDeduped }}
{{ if and $journos (not $.HideJournalists) }}
<span class="doc-dot">&middot;</span>
{{ range $i, $j := $journos }}
{{ if $i }}<span class="doc-dot">&middot;</span>{{ end }}
<a href="/browse/journalist/{{ pathEscape $j }}{{ $qs }}"
class="doc-journalist{{ if and (eq $cat "journalist") (eqFold $j $val) }} active{{ end }}">{{ $j }}</a>
{{ end }}
{{ end }}
</div>
</div>
</div>
</li>
{{ end }}
</ul>
{{ end }}
{{ define "type-filter" }}
<form method="GET" action="{{ .Action }}" class="type-filter">
{{ range $k, $vals := .Params }}
{{ if and (ne $k "type") (ne $k "p") (ne $k "action") }}
{{ range $vals }}
<input type="hidden" name="{{ $k }}" value="{{ . }}">
{{ end }}
{{ end }}
{{ end }}
<span class="muted">Show:</span>
{{ range $type, $count := .AllTypes }}
<label class="type-filter-option">
<input type="checkbox" name="type" value="{{ $type }}"
{{ if index $.SelectedTypes $type }}checked{{ end }}>
{{ template "type-badge" $type }}
{{ if eq $type "REPORTED" }}<span class="type-filter-leak">Reported</span>{{ end }}
<span class="faint type-filter-count">({{ $count }})</span>
</label>
{{ end }}
<button type="submit" class="ctl">Apply</button>
</form>
{{ end }}
{{ define "pagination" }}
{{ if gt .TotalPages 1 }}
{{ $sortParam := "" }}{{ if .Sort }}{{ $sortParam = print "&sort=" .Sort }}{{ end }}
{{ $orderParam := "" }}{{ if .Order }}{{ $orderParam = print "&order=" .Order }}{{ end }}
<nav class="pagination" aria-label="Pagination">
{{ if gt .Page 1 }}
<a href="{{ .BaseURL }}p={{ add .Page -1 }}{{ $sortParam }}{{ $orderParam }}" class="pag-btn" rel="prev">&laquo; Prev</a>
{{ end }}
{{ range pages .Page .TotalPages }}
{{ if eq . 0 }}
<span class="pag-ellipsis" aria-hidden="true">...</span>
{{ else if eq . $.Page }}
<span class="pag-btn pag-current" aria-current="page">{{ . }}</span>
{{ else }}
<a href="{{ $.BaseURL }}p={{ . }}{{ $sortParam }}{{ $orderParam }}" class="pag-btn">{{ . }}</a>
{{ end }}
{{ end }}
{{ if lt .Page .TotalPages }}
<a href="{{ .BaseURL }}p={{ add .Page 1 }}{{ $sortParam }}{{ $orderParam }}" class="pag-btn" rel="next">Next &raquo;</a>
{{ end }}
</nav>
{{ end }}
{{ end }}
{{ define "timeline-hist" }}
<div class="timeline-block">
<h2 class="timeline-label">{{ .Label }} <span class="faint">({{ len .Bars }} years)</span></h2>
<div class="date-hist entity-timeline-hist">
{{ range .Bars }}
{{ if .URL }}
<a class="hist-col" href="{{ .URL }}" title="{{ .Count }} documents {{ $.Verb }} {{ .Year }}">
<span class="hist-count">{{ .Count }}</span>
<span class="hist-track"><span class="hist-bar" style="height: {{ .Pct }}%"></span></span>
<span class="hist-year">{{ .Year }}</span>
</a>
{{ else }}
<span class="hist-col" title="{{ .Count }} documents {{ $.Verb }} {{ .Year }}">
<span class="hist-count">{{ .Count }}</span>
<span class="hist-track"><span class="hist-bar" style="height: {{ .Pct }}%"></span></span>
<span class="hist-year">{{ .Year }}</span>
</span>
{{ end }}
{{ end }}
</div>
</div>
{{ end }}
{{ define "sort-bar" }}
{{ $base := .SortURL }}
{{ $order := .Order }}
{{ $showRelevance := .ShowRelevance }}
{{ $cur := .SortBy }}
{{ if eq $cur "" }}{{ $cur = .DefaultSort }}{{ end }}
{{ $activeLabel := sortLabel $cur }}
{{ $directed := ne $cur "relevance" }}
{{ $orderLabel := "↑" }}{{ if eq $order "desc" }}{{ $orderLabel = "↓" }}{{ end }}
<details class="sort-dropdown">
<summary
aria-label="Sort documents, currently sorted by {{ $activeLabel }}{{ if $directed }} {{ if eq $order "desc" }}descending{{ else }}ascending{{ end }}{{ end }}">
Sort: <strong>{{ $activeLabel }}</strong>
{{ if $directed }}<span aria-hidden="true">{{ $orderLabel }}</span>{{ end }}
</summary>
<div role="navigation" aria-label="Sort options" class="sort-options">
{{ range sortOptions }}
{{ if or (ne .Key "relevance") $showRelevance }}
{{ $active := eq .Key $cur }}
{{ $flips := and $active (ne .Key "relevance") }}
{{ $href := print $base "sort=" .Key }}
{{ if and $flips (ne $order "desc") }}{{ $href = print $href "&order=desc" }}{{ end }}
<a href="{{ $href }}" class="{{ if $active }}active{{ end }}" {{ if $active }}aria-current="true"{{ end }}>
{{ .Label }}
{{ if $flips }}
<span aria-hidden="true">{{ $orderLabel }}</span>
<span class="sr-only">{{ if eq $order "desc" }}descending, click for ascending{{ else }}ascending, click for descending{{ end }}</span>
{{ end }}
</a>
{{ end }}
{{ end }}
</div>
</details>
{{ end }}
{{ define "type-badge" }}
{{ if eq . "FOIA" }}
<span class="type-badge type-badge--foia">FOIA</span>
{{ else if eq . "SIDTODAY" }}
<span class="type-badge type-badge--sidtoday" title="This document is part of SIDtoday">SIDtoday</span>
{{ end }}
{{ end }}
{{ define "lightbox-overlays" }}
{{ $lbBase := .ImageBase }}{{ $lbN := len .Images }}
{{ range $i, $img := .Images }}
<div class="lightbox" id="lb-{{ add $i 1 }}" aria-label="Page {{ add $i 1 }} of {{ $lbN }}">
<a class="lb-backdrop" href="#lb-closed" tabindex="-1" aria-hidden="true"></a>
<a class="lb-close" href="#lb-closed" title="Close">&times;</a>
{{ if $i }}<a class="lb-prev" href="#lb-{{ $i }}" title="Previous page">&lsaquo;</a>{{ end }}
{{ if lt (add $i 1) $lbN }}<a class="lb-next" href="#lb-{{ add $i 2 }}" title="Next page">&rsaquo;</a>{{ end }}
<img src="{{ $lbBase }}{{ pathEscape $img }}"{{ $.ImageSize $i }} alt="Page {{ add $i 1 }} of {{ $lbN }}" loading="lazy" decoding="async">
<span class="lb-counter">{{ add $i 1 }} / {{ $lbN }}{{ if and $.Version ($.Version.PageChanged (add $i 1)) }} &middot; differs from the main document{{ end }}</span>
</div>
{{ end }}
{{ end }}
{{ define "changed-page-links" }}
{{- $v := . -}}
{{- range $k, $p := .ChangedPages -}}
{{- if $k }}, {{ end -}}
{{- if $p.Pos }}<a href="/doc/{{ $v.DocID }}/versions/{{ $v.Slug }}/pages#lb-{{ $p.Pos }}" title="Open page {{ $p.Page }} of this copy">{{ $p.Page }}</a>{{ else }}{{ $p.Page }}{{ end -}}
{{- end -}}
{{ end }}
{{ define "doc-viewport" }}
<input type="radio" name="view-control" id="view-images" aria-controls="doc-viewport" {{ if eq .Prefs.DocView "images" }}checked{{ end }}>
<input type="radio" name="view-control" id="view-split" aria-controls="doc-viewport" {{ if eq .Prefs.DocView "split" }}checked{{ end }}>
<input type="radio" name="view-control" id="view-text" aria-controls="doc-viewport" {{ if eq .Prefs.DocView "text" }}checked{{ end }}>
<div class="view-picker" role="radiogroup" aria-label="Document display mode">
<label for="view-images" class="btn-img">Images Only</label>
<label for="view-split" class="btn-split">Split View</label>
<label for="view-text" class="btn-txt">Text Only</label>
</div>
<div id="doc-viewport" class="content-wrapper{{ if .Prefs.ScrollDoc }} scroll-panes{{ end }}">
<div class="pane images-pane">
{{ if gt (len .Images) 1 }}
<nav class="page-jump" aria-label="Jump to page">
<span class="faint">Page:</span>
{{ range $i, $img := .Images }}<a href="#page-{{ add $i 1 }}"{{ if and $.Version ($.Version.PageChanged (add $i 1)) }} class="page-changed-jump" title="Differs from the main document"{{ end }}>{{ add $i 1 }}</a>{{ end }}
</nav>
{{ end }}
<div class="image-list">
{{ if .Prefs.TextOnly }}
<p class="muted textonly-note">Text-only mode is on - page images are not loaded. <a href="/preferences">Preferences</a></p>
<ul class="page-link-list">
{{ range $i, $img := .Images }}
<li id="page-{{ add $i 1 }}"><a href="{{ $.ImageBase }}{{ pathEscape $img }}" target="_blank" rel="noopener">Page {{ add $i 1 }}</a>{{ if and $.Version ($.Version.PageChanged (add $i 1)) }} <span class="muted">(differs from the main document)</span>{{ end }}</li>
{{ end }}
</ul>
{{ else }}
{{ range $i, $img := .Images }}
<a href="#lb-{{ add $i 1 }}" id="page-{{ add $i 1 }}" class="page-open{{ if and $.Version ($.Version.PageChanged (add $i 1)) }} page-changed{{ end }}" title="Open page {{ add $i 1 }} full screen">
<img src="{{ $.ImageBase }}{{ pathEscape $img }}"{{ $.ImageSize $i }}
alt="Page {{ add $i 1 }}"
loading="lazy">
{{- if and $.Version ($.Version.PageChanged (add $i 1)) }}<span class="page-changed-badge">differs</span>{{ end }}
</a>
{{ end }}
{{ end }}
</div>
</div>
<div class="pane text-pane">
<div class="prose">
{{ .Content }}
</div>
</div>
</div>
{{ end }}

46
templates/pinned.html Normal file
View file

@ -0,0 +1,46 @@
{{ define "title" }}Pinned - Snowden Archive{{ end }}
{{ define "content" }}
<header class="list-header">
<h1>Pinned documents
<small class="count-note">{{ .Total }} pinned</small>
{{ if .Total }}<a href="/pinned/export" class="export-link" title="Download your pinned list as JSON">export JSON</a>{{ end }}
</h1>
</header>
<p class="muted">Pins are stored only in this browser's cookies.</p>
{{ if .Full }}
<p class="muted" role="status"><strong>Your pinned list is full.</strong> A browser cookie holds only so much, so that pin was not added. Unpin a document or a search below to make room.</p>
{{ end }}
{{ if .Cards.Docs }}
<h2 class="sr-only">Documents</h2>
{{ template "doc-cards" .Cards }}
{{ else }}
<p class="muted">Nothing pinned yet. Open a document and press <strong>&#9734; Pin</strong> to start a reading list.</p>
{{ end }}
{{ if .Searches }}
<h2 class="pinned-searches-heading">Pinned searches</h2>
<ul class="pinned-searches">
{{ range .Searches }}
<li class="pinned-search-item">
<a class="pill" href="{{ .URL }}">{{ .Label }}</a>
<form method="POST" action="/pinsearch" class="inline-form">
<input type="hidden" name="qs" value="{{ .QS }}">
<input type="hidden" name="back" value="/pinned">
<button type="submit" class="link-btn unpin-btn" title="Remove this search">&#9733; unpin</button>
</form>
</li>
{{ end }}
</ul>
{{ end }}
<section class="note-list-box" id="note-list-box" hidden>
<h2>My notes
<button type="button" class="export-link" id="note-export" title="Download all notes as JSON">export JSON</button>
</h2>
<p class="muted">Stored only in this browser (localStorage).</p>
<ul class="note-list" id="note-list"></ul>
</section>
{{ end }}
{{ template "base.html" . }}

82
templates/portions.html Normal file
View file

@ -0,0 +1,82 @@
{{ define "title" }}Portion markings - Snowden Archive{{ end }}
{{ define "content" }}
<header class="list-header">
<p class="crumb"><a href="/explore">&larr; Explore</a></p>
<h1>Portion markings</h1>
</header>
<p class="muted">The marking each classified portion of a document carries at the
start of its own line (for example "<code>(U//FOUO)</code>", "<code>(TS//SI//NF)</code>").</p>
<h2>By level</h2>
<div class="table-scroll">
<table class="cat-table explore-table">
<thead><tr><th>Level</th><th></th><th class="num">Markings</th><th class="num">Documents</th></tr></thead>
<tbody>
{{ range .Stats.Levels }}
<tr>
<td><a href="{{ .SearchURL }}"><code>{{ .Name }}</code></a></td>
<td class="faint">{{ .Label }}</td>
<td class="num">{{ .Marks }}</td>
<td class="num">{{ .Docs }}</td>
</tr>
{{ end }}
</tbody>
</table>
</div>
<h2>Markings</h2>
<p class="muted">Every distinct marking, most used first.
{{ if gt .Stats.Distinct .RowLimit }}Top {{ .RowLimit }} of {{ .Stats.Distinct }}.{{ end }}</p>
<div class="table-scroll">
<table class="cat-table explore-table">
<thead><tr><th>Marking</th><th class="num">Markings</th><th class="num">Documents</th></tr></thead>
<tbody>
{{ range .Stats.Markings }}
<tr>
<td><a href="{{ .SearchURL }}"><code>{{ .Name }}</code></a></td>
<td class="num">{{ .Marks }}</td>
<td class="num">{{ .Docs }}</td>
</tr>
{{ end }}
</tbody>
</table>
</div>
<h2>Control markings</h2>
<p class="muted">The compartments and dissemination controls appearing inside
those markings.</p>
<div class="table-scroll">
<table class="cat-table explore-table">
<thead><tr><th>Control</th><th class="num">Markings</th><th class="num">Documents</th></tr></thead>
<tbody>
{{ range .Stats.Controls }}
<tr>
<td><a href="{{ .SearchURL }}"><code>{{ .Name }}</code></a></td>
<td class="num">{{ .Marks }}</td>
<td class="num">{{ .Docs }}</td>
</tr>
{{ end }}
</tbody>
</table>
</div>
{{ if .Stats.UnrecogEg }}
<h2>Not recognised</h2>
<p class="muted">{{ .Stats.Unrecog }} markings open a line with a classification
level but carry a part this server has no rule for.</p>
<div class="table-scroll">
<table class="cat-table explore-table">
<thead><tr><th>As the transcript writes it</th><th class="num">Occurrences</th></tr></thead>
<tbody>
{{ range .Stats.UnrecogEg }}
<tr><td><code>({{ .Name }})</code></td><td class="num">{{ .Marks }}</td></tr>
{{ end }}
</tbody>
</table>
</div>
{{ end }}
{{ end }}
{{ template "base.html" . }}

View file

@ -0,0 +1,97 @@
{{ define "title" }}Preferences - Snowden Archive{{ end }}
{{ define "content" }}
<h1>Preferences</h1>
<p class="muted">Preferences are saved in browser cookies.</p>
{{ if .Saved }}<p class="pref-saved">Settings saved.</p>{{ end }}
{{ if .Reset }}<p class="pref-saved">Preferences reset to defaults.</p>{{ end }}
<form action="/preferences/save" method="POST" class="pref-form">
<fieldset>
<legend>Theme</legend>
<label><input type="radio" name="theme" value="auto" {{ if eq .Prefs.Theme "auto" }}checked{{ end }}> Auto (follow browser)</label>
<label><input type="radio" name="theme" value="light" {{ if eq .Prefs.Theme "light" }}checked{{ end }}> Light</label>
<label><input type="radio" name="theme" value="dark" {{ if eq .Prefs.Theme "dark" }}checked{{ end }}> Dark</label>
<label><input type="checkbox" name="text_only" value="1" {{ if .Prefs.TextOnly }}checked{{ end }}> Text-only mode - no thumbnails or page images, links instead</label>
</fieldset>
<fieldset>
<legend>Browse lists</legend>
<label>Documents per page:
<select name="per_page">
<option value="10" {{ if eq .Prefs.PerPage 10 }}selected{{ end }}>10</option>
<option value="20" {{ if eq .Prefs.PerPage 20 }}selected{{ end }}>20</option>
<option value="50" {{ if eq .Prefs.PerPage 50 }}selected{{ end }}>50</option>
<option value="100" {{ if eq .Prefs.PerPage 100 }}selected{{ end }}>100</option>
<option value="all" {{ if eq .Prefs.PerPage 0 }}selected{{ end }}>All (up to {{ .MaxCards }} a page)</option>
</select>
</label>
<label><input type="checkbox" name="hide_classification" value="1" {{ if not .Prefs.ShowClassification }}checked{{ end }}> Hide classification and dissemination markings on document cards</label>
<label><input type="checkbox" name="hide_journalists" value="1" {{ if not .Prefs.ShowJournalists }}checked{{ end }}> Hide journalist names on document cards</label>
<label><input type="checkbox" name="hide_people" value="1" {{ if not .Prefs.ShowPeople }}checked{{ end }}> Hide the people line on document cards</label>
<label><input type="checkbox" name="hide_programs" value="1" {{ if not .Prefs.ShowPrograms }}checked{{ end }}> Hide program names on document cards</label>
<label><input type="checkbox" name="hide_summary" value="1" {{ if not .Prefs.ShowSummary }}checked{{ end }}> Hide short summaries on document cards</label>
<label><input type="checkbox" name="hide_thumbs" value="1" {{ if not .Prefs.ShowThumbs }}checked{{ end }}> Hide thumbnails on search results and document cards</label>
<label><input type="checkbox" name="show_coll_cols" value="1" {{ if .Prefs.ShowCollCols }}checked{{ end }}> Always show the per-type count columns on category tables <span class="muted pref-note">(Reported / SIDtoday / FOIA breakdown - otherwise behind a toggle on each table)</span></label>
<label><input type="checkbox" name="page_preview" value="1" {{ if .Prefs.CardPagePreview }}checked{{ end }}> Page preview on card-thumbnail hover <span class="muted pref-note">(desktop only - hover a thumbnail to preview the full pages; press h / l or the arrow keys to flip pages.)</span></label>
</fieldset>
<fieldset>
<legend>FOIA releases</legend>
<label><input type="checkbox" name="include_foia" value="1" {{ if .Prefs.IncludeFOIA }}checked{{ end }}> Include FOIA releases <span class="muted pref-note">(official releases from FOIA litigation and FISC and ODNI declassifications, not Snowden documents)</span></label>
</fieldset>
<fieldset>
<legend>Document layout</legend>
<label>Default view:
<select name="doc_view">
<option value="images" {{ if eq .Prefs.DocView "images" }}selected{{ end }}>Images only</option>
<option value="split" {{ if eq .Prefs.DocView "split" }}selected{{ end }}>Split view (images + text)</option>
<option value="text" {{ if eq .Prefs.DocView "text" }}selected{{ end }}>Text only</option>
</select>
</label>
<label>Document info:
<select name="info_position">
<option value="above" {{ if not .Prefs.InfoSidebar }}selected{{ end }}>Above the document</option>
<option value="beside" {{ if .Prefs.InfoSidebar }}selected{{ end }}>Beside the document (sidebar)</option>
</select>
<span class="muted pref-note">(desktop only)</span>
</label>
<label><input type="checkbox" name="scroll_doc" value="1" {{ if .Prefs.ScrollDoc }}checked{{ end }}> Scroll within the document panes <span class="muted pref-note">(cap the image/text area height instead of letting a long document stretch the page; ignored when the info sidebar is on)</span></label>
</fieldset>
<fieldset>
<legend>Document details</legend>
<label><input type="checkbox" name="open_info" value="1" {{ if .Prefs.OpenInfo }}checked{{ end }}> Expand &ldquo;More info&rdquo; by default</label>
<label><input type="checkbox" name="collapse_pub" value="1" {{ if .Prefs.CollapsePubTogether }}checked{{ end }}> Collapse &ldquo;Published in the same report&rdquo; by default</label>
<label><input type="checkbox" name="collapse_related" value="1" {{ if .Prefs.CollapseRelated }}checked{{ end }}> Collapse &ldquo;Related documents&rdquo; by default</label>
<label><input type="checkbox" name="programs_list" value="1" {{ if .Prefs.ProgramsAsList }}checked{{ end }}> Show programs one per line <span class="muted pref-note">(instead of comma-separated)</span></label>
<label><input type="checkbox" name="tags_list" value="1" {{ if .Prefs.TagsAsList }}checked{{ end }}> Show tags one per line <span class="muted pref-note">(instead of comma-separated)</span></label>
</fieldset>
<fieldset>
<legend>History <span class="muted pref-note">(off by default - nothing is recorded unless you opt in; saved in this browser's cookies; turning this off deletes the history)</span></legend>
<label><input type="checkbox" name="track_recent" value="1" {{ if .Prefs.TrackRecent }}checked{{ end }}> Remember recently viewed documents (shown on the home page)</label>
<label><input type="checkbox" name="track_searches" value="1" {{ if .Prefs.TrackSearches }}checked{{ end }}> Remember recent searches (shown on the search page)</label>
</fieldset>
<div class="pref-actions">
<button type="submit" class="pref-save">Save preferences</button>
<button type="submit" name="reset" value="1" class="pref-reset">Reset to defaults</button>
</div>
</form>
<details id="custom-css-details" class="custom-css-details" hidden>
<summary>Custom CSS <span class="muted pref-note">(for advanced users)</span></summary>
<p class="muted pref-note">Stored only in the localStorage and applied on every page. To turn it off, empty the box and save. If a bad rule ever makes the site unusable, open any page with <code>?nocss=1</code> (e.g. <code>/preferences?nocss=1</code>).</p>
<textarea id="custom-css" rows="8" spellcheck="false" placeholder="/* e.g. */
.prose { font-size: 1.05rem; line-height: 1.8; }
.doc-card { border-width: 2px; }"></textarea>
<div class="custom-css-actions">
<button type="button" id="custom-css-save" class="ctl">Save CSS</button>
<span class="custom-css-status faint" id="custom-css-status"></span>
</div>
</details>
{{ end }}
{{ template "base.html" . }}

280
templates/search.html Normal file
View file

@ -0,0 +1,280 @@
{{ define "title" }}{{ with .QueryLabel }}{{ . }} - {{ end }}Search - Snowden Archive{{ end }}
{{ define "search-result-item" }}
{{ $r := .Result }}{{ $d := $r.Doc }}
{{ $qs := "" }}{{ if eq $d.Collection "foia" }}{{ $qs = .FOIAQS }}{{ end }}
<li class="doc-card search-result">
{{ if .ShowThumbs }}
<a href="/doc/{{ $d.ID }}" class="result-thumb" tabindex="-1" aria-hidden="true"{{ if and .CardPagePreview $d.Images }} data-pp-id="{{ $d.ID }}" data-pp-imgs="{{ range $i, $im := $d.Images }}{{ if $i }} {{ end }}{{ pathEscape $im }}{{ end }}"{{ end }}>
{{ if $d.HasThumb }}
<img src="/files/{{ $d.ID }}/thumb.jpg"{{ if $d.HasGifThumb }} data-gif="/files/{{ $d.ID }}/thumb.gif"{{ end }}
class="doc-img" alt="" loading="lazy">
{{ end }}
</a>
{{ end }}
<div class="result-body">
<h3 class="result-title">
<a href="/doc/{{ $d.ID }}" class="result-title-link">{{ $r.Title }}</a>
<a href="/files/{{ $d.ID }}/{{ pathEscape $d.PDFFile }}" target="_blank" rel="noopener" class="doc-pdf">[pdf]</a>
</h3>
<span class="doc-type">{{ template "type-badge" $d.Type }}</span>
{{ if $d.Publication }}
{{/* A year links to its publication-year page only when that page
lists the document: the year of its FIRST publication. */}}
<div class="result-pubs">
{{- $shown := false -}}
{{- range $pub := $d.Publication -}}
{{- if $pub.Publisher -}}
{{- if $shown }} &middot; {{ end }}{{ $shown = true -}}
{{ template "pub-publisher" (dict "Pub" $pub "QS" $qs) }}
{{- if ge (len $pub.DatePublished) 4 }}{{ $y := slice $pub.DatePublished 0 4 }} {{ if eq $y $d.FirstPubYear }}<a href="/browse/published/{{ $y }}{{ $qs }}" class="meta-link-softer">{{ $y }}</a>{{ else }}<span class="meta-link-softer">{{ $y }}</span>{{ end }}{{ end -}}
{{- range $j, $r := $pub.Journalists }}{{ if $j }}, {{ else }} - {{ end }}<a href="/browse/journalist/{{ pathEscape $r }}{{ $qs }}">{{ $r }}</a>{{ end -}}
{{- end -}}
{{- end }}
</div>
{{ end }}
<div class="result-meta">
{{ if $d.DocMeta.Creator }}
<a href="/browse/agency/{{ pathEscape $d.DocMeta.Creator }}{{ $qs }}" class="meta-link-soft">{{ $d.DocMeta.Creator }}</a>
{{ end }}
{{ if and $d.DocMeta.Creator $d.DocMeta.DateCreated }}
<span class="doc-dot">·</span>
{{ end }}
{{ if $d.DocMeta.DateCreated }}
{{ if ge (len $d.DocMeta.DateCreated) 4 }}
<a href="/browse/docdate/{{ slice $d.DocMeta.DateCreated 0 4 }}{{ $qs }}" class="meta-link-softer">{{ $d.DocMeta.DateCreated }}</a>
{{ else }}
<span class="meta-link-softer">{{ $d.DocMeta.DateCreated }}</span>
{{ end }}
{{ end }}
{{ if $d.DocMeta.Programs }}
<span class="doc-dot">·</span>
{{ $progs := $d.DocMeta.Programs }}
{{ $shown := $progs }}{{ if gt (len $progs) 8 }}{{ $shown = slice $progs 0 8 }}{{ end }}
{{ facetLinks (dict "Vals" $shown "Cat" "program" "Sep" "dot" "Class" "meta-link-soft" "QS" $qs) }}
{{ if gt (len $progs) 8 }}<span class="faint">+{{ add (len $progs) -8 }} more</span>{{ end }}
{{ end }}
</div>
<div class="result-desc">{{ $r.Desc }}</div>
{{ range $r.Snippets }}
<div class="search-result-snippet">{{ . }}</div>
{{ end }}
{{ range $r.VersionSnippets }}
<div class="search-result-snippet version-snippet"><a href="/doc/{{ $d.ID }}/versions/{{ .Slug }}" class="snippet-source">Copy: {{ .Label }}</a> {{ .HTML }}</div>
{{ end }}
{{ with $r.MatchesIn }}
<div class="result-matches-in muted">Matches in:
{{- range $i, $v := . }}{{ if $i }},{{ end }} <a href="/doc/{{ $d.ID }}/versions/{{ $v.Slug }}">{{ $v.Label }}</a>{{ end }}
</div>
{{ end }}
</div>
</li>
{{ end }}
{{ define "content" }}
<h1 class="search-title">Search</h1>
<details class="search-ops-hint">
<summary>Search syntax help</summary>
<p>
Ways to write a value, in any row or after any operator:
<ul>
<li><code>cable tapping</code> finds both words anywhere, in their other forms too (<em>tapping</em> also finds <em>tapped</em>);</li>
<li><code>"apping"</code> finds that text anywhere, even inside a word, like <code>grep -i</code> (it finds for example <em>mapping</em> and <em>tapping</em>);</li>
<li><code>=tap</code> or <code>="cable tapping"</code> finds exactly those whole words, so <code>=tap</code> skips <em>taps</em> and <em>strap</em>, and on a metadata field it matches one whole value, so <code>agency:=NSA</code> finds documents whose agency is NSA but not <em>NSA, GCHQ</em>;</li>
<li><code>*</code> wildcard finds part of a word: <code>tap*</code> finds words starting with <em>tap</em>, <code>*land</code> words ending in <em>land</em> and <code>*keys*</code> words containing <em>keys</em>. A leading <code>*</code> needs at least three letters after it.</li>
<li><code>"upstream collection"~10</code> finds the words within 10 words of each other (in any order, common stopwords ignored).</li>
<li>A leading <code>-</code> excludes within its row: <code>xkeyscore -tempora</code> finds <em>xkeyscore</em> without <em>tempora</em>, and the row keeps that meaning under OR and NOT.</li>
</ul>
</p>
<p>
The simple search box (on home page, on 404 page, and via URL <code>?q=</code>) understands field operators:
<ul>
<li><code>program:XKEYSCORE country:Germany cable</code> or <code>journalist:="Poitras, Laura"</code>.</li>
<li>A leading <code>-</code> negates (<code>-tag:=junk</code>, <code>-draft</code>);</li>
<li><code>date:2013</code>, <code>date:2013..2015</code>, and <code>published:2014-06</code> filter by date.</li>
<li>Operators: text, title, content, description, agency, journalist, publisher, classification, program, dissemination, banner, genre, series, country, people, tag, type.</li>
<li>Bare words search all fields.</li>
</ul>
</p>
</details>
<form method="GET" action="/search" class="search-form search-panel">
<input type="hidden" name="do_search" value="1">
<button type="submit" name="action" value="search" aria-hidden="true" tabindex="-1" class="sr-only">Search</button>
<div class="query-rows">
{{ range $i, $val := .Queries }}
<div class="query-row">
<div class="qr-conj">
{{ if eq $i 0 }}
<span class="faint">Find</span>
<input type="hidden" name="fqc" value="and">
{{ else }}
<select name="fqc" aria-label="How row {{ add $i 1 }} combines with the rows above">
<option value="and" {{ if eq (index $.Conjs $i) "and" }}selected{{ end }}>and</option>
<option value="or" {{ if eq (index $.Conjs $i) "or" }}selected{{ end }}>or</option>
<option value="not" {{ if eq (index $.Conjs $i) "not" }}selected{{ end }}>and not</option>
</select>
{{ end }}
</div>
<input type="text" name="fqv" value="{{ $val }}" class="qr-term" aria-label="Search terms, row {{ add $i 1 }}"
placeholder='word, "text", =exact or *part*'>
<div class="qr-field">
<span class="faint qr-in">in</span>
{{ $fi := index $.Fields $i }}
<select name="fqf" aria-label="Field to search, row {{ add $i 1 }}">
{{ range $.FieldOptions }}
<option value="{{ .Code }}" {{ if eq $fi .Code }}selected{{ end }}>{{ .Label }}</option>
{{ end }}
</select>
</div>
{{ if ne $i 0 }}
<button type="submit" name="action" value="remove-{{ $i }}"
class="remove-row-btn" title="Remove this row">&times;</button>
{{ else }}
<span class="remove-row-spacer"></span>
{{ end }}
</div>
{{ end }}
<div class="query-row query-row-add">
<div class="qr-conj"></div>
{{ if lt (len .Queries) .ClauseLimit }}
<button type="submit" name="action" value="add" class="add-row-btn">+ Add row</button>
{{ else }}
<span class="faint">{{ .ClauseLimit }} rows is the most a search can have</span>
{{ end }}
</div>
</div>
{{ $hasDates := or .DateDocStart .DateDocEnd .DatePubStart .DatePubEnd }}
{{ $hasFilters := or $hasDates .CollFiltered }}
<details class="date-filters"{{ if $hasFilters }} open{{ end }}>
<summary>Filters{{ if $hasFilters }} <span class="faint">(active)</span>{{ end }}</summary>
<div class="coll-filter-row">
<span class="faint df-label">Collections</span>
<label class="coll-check"><input type="checkbox" name="coll" value="news"{{ if index .SelColl "news" }} checked{{ end }}> Reported</label>
<label class="coll-check"><input type="checkbox" name="coll" value="sidtoday-archive"{{ if index .SelColl "sidtoday-archive" }} checked{{ end }}> SIDtoday archive</label>
<label class="coll-check"><input type="checkbox" name="coll" value="foia"{{ if index .SelColl "foia" }} checked{{ end }}> FOIA</label>
</div>
<div class="date-filter-rows">
<div class="date-filter-row">
<span class="faint df-label">Doc date</span>
<input type="text" name="date_doc_start" value="{{ .DateDocStart }}" aria-label="Document date from" placeholder="yyyy-mm-dd">
<span class="faint">to</span>
<input type="text" name="date_doc_end" value="{{ .DateDocEnd }}" aria-label="Document date to" placeholder="yyyy-mm-dd">
</div>
<div class="date-filter-row">
<span class="faint df-label">Published</span>
<input type="text" name="date_pub_start" value="{{ .DatePubStart }}" aria-label="Published from" placeholder="yyyy-mm-dd">
<span class="faint">to</span>
<input type="text" name="date_pub_end" value="{{ .DatePubEnd }}" aria-label="Published to" placeholder="yyyy-mm-dd">
</div>
</div>
</details>
<div class="search-actions">
<button type="submit" name="action" value="search" class="search-btn">Search</button>
<a href="/search" class="faint reset-link">Reset</a>
</div>
</form>
{{ if .RecentSearches }}
<details class="recent-searches">
<summary>Recent searches <span class="faint">({{ len .RecentSearches }})</span></summary>
<div class="recent-list">
{{ range .RecentSearches }}<a class="pill" href="{{ .URL }}">{{ .Label }}</a>{{ end }}
<form method="POST" action="/history/clear" class="inline-form">
<input type="hidden" name="which" value="searches">
<input type="hidden" name="back" value="/search">
<button type="submit" class="link-btn faint">clear</button>
</form>
</div>
</details>
{{ end }}
<hr class="hr-soft">
{{ if .OverClauses }}
<p class="muted"><strong>Search not run:</strong> it has {{ .OverClauses }} parts - rows, or the
<code>field:</code>, &ldquo;quoted&rdquo; and <code>-excluded</code> terms of a typed query -
and a search can have at most {{ .ClauseLimit }}. The form above keeps the first {{ .ClauseLimit }};
the rest, including any exclusions at the end, were left out. Check the rows and search again.</p>
{{ else if .OverWords }}
<p class="muted"><strong>Search not run:</strong> it has {{ .OverWords }} words, counting the words of
&ldquo;quoted&rdquo; phrases and <code>-excluded</code> terms, and a search can have at most
{{ .WordLimit }}. Shorten it and search again.</p>
{{ end }}
{{ if .Searched }}
{{ if gt (len .SearchTypeCounts) 1 }}{{ template "type-filter" (dict "Action" "/search" "Params" .FormParams "AllTypes" .SearchTypeCounts "SelectedTypes" .SelTypeSet) }}{{ end }}
<div class="results-bar">
<h2 class="results-heading">{{ .Total }} result{{ if ne .Total 1 }}s{{ end }}</h2>
{{ template "sort-bar" (dict "SortURL" .SearchBaseURL "SortBy" .Sort "Order" .Order "ShowRelevance" true "DefaultSort" "relevance") }}
{{ if .Savable }}
<form method="POST" action="/pinsearch" class="inline-form">
<input type="hidden" name="qs" value="{{ .CanonicalQS }}">
<input type="hidden" name="back" value="{{ .SearchBaseURL }}">
<button type="submit" class="export-link pin-search-btn" title="{{ if .SearchPinned }}Remove this search from your pinned list{{ else }}Pin this search to your pinned list{{ end }}">{{ if .SearchPinned }}&#9733; unpin search{{ else }}&#9734; pin search{{ end }}</button>
</form>
{{ end }}
</div>
{{ if gt (len .DateHist) 1 }}
<div class="date-hist" aria-label="Matches per document year">
{{ range .DateHist }}
{{ if .URL }}
<a class="hist-col" href="{{ .URL }}" title="{{ .Count }} matches dated {{ .Year }}">
<span class="hist-count">{{ .Count }}</span>
<span class="hist-track"><span class="hist-bar" style="height: {{ .Pct }}%"></span></span>
<span class="hist-year">{{ .Year }}</span>
</a>
{{ else }}
<span class="hist-col hist-col-undated" title="{{ .Count }} matches with no document date">
<span class="hist-count">{{ .Count }}</span>
<span class="hist-track"><span class="hist-bar" style="height: {{ .Pct }}%"></span></span>
<span class="hist-year">{{ .Year }}</span>
</span>
{{ end }}
{{ end }}
</div>
{{ end }}
{{ if .Results }}
{{ if .Facets }}
<details class="search-facets">
<summary>Refine results</summary>
<div class="facets-body">
{{ range .Facets }}
<div class="facet-group">
<span class="facet-label">{{ .Label }}</span>
<span class="facet-values">{{ range .Values }}<a class="facet-link" href="{{ .URL }}">{{ .Name }}<span class="facet-count">{{ .Count }}</span></a>{{ end }}</span>
</div>
{{ end }}
</div>
</details>
{{ end }}
<ul class="search-results{{ if .ShowThumbs }} has-thumbs{{ end }}">
{{ $showThumbs := .ShowThumbs }}
{{ range .Results }}{{ template "search-result-item" (dict "Result" . "ShowThumbs" $showThumbs "CardPagePreview" $.CardPagePreview "FOIAQS" $.FOIAQS) }}{{ end }}
</ul>
{{ if .SnippetsCapped }}
<p class="muted">Text snippets are shown for the first {{ .SnippetLimit }} results on this page -
choose a page size in <a href="/preferences">preferences</a>, or narrow the search, to see
snippets for the rest.</p>
{{ end }}
{{ template "pagination" (dict "Page" .Page "TotalPages" .TotalPages "BaseURL" .SearchBaseURL "Sort" .Sort "Order" .Order) }}
{{ else }}
<p class="muted">No matches found.</p>
{{ end }}
{{ end }}
{{ end }}
{{ template "base.html" . }}

59
templates/version.html Normal file
View file

@ -0,0 +1,59 @@
{{ define "title" }}{{ .Doc.Title }} ({{ .Version.Slug }}) - Snowden Archive{{ end }}
{{ define "content" }}
<header class="list-header">
<p class="crumb"><a href="/doc/{{ .Doc.ID }}">&larr; {{ .Doc.Title }}</a></p>
<h1>{{ .Doc.Title }} <small class="count-note">copy: {{ .Version.Slug }}</small></h1>
<p class="muted">Another published copy of this document that differs from the one on the
<a href="/doc/{{ .Doc.ID }}">document page</a> - for example less redacted, with extra pages, or a different scan.</p>
</header>
<div class="version-actions">
<a href="{{ .PDFURL }}" target="_blank" rel="noopener" class="doc-top-btn doc-pdf-link">Download this copy (PDF)</a>
{{ with .Version.Images }}<a href="/doc/{{ $.Doc.ID }}/versions/{{ $.Version.Slug }}/pages" class="doc-top-btn">All pages ({{ len . }})</a>{{ end }}
</div>
{{ if .Version.ChangedPages }}
<p class="changed-pages-note">Pages that differ from the main document: {{ template "changed-page-links" .Version }}</p>
{{ end }}
<div class="rule-top pub-block">
{{- range $i, $pub := .Publications }}
<div class="pub-entry{{ if $i }} pub-sep{{ end }}">
<div class="pub-title">
{{- with $pub.Publisher }}{{ . }}{{ else }}Unknown publisher{{ end -}}
{{- with $pub.DatePublished }} <span class="muted">{{ . }}</span>{{ end -}}
</div>
{{- if $pub.Title }}
<div class="pub-people"><em>{{ $pub.Title }}</em>{{ with $pub.Link }} <a href="{{ . }}" target="_blank" rel="noopener" class="pub-original">article &nearr;</a>{{ end }}</div>
{{- else if $pub.Link }}
<div class="pub-people"><a href="{{ $pub.Link }}" target="_blank" rel="noopener" class="pub-original">article &nearr;</a></div>
{{- end }}
{{- with $pub.Note }}
<div class="pub-note">{{ . }}</div>
{{- end }}
{{- if or $pub.DocSource $pub.DocURL }}
<div class="pub-copy muted">
{{- with $pub.DocSource }}<a href="{{ . }}" target="_blank" rel="noopener">document page &nearr;</a>{{ end -}}
{{- with $pub.DocURL }}<a href="{{ . }}" target="_blank" rel="noopener">original PDF &nearr;</a>{{ end -}}
</div>
{{- end }}
</div>
{{- else }}
<p class="muted">No publication record points at this copy.</p>
{{- end }}
</div>
{{ if or .Images .Version.HTMLContent }}
<div class="doc-container">
{{ template "doc-viewport" . }}
</div>
{{ else }}
<p class="muted rule-top">This copy has no transcript and no page images - only the PDF above.</p>
{{ end }}
{{ if and .Images (not .Prefs.TextOnly) }}
{{ template "lightbox-overlays" . }}
{{ end }}
{{ end }}
{{ template "base.html" . }}

122
tests/api_test.go Normal file
View file

@ -0,0 +1,122 @@
package main
import (
"encoding/json"
"net/http"
"net/http/httptest"
"net/url"
"path/filepath"
"testing"
)
func TestAPIPageSize(t *testing.T) {
for raw, want := range map[string]int{
"": apiPerPageDefault,
"25": 25,
"500": 500,
"1000": apiMaxPerPage,
"99999999": apiMaxPerPage,
"0": apiPerPageDefault, // not a page size; fall back
"-5": apiPerPageDefault,
"lots": apiPerPageDefault,
"all": 0,
} {
v := url.Values{}
if raw != "" {
v.Set("per_page", raw)
}
if got := apiPageSize(v); got != want {
t.Errorf("apiPageSize(per_page=%q) = %d, want %d", raw, got, want)
}
}
}
func TestAPIRespPerPage(t *testing.T) {
for _, c := range []struct{ perPage, total, want int }{
{50, 3106, 50},
{0, 3106, 3106},
{0, 0, 1},
} {
if got := apiRespPerPage(c.perPage, c.total); got != c.want {
t.Errorf("apiRespPerPage(%d, %d) = %d, want %d", c.perPage, c.total, got, c.want)
}
}
}
func TestAPIVersionsShape(t *testing.T) {
root := t.TempDir()
writeTestFile(t, filepath.Join(root, "doc-a", "metadata.json"), `{
"title": "A", "type": "REPORTED",
"publication": [{"publisher": "Le Monde", "version": "entier"}],
"versions": {"entier": {"changed_pages": [1]}}
}`)
writeTestFile(t, filepath.Join(root, "doc-a", "a.pdf"), "%PDF-1.4")
writeTestFile(t, filepath.Join(root, "doc-a", "versions", "entier", "entier.pdf"), "%PDF-1.4")
withCorpus(t, root)
type versionsObj struct {
Versions map[string]VersionInfo `json:"versions"`
}
var detail struct {
versionsObj
VersionCopies []apiVersion `json:"version_copies"`
}
decode := func(path string, v interface{}) {
t.Helper()
w := get(t, path)
if w.Code != 200 {
t.Fatalf("%s: %d", path, w.Code)
}
if err := json.Unmarshal(w.Body.Bytes(), v); err != nil {
t.Fatalf("%s: %v", path, err)
}
}
decode("/api/doc/doc-a", &detail)
if len(detail.Versions["entier"].ChangedPages) != 1 {
t.Errorf("/api/doc versions = %+v, want the metadata object", detail.Versions)
}
if len(detail.VersionCopies) != 1 || detail.VersionCopies[0].Slug != "entier" {
t.Errorf("/api/doc version_copies = %+v, want the entier copy", detail.VersionCopies)
}
var list struct{ Docs []versionsObj }
decode("/api/docs", &list)
var search struct{ Results []versionsObj }
decode("/api/search?q=A", &search)
for name, docs := range map[string][]versionsObj{"/api/docs": list.Docs, "/api/search": search.Results} {
if len(docs) != 1 || len(docs[0].Versions["entier"].ChangedPages) != 1 {
t.Errorf("%s versions = %+v, want the metadata object", name, docs)
}
}
}
func TestAPIFacetsRevalidates(t *testing.T) {
w := get(t, "/api/facets")
tag := w.Header().Get("ETag")
if w.Code != 200 || tag == "" || w.Header().Get("Access-Control-Allow-Origin") != "*" {
t.Fatalf("/api/facets: %d, ETag %q, CORS %q", w.Code, tag, w.Header().Get("Access-Control-Allow-Origin"))
}
mux, _ := buildFixture(t)
r := httptest.NewRequest("GET", "/api/facets", nil)
r.Header.Set("If-None-Match", tag)
if w := record(mux, r); w.Code != http.StatusNotModified || w.Body.Len() != 0 {
t.Errorf("revalidation: %d with %d bytes, want 304 and none", w.Code, w.Body.Len())
}
}
// /api/docs takes coll= as /api/search and the pages do.
func TestAPIDocsCollections(t *testing.T) {
total := func(query string) int {
var body struct{ Total int }
if err := json.Unmarshal(get(t, "/api/docs?"+query).Body.Bytes(), &body); err != nil {
t.Fatalf("%s: %v", query, err)
}
return body.Total
}
all, news, foia := total(""), total("coll=news"), total("coll=foia")
if news == 0 || foia == 0 || news+foia >= all {
t.Fatalf("fixture totals: %d documents, %d news, %d foia", all, news, foia)
}
if got := total("coll=news&coll=foia"); got != news+foia {
t.Errorf("coll=news&coll=foia lists %d, want %d", got, news+foia)
}
}

211
tests/assemblies_test.go Normal file
View file

@ -0,0 +1,211 @@
package main
import (
"encoding/json"
"fmt"
"path/filepath"
"reflect"
"slices"
"strings"
"testing"
)
func writeAssemblyDoc(t *testing.T, root, id string, meta map[string]interface{}, pages []string, versions map[string][]string) {
t.Helper()
dir := filepath.Join(root, id)
b, err := json.Marshal(meta)
if err != nil {
t.Fatal(err)
}
writeFixtureFile(t, filepath.Join(dir, "metadata.json"), string(b))
writeFixtureFile(t, filepath.Join(dir, id+".pdf"), "%PDF-1.4\n")
for _, p := range pages {
writeFixtureFile(t, filepath.Join(dir, "images", p), fixtureJPEG)
}
for slug, imgs := range versions {
v := filepath.Join(dir, "versions", slug)
writeFixtureFile(t, filepath.Join(v, slug+".pdf"), "%PDF-1.4\n")
for _, p := range imgs {
writeFixtureFile(t, filepath.Join(v, "images", p), fixtureJPEG)
}
}
}
func assemblyCorpus(t *testing.T, manifest string) string {
t.Helper()
root := t.TempDir()
writeAssemblyDoc(t, root, "deck", map[string]interface{}{
"title": "The Deck",
"publication": []map[string]interface{}{
{"publisher": "Outlet A"},
{"publisher": "Partner", "reporting_only": true},
{"publisher": "Outlet B", "version": "fuller"},
},
}, []string{"page-1.jpg", "page-2.jpg", "page-3.jpg"}, map[string][]string{"fuller": {"page-1.jpg", "page-2.jpg"}})
writeAssemblyDoc(t, root, "single", map[string]interface{}{
"title": "One Slide",
"publication": []map[string]interface{}{{"publisher": "Outlet C"}},
}, []string{"page-1.jpg"}, nil)
writeAssemblyDoc(t, root, "bystander", map[string]interface{}{"title": "Bystander"}, []string{"page-1.jpg"}, nil)
if manifest != "" {
writeFixtureFile(t, filepath.Join(root, "assemblies.json"), manifest)
}
return root
}
const assemblyManifest = `[{
"id": "set", "title": "A Set", "note": "Order is a guess.",
"slides": [
{"doc": "deck", "pages": "1-2", "alt": [{"doc": "deck", "version": "fuller", "pages": "1-2"}]},
{"doc": "deck", "version": "fuller", "pages": "2", "note": "Only here."},
{"doc": "single", "pages": "1"}
]
}]`
func TestParsePageList(t *testing.T) {
for in, want := range map[string][]int{
"8": {8}, "1-3": {1, 2, 3}, "1,3,5-6": {1, 3, 5, 6}, " 2 , 4 ": {2, 4}, "9,1": {9, 1},
} {
got, err := parsePageList(in)
if err != nil || !reflect.DeepEqual(got, want) {
t.Errorf("parsePageList(%q) = %v, %v; want %v", in, got, err, want)
}
}
for _, in := range []string{"", "a", "0", "3-1", "1-", "1,,2"} {
if got, err := parsePageList(in); err == nil {
t.Errorf("parsePageList(%q) = %v, want an error", in, got)
}
}
}
func TestAssemblyResolves(t *testing.T) {
withCorpus(t, assemblyCorpus(t, assemblyManifest))
a := corpus().assemblyIndex["set"]
if a == nil {
t.Fatal("assembly not loaded")
}
if len(a.Slides) != 4 {
t.Fatalf("%d slides, want 4", len(a.Slides))
}
for i, s := range a.Slides {
if s.Pos != i+1 {
t.Errorf("slide %d has Pos %d", i+1, s.Pos)
}
}
first := a.Slides[0]
if first.Image != "/files/deck/images/page-1.jpg" || first.Source != "Outlet A" {
t.Errorf("first slide %+v", first)
}
if len(first.Alts) != 1 || first.Alts[0].Image != "/files/deck/versions/fuller/images/page-1.jpg" || first.Alts[0].Source != "Outlet B" {
t.Errorf("first slide alts %+v", first.Alts)
}
if a.Slides[2].Note != "Only here." || a.Slides[2].Path != "/doc/deck/versions/fuller" {
t.Errorf("third slide %+v", a.Slides[2])
}
if fmt.Sprint(a.DocList) != "[deck single]" {
t.Errorf("DocList %v", a.DocList)
}
if a.Copies != 3 {
t.Errorf("Copies %d, want 3", a.Copies)
}
if len(corpus().assembliesByDoc["single"]) != 1 || len(corpus().assembliesByDoc["bystander"]) != 0 {
t.Errorf("assembliesByDoc %v", corpus().assembliesByDoc)
}
}
func TestAssemblyPages(t *testing.T) {
withCorpus(t, assemblyCorpus(t, assemblyManifest))
h := withBaseHeaders(newMux())
body := func(path string, code int) string {
t.Helper()
w := record(h, httptest_NewGet(path))
if w.Code != code {
t.Fatalf("%s: %d, want %d", path, w.Code, code)
}
return w.Body.String()
}
list := body("/explore/assemblies", 200)
if !strings.Contains(list, `href="/explore/assemblies/set"`) || !strings.Contains(list, "4 slides from 3 copies") {
t.Errorf("/explore/assemblies:\n%s", list)
}
page := body("/explore/assemblies/set", 200)
for _, want := range []string{
"<title>A Set - Snowden Archive</title>", "from 3 separately published copies.", "Order is a guess.", "not a document in its own right",
`id="lb-4"`, `href="#lb-a1-1"`, `id="lb-a1-1"`, "Other copy: Outlet B", "Back to slide 1", "(copy: fuller)", "Outlet C", "&middot; Only here.",
} {
if !strings.Contains(page, want) {
t.Errorf("/explore/assemblies/set lacks %q", want)
}
}
if strings.Contains(page, `id="lb-5"`) || strings.Contains(page, "Partner") {
t.Errorf("/explore/assemblies/set has a fifth overlay or a reporting-only credit")
}
if !strings.Contains(body("/doc/single", 200), `href="/explore/assemblies/set"`) {
t.Errorf("/doc/single has no assembly link")
}
if strings.Contains(body("/doc/bystander", 200), "/explore/assemblies/") {
t.Errorf("/doc/bystander links an assembly")
}
// Assemblies live under /explore only: listed there, not on /browse.
if !strings.Contains(body("/explore", 200), `href="/explore/assemblies"`) {
t.Errorf("/explore has no assemblies link")
}
if strings.Contains(body("/browse", 200), "assemblies") {
t.Errorf("/browse still links assemblies")
}
body("/explore/assemblies/nope", 404)
if !slicesContains(corpus().sitemapPaths, "/explore/assemblies/set") {
t.Errorf("sitemap lacks the assembly page")
}
}
func TestAssemblyProblemsWarn(t *testing.T) {
cases := map[string]struct {
manifest string
slides int
loaded bool
}{
"missing doc": {`[{"id":"x","title":"X","slides":[{"doc":"ghost","pages":"1"},{"doc":"single","pages":"1"}]}]`, 1, true},
"missing page": {`[{"id":"x","title":"X","slides":[{"doc":"deck","pages":"3-4"}]}]`, 1, true},
"bad version": {`[{"id":"x","title":"X","slides":[{"doc":"deck","version":"no","pages":"1"},{"doc":"single","pages":"1"}]}]`, 1, true},
"bad pages": {`[{"id":"x","title":"X","slides":[{"doc":"deck","pages":"two"},{"doc":"single","pages":"1"}]}]`, 1, true},
"alt mismatch": {`[{"id":"x","title":"X","slides":[{"doc":"deck","pages":"1-2","alt":[{"doc":"single","pages":"1"}]}]}]`, 2, true},
"nothing left": {`[{"id":"x","title":"X","slides":[{"doc":"ghost","pages":"1"}]}]`, 0, false},
"unsafe id": {`[{"id":"a/b","title":"X","slides":[{"doc":"single","pages":"1"}]}]`, 0, false},
"malformed": {`[{`, 0, false},
}
for name, c := range cases {
t.Run(name, func(t *testing.T) {
withCorpus(t, assemblyCorpus(t, c.manifest))
if loadWarnings == 0 {
t.Errorf("no warning")
}
a := corpus().assemblyIndex["x"]
if (a != nil) != c.loaded {
t.Fatalf("loaded = %v, want %v", a != nil, c.loaded)
}
if a != nil && len(a.Slides) != c.slides {
t.Errorf("%d slides, want %d", len(a.Slides), c.slides)
}
})
}
}
func TestNoAssembliesFile(t *testing.T) {
withCorpus(t, assemblyCorpus(t, ""))
if len(corpus().assemblies) != 0 {
t.Fatalf("assemblies %v", corpus().assemblies)
}
h := withBaseHeaders(newMux())
if w := record(h, httptest_NewGet("/explore/assemblies")); w.Code != 200 {
t.Errorf("/explore/assemblies: %d", w.Code)
}
if w := record(h, httptest_NewGet("/explore")); strings.Contains(w.Body.String(), `href="/explore/assemblies"`) {
t.Errorf("/explore links assemblies with none loaded")
}
if slicesContains(corpus().sitemapPaths, "/explore/assemblies") {
t.Errorf("sitemap lists assemblies with none loaded")
}
}
func slicesContains(s []string, v string) bool { return slices.Contains(s, v) }

312
tests/bench_test.go Normal file
View file

@ -0,0 +1,312 @@
package main
import (
"fmt"
"net/http"
"net/http/httptest"
"os"
"runtime"
"sort"
"strconv"
"sync"
"testing"
"time"
)
// Page benchmarks against a REAL corpus - the fixture is four documents, so
// it cannot show what 3,000 do. Point $SNOWCHIVE_BENCH_ROOT at an archive
// root (the directory holding data/ and project/) and run:
// tests/run.sh -run XXX -bench . -benchtime 40x
//
// What they are for: every page but /search is built from data indexed at
// load, so a request should cost what its OUTPUT costs (template execution)
// and nothing more. A page whose time or allocations grow with the CORPUS
// rather than with the page has picked up a scan.
var (
benchOnce sync.Once
benchMux http.Handler
)
func benchCorpus(tb testing.TB) http.Handler {
root := os.Getenv("SNOWCHIVE_BENCH_ROOT")
if root == "" {
tb.Skip("set SNOWCHIVE_BENCH_ROOT to an archive root to run the page benchmarks")
}
benchOnce.Do(func() {
setRoot(root)
devMode = false
apiEnabled = true
initAssets()
if err := parseTemplates(); err != nil {
tb.Fatal(err)
}
loadData()
benchMux = withBaseHeaders(newMux())
})
return benchMux
}
func benchPage(b *testing.B, path string) {
h := benchCorpus(b)
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
r := httptest.NewRequest("GET", path, nil)
w := httptest.NewRecorder()
h.ServeHTTP(w, r)
if w.Code != 200 {
b.Fatalf("%s: %d", path, w.Code)
}
}
}
func BenchmarkPageHome(b *testing.B) { benchPage(b, "/") }
func BenchmarkPageBrowse(b *testing.B) { benchPage(b, "/browse") }
func BenchmarkPageArchiveTab(b *testing.B) { benchPage(b, "/browse?collection=sidtoday-archive") }
func BenchmarkPageCategoryList(b *testing.B) { benchPage(b, "/browse/program") }
func BenchmarkPageCategoryValue(b *testing.B) { benchPage(b, "/browse/agency/NSA") }
func BenchmarkPageDoc(b *testing.B) {
h := benchCorpus(b)
if len(corpus().archive.Docs) == 0 {
b.Skip("empty corpus")
}
benchPageWith(b, h, "/doc/"+corpus().archive.Docs[0].ID)
}
func BenchmarkPageExploreIDs(b *testing.B) { benchPage(b, "/explore/ids") }
func BenchmarkPageGlossary(b *testing.B) { benchPage(b, "/glossary") }
func BenchmarkPageSitemap(b *testing.B) { benchPage(b, "/sitemap.xml") }
func BenchmarkPageFacetsAPI(b *testing.B) { benchPage(b, "/api/facets") }
func BenchmarkPageSearch(b *testing.B) { benchPage(b, "/search?q=nsa&do_search=1") }
func benchPageWith(b *testing.B, h http.Handler, path string) {
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
r := httptest.NewRequest("GET", path, nil)
w := httptest.NewRecorder()
h.ServeHTTP(w, r)
if w.Code != 200 {
b.Fatalf("%s: %d", path, w.Code)
}
}
}
func BenchmarkPageCategoryTag(b *testing.B) { benchPage(b, "/browse/tag") }
func BenchmarkPageCategoryPeople(b *testing.B) { benchPage(b, "/browse/people") }
func BenchmarkPageCategoryTagSorted(b *testing.B) { benchPage(b, "/browse/tag?sort=count") }
func BenchmarkPageExploreMetric(b *testing.B) { benchPage(b, "/explore/pages") }
func BenchmarkPageCodewords(b *testing.B) { benchPage(b, "/explore/terms") }
func BenchmarkPageOutlets(b *testing.B) { benchPage(b, "/explore/outlets") }
func BenchmarkPageClassPatterns(b *testing.B) { benchPage(b, "/explore/classification") }
func BenchmarkPageArticles(b *testing.B) { benchPage(b, "/articles") }
func BenchmarkPageChangelog(b *testing.B) { benchPage(b, "/changelog") }
func BenchmarkPageAbout(b *testing.B) { benchPage(b, "/about") }
func BenchmarkPageAPIDocs(b *testing.B) { benchPage(b, "/api/docs") }
func BenchmarkPageExploreIndex(b *testing.B) { benchPage(b, "/explore") }
func BenchmarkPageContactSheet(b *testing.B) {
h := benchCorpus(b)
id := ""
for i := range corpus().archive.Docs {
if len(corpus().archive.Docs[i].Images) > 40 {
id = corpus().archive.Docs[i].ID
break
}
}
if id == "" {
b.Skip("no document with many pages")
}
benchPageWith(b, h, "/doc/"+id+"/pages")
}
// Pathological documents: the biggest thing in the corpus is what breaks
// first. What IS the biggest thing?
func TestProbeExtremes(t *testing.T) {
if os.Getenv("SNOWCHIVE_BENCH_ROOT") == "" {
t.Skip("needs a real corpus")
}
benchCorpus(t)
var maxImgs, maxText, maxPubs, maxVersions *Document
for i := range corpus().archive.Docs {
d := &corpus().archive.Docs[i]
if maxImgs == nil || len(d.Images) > len(maxImgs.Images) {
maxImgs = d
}
if maxText == nil || len(d.FullText) > len(maxText.FullText) {
maxText = d
}
if maxPubs == nil || len(d.Publication) > len(maxPubs.Publication) {
maxPubs = d
}
if maxVersions == nil || len(d.Versions) > len(maxVersions.Versions) {
maxVersions = d
}
}
h := benchCorpus(t)
timeIt := func(p string) time.Duration {
start := time.Now()
r := httptest.NewRequest("GET", p, nil)
w := httptest.NewRecorder()
h.ServeHTTP(w, r)
return time.Since(start)
}
fmt.Printf("most pages: %d %s doc %s, contact sheet %s\n", len(maxImgs.Images), trunc30(maxImgs.ID),
timeIt("/doc/"+maxImgs.ID).Round(time.Millisecond), timeIt("/doc/"+maxImgs.ID+"/pages").Round(time.Millisecond))
fmt.Printf("most text: %d KB %s doc page %s\n", len(maxText.FullText)/1024, trunc30(maxText.ID),
timeIt("/doc/"+maxText.ID).Round(time.Millisecond))
fmt.Printf("most pubs: %d %s\n", len(maxPubs.Publication), trunc30(maxPubs.ID))
fmt.Printf("most copies: %d %s\n", len(maxVersions.Versions), trunc30(maxVersions.ID))
}
// Distribution, not the median: which document pages are slow, and why?
func TestProbeDocPageDistribution(t *testing.T) {
if os.Getenv("SNOWCHIVE_BENCH_ROOT") == "" {
t.Skip("needs a real corpus")
}
h := benchCorpus(t)
totalImgs := 0
for i := range corpus().archive.Docs {
totalImgs += len(corpus().archive.Docs[i].Images)
}
var ds []time.Duration
for i := 0; i < len(corpus().archive.Docs); i += 7 { // a sample, every 7th
d := &corpus().archive.Docs[i]
start := time.Now()
r := httptest.NewRequest("GET", "/doc/"+d.ID, nil)
w := httptest.NewRecorder()
h.ServeHTTP(w, r)
ds = append(ds, time.Since(start))
}
sort.Slice(ds, func(i, j int) bool { return ds[i] < ds[j] })
p := func(q float64) time.Duration { return ds[int(float64(len(ds)-1)*q)] }
fmt.Printf("%d page images in the corpus (%d docs sampled)\n", totalImgs, len(ds))
fmt.Printf("doc page: p50 %s p90 %s p99 %s max %s\n",
p(0.5).Round(time.Microsecond), p(0.9).Round(time.Microsecond),
p(0.99).Round(time.Millisecond), ds[len(ds)-1].Round(time.Millisecond))
}
func TestProbeWorstDocSplit(t *testing.T) {
if os.Getenv("SNOWCHIVE_BENCH_ROOT") == "" {
t.Skip("needs a real corpus")
}
h := benchCorpus(t)
var worst *Document
for i := range corpus().archive.Docs {
if worst == nil || len(corpus().archive.Docs[i].Images) > len(worst.Images) {
worst = &corpus().archive.Docs[i]
}
}
run := func(path string, cookies ...*http.Cookie) (time.Duration, int) {
var best time.Duration
var size int
for i := 0; i < 5; i++ {
r := httptest.NewRequest("GET", path, nil)
for _, c := range cookies {
r.AddCookie(c)
}
w := httptest.NewRecorder()
start := time.Now()
h.ServeHTTP(w, r)
d := time.Since(start)
if i == 0 || d < best {
best = d
}
size = w.Body.Len()
}
return best, size
}
full, fullSize := run("/doc/" + worst.ID)
textOnly, textSize := run("/doc/"+worst.ID, &http.Cookie{Name: "pref_text_only", Value: "1"})
sheet, sheetSize := run("/doc/" + worst.ID + "/pages")
fmt.Printf("worst doc: %d images, %d KB transcript\n", len(worst.Images), len(worst.FullText)/1024)
fmt.Printf(" full page %6s %d KB\n", full.Round(time.Microsecond), fullSize/1024)
fmt.Printf(" text-only %6s %d KB (no image grid, no lightbox)\n", textOnly.Round(time.Microsecond), textSize/1024)
fmt.Printf(" contact sheet %6s %d KB\n", sheet.Round(time.Microsecond), sheetSize/1024)
}
func trunc30(s string) string {
if len(s) > 30 {
return s[:30]
}
return s
}
// TestSoak hammers the server for a while and watches its memory. The claim
// it checks: nothing accumulates between requests - no cache, no growing
// map, no goroutine left behind. Run it against a real corpus:
//
// SNOWCHIVE_BENCH_ROOT=/path/to/archive tests/run.sh -run TestSoak -timeout 30m
//
// SNOWCHIVE_SOAK_REQUESTS sets the count (default 20,000).
func TestSoak(t *testing.T) {
h := benchCorpus(t)
n := 20000
if v := os.Getenv("SNOWCHIVE_SOAK_REQUESTS"); v != "" {
if parsed, err := strconv.Atoi(v); err == nil && parsed > 0 {
n = parsed
}
}
paths := []string{"/", "/browse", "/browse?collection=foia", "/browse/program",
"/browse/tag?sort=count", "/browse/agency/NSA", "/search?q=nsa&do_search=1",
"/search?q=%22upstream+collection%22&do_search=1", "/explore/ids", "/glossary",
"/articles", "/api/facets", "/api/search?q=prism", "/sitemap.xml"}
cookies := []*http.Cookie{
{Name: "pref_theme", Value: "dark"},
{Name: "pref_per_page", Value: "100"},
{Name: "pref_text_only", Value: "1"},
}
read := func() (heap, sys uint64, goroutines int) {
runtime.GC()
var m runtime.MemStats
runtime.ReadMemStats(&m)
return m.HeapAlloc, m.Sys, runtime.NumGoroutine()
}
// Warm up first: the pre-rendered bodies and the first template
// executions allocate once and stay.
for i := 0; i < 200; i++ {
r := httptest.NewRequest("GET", paths[i%len(paths)], nil)
h.ServeHTTP(httptest.NewRecorder(), r)
}
heap0, sys0, gor0 := read()
start := time.Now()
var wg sync.WaitGroup
const workers = 8
for w := 0; w < workers; w++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
for i := id; i < n; i += workers {
r := httptest.NewRequest("GET", paths[i%len(paths)], nil)
if i%4 == 0 {
r.AddCookie(cookies[i%len(cookies)])
}
rec := httptest.NewRecorder()
h.ServeHTTP(rec, r)
if rec.Code != 200 {
t.Errorf("%s: %d", paths[i%len(paths)], rec.Code)
return
}
}
}(w)
}
wg.Wait()
elapsed := time.Since(start)
heap1, sys1, gor1 := read()
mb := func(b uint64) float64 { return float64(b) / (1 << 20) }
fmt.Printf("%d requests in %s (%.0f req/s, %d workers)\n",
n, elapsed.Round(time.Millisecond), float64(n)/elapsed.Seconds(), workers)
fmt.Printf("heap %.1f MB -> %.1f MB (%+.1f)\n", mb(heap0), mb(heap1), mb(heap1)-mb(heap0))
fmt.Printf("sys %.1f MB -> %.1f MB (%+.1f)\n", mb(sys0), mb(sys1), mb(sys1)-mb(sys0))
fmt.Printf("goroutines %d -> %d\n", gor0, gor1)
if gor1 > gor0+2 {
t.Errorf("goroutines grew from %d to %d", gor0, gor1)
}
// The heap after a GC should be the corpus and the pre-rendered pages,
// not a function of how many requests have been served.
if grown := mb(heap1) - mb(heap0); grown > 32 {
t.Errorf("heap grew by %.1f MB over %d requests - something is accumulating", grown, n)
}
}

78
tests/bugs_test.go Normal file
View file

@ -0,0 +1,78 @@
package main
import (
"encoding/json"
"fmt"
"net/url"
"strings"
"testing"
)
// A hyphenated term must match its words in sequence, not just the first one.
func TestHyphenatedTermMatchesInSequence(t *testing.T) {
hit := Document{ID: "hit", FullText: "A cyber attack on the power grid."}
miss := Document{ID: "miss", FullText: "Cyber policy, and an attack helicopter."}
for _, d := range []*Document{&hit, &miss} {
setBodySearchText(d)
d.StemmedText, d.SearchText = d.StemmedFull, d.FoldedFull
}
for _, field := range []string{"FT", "TE"} {
if !compileQuery([]string{"cyber-attack"}, []string{field}, []string{"and"}).matches(&hit) {
t.Errorf("%s: cyber-attack should match text containing the words in sequence", field)
}
if compileQuery([]string{"cyber-attack"}, []string{field}, []string{"and"}).matches(&miss) {
t.Errorf("%s: cyber-attack matched a document where the words are apart", field)
}
}
// The words on their own still match both, as before.
if !compileQuery([]string{"cyber"}, []string{"FT"}, []string{"and"}).matches(&miss) {
t.Error("a plain word should still match")
}
}
// A value page's publication-year bars count every year a document was
// published in, because each bar links to a search that matches any of them.
func TestPubYearsOf(t *testing.T) {
var d Document
if err := json.Unmarshal([]byte(`{"publication": [
{"publisher": "The Guardian", "date_published": "2013-06-06"},
{"publisher": "Le Monde", "date_published": "2014-10-21"},
{"publisher": "Der Spiegel", "date_published": "2013-12-30"},
{"publisher": "No date"},
{"publisher": "Bad date", "date_published": "201"}
]}`), &d); err != nil {
t.Fatal(err)
}
if got := fmt.Sprint(pubYearsOf(&d)); got != "[2013 2014]" {
t.Errorf("pubYearsOf = %s, want [2013 2014]", got)
}
if got := fmt.Sprint(pubYearsOf(&Document{})); got != "[]" {
t.Errorf("pubYearsOf(no publications) = %s, want []", got)
}
}
// A saved search's label must name the type filter, or two saved searches
// differing only by Reported/SIDtoday are indistinguishable.
func TestSearchLabelIncludesTypeFilter(t *testing.T) {
entry := func(v url.Values) string { return url.QueryEscape(v.Encode()) }
base := url.Values{"fqv": {"xkeyscore"}, "fqf": {"TE"}, "fqc": {"and"}}
typed := url.Values{"fqv": {"xkeyscore"}, "fqf": {"TE"}, "fqc": {"and"}, "type": {"REPORTED"}}
both := url.Values{"fqv": {"xkeyscore"}, "fqf": {"TE"}, "fqc": {"and"},
"type": {"SIDTODAY"}, "coll": {"news"}}
got := corpus().parseSearchEntries([]string{entry(typed), entry(base), entry(both)})
if len(got) != 3 {
t.Fatalf("parsed %d entries, want 3", len(got))
}
if !strings.Contains(got[0].Label, "REPORTED") {
t.Errorf("label %q should name the type filter", got[0].Label)
}
if got[0].Label == got[1].Label {
t.Errorf("searches differing only by type share the label %q", got[0].Label)
}
// Collection and type share one bracket rather than two.
if !strings.Contains(got[2].Label, "(news, SIDTODAY)") {
t.Errorf("label %q should list collection and type together", got[2].Label)
}
}

538
tests/consistency_test.go Normal file
View file

@ -0,0 +1,538 @@
package main
import (
"encoding/json"
"html"
"net/http"
"net/http/httptest"
"net/url"
"path/filepath"
"reflect"
"regexp"
"slices"
"sort"
"strings"
"testing"
)
// A corpus built to exercise the places where one fact used to be computed
// twice: search rows and their negations, the page and the API, counts and
// the pages they link to. Each document's words are chosen so the expected
// result sets can be written down by hand.
//
// n1 news alpha gamma ZEBRA 2010 The Intercept 2014 + the intercept 2016
// n2 news alpha ZEBRA 2011 The Intercept 2014 + Der Spiegel 2014
// n3 news beta gamma 2012 Der Spiegel 2015
// n4 news beta 2012 Der Spiegel 2015
// f1 foia alpha ZEBRA 2010 (court filing: "(c)" list labels)
// s1 sidtoday ZEBRA 2011
// a1 news "aka_id": null
// d1 news date_created "ODNI", date_published "15 May 2015"
func writeConsistencyCorpus(t *testing.T) string {
t.Helper()
root := t.TempDir()
doc := func(dir, id string, meta map[string]interface{}, text string) {
b, err := json.Marshal(meta)
if err != nil {
t.Fatal(err)
}
d := filepath.Join(root, dir, id)
writeFixtureFile(t, filepath.Join(d, "metadata.json"), string(b))
writeFixtureFile(t, filepath.Join(d, id+".pdf"), "%PDF-1.4\n")
if text != "" {
writeFixtureFile(t, filepath.Join(d, id+".md"), text)
}
}
pub := func(publisher, date, link string, reporters ...string) map[string]interface{} {
return map[string]interface{}{"publisher": publisher, "date_published": date,
"link": link, "reporters": reporters}
}
meta := func(title, typ, created string, programs []string, pubs ...map[string]interface{}) map[string]interface{} {
return map[string]interface{}{"title": title, "type": typ,
"document": map[string]interface{}{"creator": "NSA", "date_created": created, "programs": programs},
"publication": pubs}
}
zebra := []string{"ZEBRA"}
doc("", "n1", meta("Report one", "REPORTED", "2010-03-01", zebra,
pub("The Intercept", "2014-01-01", "https://example.org/a", "Glenn Greenwald"),
pub("the intercept", "2016-01-01", "https://example.org/b", "glenn greenwald")),
"alpha gamma\n\n(S//SI) a secret line\n(S/SI) the same marking, spelled by OCR\n(U) plain\n")
doc("", "n2", meta("Report two", "REPORTED", "2011-03-01", zebra,
pub("The Intercept", "2014-02-01", "https://example.org/c"),
pub("Der Spiegel", "2014-02-02", "https://example.org/f")),
"alpha\n\n(TS//SI//NF) top line\n")
doc("", "n3", meta("Report three", "REPORTED", "2012-03-01", nil,
pub("Der Spiegel", "2015-01-01", "https://example.org/d")), "beta gamma\n")
doc("", "n4", meta("Report four", "REPORTED", "2012-04-01", nil,
pub("Der Spiegel", "2015-01-01", "https://example.org/d")), "beta\n")
doc("FOIA", "f1", meta("Court filing", "FOIA", "2010-05-01", zebra),
"alpha\n\n(c) The Court shall issue an order.\n(s) another subsection\n")
doc("SIDtoday-archive", "s1", meta("Issue", "SIDTODAY", "2011-05-01", zebra), "(S) issue line\n")
a1 := meta("Null aka", "REPORTED", "", nil)
a1["aka_id"] = nil
doc("", "a1", a1, "")
// A date of no shape the archive reads: served, undated.
doc("", "d1", meta("Odd date", "REPORTED", "ODNI", nil, pub("Der Spiegel", "15 May 2015", "https://example.org/e")), "")
writeFixtureFile(t, filepath.Join(root, "glossary.json"), `[]`)
return root
}
// matchingIDs runs the search engine and returns the matching ids, sorted.
func matchingIDs(queries, fields, conjs []string) []string {
q, f, c := normalizeClauses(queries, fields, conjs)
var ids []string
scored, _ := corpus().executeSearch(searchParams{Queries: q, Fields: f, Conjs: c})
for _, s := range scored {
ids = append(ids, s.Doc.ID)
}
sort.Strings(ids)
return ids
}
// A row is one unit: its -terms are exclusions within it, whatever joins
// it to the rows before. And an empty row starts nothing - a blank first
// row with "OR x" under it is a search for x.
func TestSearchRowsAndNegations(t *testing.T) {
withCorpus(t, writeConsistencyCorpus(t))
te2 := []string{"TE", "TE"}
for _, c := range []struct {
name string
queries, conjs []string
want []string
}{
{"inline negation", []string{"alpha -gamma"}, []string{"and"}, []string{"f1", "n2"}},
// alpha AND NOT (beta AND NOT gamma): no alpha document is lost
// for mentioning gamma.
{"negation inside NOT", []string{"alpha", "beta -gamma"}, []string{"and", "not"}, []string{"f1", "n1", "n2"}},
// alpha OR (beta AND NOT gamma): the exclusion stays inside its row.
{"negation inside OR", []string{"alpha", "beta -gamma"}, []string{"and", "or"}, []string{"f1", "n1", "n2", "n4"}},
{"blank first row, OR", []string{"", "alpha"}, []string{"and", "or"}, []string{"f1", "n1", "n2"}},
{"spaces first row, OR", []string{" ", "alpha"}, []string{"and", "or"}, []string{"f1", "n1", "n2"}},
{"blank first row, OR nothing", []string{"", "zzzzunmatched"}, []string{"and", "or"}, nil},
{"blank first row, NOT", []string{"", "gamma"}, []string{"and", "not"}, []string{"a1", "d1", "f1", "n2", "n4", "s1"}},
{"only exclusions", []string{"-gamma -alpha"}, []string{"and"}, []string{"a1", "d1", "n4", "s1"}},
} {
if got := matchingIDs(c.queries, te2[:len(c.queries)], c.conjs); !reflect.DeepEqual(got, c.want) {
t.Errorf("%s: %q %v matched %v, want %v", c.name, c.queries, c.conjs, got, c.want)
}
}
// The scorer and the highlighter read the same rows: an excluded term
// is neither rewarded nor marked.
if got := positiveTerms([]string{"alpha -gamma", "", "beta"}, []string{"and", "and", "not"}); !reflect.DeepEqual(got, []string{"alpha"}) {
t.Errorf("positiveTerms = %q, want [alpha]", got)
}
if term, _ := snippetAnchor([]string{"-gamma", "alpha -beta"}, te2, []string{"and", "and"}); term != "alpha" {
t.Errorf("snippetAnchor = %q, want alpha", term)
}
}
// "+ Add row" and "remove" re-run the search: on the same rows, meaning the
// same thing, as the Search button would.
func TestRowEditsSearchTheSameRows(t *testing.T) {
withCorpus(t, writeConsistencyCorpus(t))
rows := "do_search=1&fqv=alpha+-gamma&fqf=TE&fqc=and"
want := resultTotal(t, "/search?action=search&"+rows, foiaOn)
if want != 2 {
t.Fatalf("search for alpha -gamma found %d, want 2", want)
}
for _, action := range []string{"add", "remove-1"} {
path := "/search?action=" + action + "&" + rows + "&fqv=beta&fqf=TE&fqc=or"
if action == "add" {
path = "/search?action=add&" + rows
}
if got := resultTotal(t, path, foiaOn); got != want {
t.Errorf("action=%s: %d results, the same rows searched find %d", action, got, want)
}
}
}
// The type filter resubmits the search the page shows, not the request: a
// removed row stays removed, and the chosen sort is kept.
func TestTypeFilterFormCarriesTheEffectiveSearch(t *testing.T) {
withCorpus(t, writeConsistencyCorpus(t))
body := get(t, "/search?do_search=1&action=remove-1&sort=title"+
"&fqv=alpha&fqf=TE&fqc=and&fqv=removedrow&fqf=TE&fqc=or", foiaOn).Body.String()
if !strings.Contains(body, `class="type-filter"`) {
t.Fatal("no type filter on a result set holding two types")
}
if strings.Contains(body, "removedrow") {
t.Error("the removed row is still in the page - the type filter would bring it back")
}
if !strings.Contains(body, `name="sort" value="title"`) {
t.Error("the type filter drops the chosen sort")
}
}
// /api/search runs the search page's pipeline, type filter included. The
// API has no preference: it matches the page of a reader who includes FOIA
// releases.
func TestAPISearchMatchesThePage(t *testing.T) {
withCorpus(t, writeConsistencyCorpus(t))
for qs, want := range map[string]int{
"q=alpha": 3,
"q=alpha&type=REPORTED": 2,
"q=alpha&type=FOIA&type=REPORTED": 3,
"q=alpha&type=reported": 2,
"q=zebra&coll=news": 2,
"q=zebra&coll=foia&type=FOIA": 1,
"do_search=1&fqv=alpha&fqf=TE&fqc=and&fqv=beta+-gamma&fqf=TE&fqc=or": 4,
} {
page := resultTotal(t, "/search?"+qs, foiaOn)
var api struct{ Total int }
if err := json.Unmarshal(get(t, "/api/search?"+qs).Body.Bytes(), &api); err != nil {
t.Fatalf("%s: %v", qs, err)
}
if page != want || api.Total != want {
t.Errorf("%s: the page finds %d, the API %d, want %d", qs, page, api.Total, want)
}
}
}
// Every count in this corpus is the number its page or search returns.
func TestConsistencyCorpusCounts(t *testing.T) {
withCorpus(t, writeConsistencyCorpus(t))
// Facets, with journalist and publisher spellings that differ by case
// across one document's publications.
for ci := range categories {
c := &categories[ci]
for val, want := range c.Facet(corpus()) {
if got := len(corpus().categoryMatches(c.Slug, val)); got != want {
t.Errorf("%s=%q: facet says %d, the page lists %d", c.Slug, val, want, got)
}
}
}
if js := corpus().archive.Facets["journalist"]; len(js) != 1 || js["Glenn Greenwald"] != 1 {
t.Errorf("journalists = %v, want Glenn Greenwald once", js)
}
if n := corpus().archive.Facets["publisher"]["The Intercept"]; n != 2 {
t.Errorf("The Intercept: %d documents, want 2 (%v)", n, corpus().archive.Facets["publisher"])
}
// The value-page timelines and refine links, scoped to a collection:
// f1 (FOIA) shares 2010 with n1 and s1 (SIDtoday) shares 2011 with n2,
// so an unscoped bar link finds more than its bar counts.
for _, p := range []string{"/browse/program/ZEBRA?coll=news", "/browse/program/ZEBRA"} {
if n := followLinks(t, p, reBarLink, "timeline bar"); n < 2 {
t.Errorf("%s: %d timeline bars checked, want the doc and pub timelines", p, n)
}
}
// /explore/outlets: rows, cells and the per-year footer.
for coll, table := range corpus().outletTables {
for _, row := range table.Rows {
if got := resultTotal(t, row.URL); got != row.Total {
t.Errorf("outlets[%q] %s: total %d, its page lists %d", coll, row.Name, row.Total, got)
}
for _, cell := range row.Cells {
if cell.URL != "" && resultTotal(t, cell.URL) != cell.Docs {
t.Errorf("outlets[%q] %s: a cell says %d, its search finds %d", coll, row.Name, cell.Docs, resultTotal(t, cell.URL))
}
}
}
// The footer adds up its column: a document several outlets
// published counts once for each of them.
for i, y := range table.Years {
sum := 0
for _, row := range table.Rows {
sum += row.Cells[i].Docs
}
if table.Totals[i] != sum {
t.Errorf("outlets[%q] %s: footer %d, its column adds up to %d", coll, y, table.Totals[i], sum)
}
}
}
cell := func(outlet, year string) (docs, footer int) {
table := corpus().outletTables[""]
for i, y := range table.Years {
if y != year {
continue
}
footer = table.Totals[i]
for _, row := range table.Rows {
if row.Name == outlet {
docs = row.Cells[i].Docs
}
}
}
return docs, footer
}
// n2: The Intercept and Der Spiegel both published it in 2014, so it
// counts for both outlets and twice in the year's footer (n1 is the
// third: The Intercept, 2014).
if spiegel, footer := cell("Der Spiegel", "2014"); spiegel != 1 || footer != 3 {
t.Errorf("2014: Der Spiegel cell %d, footer %d - want 1 and 3 (n2 once per outlet)", spiegel, footer)
}
if intercept, _ := cell("The Intercept", "2014"); intercept != 2 {
t.Errorf("2014: The Intercept cell %d, want 2", intercept)
}
intercept := 0
for _, row := range corpus().outletTables[""].Rows {
if row.Name == "The Intercept" {
intercept = row.Total
}
}
if intercept != 2 {
t.Errorf("The Intercept's outlet total = %d, want 2 documents (n1 counts once for its two years)", intercept)
}
// /explore/portions: lower-case list labels are not markings, and every
// row's document count is what its search finds.
for _, l := range corpus().portionStats.Levels {
if l.Name == "C" {
t.Errorf("a CONFIDENTIAL row from the court filing's (c) labels: %+v", l)
}
}
rows := append(append(append([]portionRow(nil), corpus().portionStats.Levels...), corpus().portionStats.Markings...), corpus().portionStats.Controls...)
if len(rows) < 6 {
t.Fatalf("only %d portion rows", len(rows))
}
for _, r := range rows {
if got := resultTotal(t, string(r.SearchURL)); got != r.Docs {
t.Errorf("portion row %s: %d documents, its search finds %d (%s)", r.Name, r.Docs, got, r.SearchURL)
}
}
if got := corpus().docIndex["n1"].PortionMarkings; !reflect.DeepEqual(got, []string{"S//SI", "U"}) {
t.Errorf("n1 markings = %v, want [S//SI U]", got)
}
// A malformed date is no date: no "ODNI" year that no year page opens.
if d := corpus().docIndex["d1"]; d == nil || d.DocMeta.DateCreated != "" || d.Publication[0].DatePublished != "" {
t.Errorf("d1's malformed dates were kept: %+v", d)
}
if _, ok := corpus().archive.Facets["docdate"]["ODNI"]; ok {
t.Error(`"ODNI" counted as a document year`)
}
// aka_id: null is no akas, not a reason to drop the document.
if corpus().docIndex["a1"] == nil {
t.Error(`a document with "aka_id": null was not loaded`)
}
}
// A corpus for the counts that turn on WHICH publication entry carries a
// date or a byline, and on the reader's collection scope. Every document
// says "alpha".
//
// p1 news 2010-03 Outlet X 2013 (Able), Outlet Y 2014 (Baker)
// p2 news 2010-06 Outlet X 2014 (Able) people: Clerk
// p3 news 2011-01 Outlet X 2014 (Baker)
// p4 news 2012-03 Outlet Y 2014
// p5 news 2012-09 Outlet Y 2014
// p6 news 2013-02 Outlet Y 2014, Partner Z 2014 (its own story; reporting only)
// f1 foia 2009-07 Outlet Y 2014 (a story of its own) people: Docket
func writeEntryCorpus(t *testing.T) string {
t.Helper()
root := t.TempDir()
pub := func(publisher, date string, reporters ...string) map[string]interface{} {
return map[string]interface{}{"publisher": publisher, "date_published": date, "reporters": reporters}
}
doc := func(dir, id, typ, created string, people []string, pubs ...map[string]interface{}) {
b, err := json.Marshal(map[string]interface{}{"title": "Document " + id, "type": typ,
"document": map[string]interface{}{"creator": "NSA", "date_created": created, "people": people},
"publication": pubs})
if err != nil {
t.Fatal(err)
}
d := filepath.Join(root, dir, id)
writeFixtureFile(t, filepath.Join(d, "metadata.json"), string(b))
writeFixtureFile(t, filepath.Join(d, id+".pdf"), "%PDF-1.4\n")
writeFixtureFile(t, filepath.Join(d, id+".md"), "alpha\n")
}
doc("", "p1", "REPORTED", "2010-03-01", nil,
pub("Outlet X", "2013-05-01", "Able, Ann"), pub("Outlet Y", "2014-02-01", "Baker, Bob"))
doc("", "p2", "REPORTED", "2010-06-01", []string{"Clerk, Carol"}, pub("Outlet X", "2014-03-01", "Able, Ann"))
doc("", "p3", "REPORTED", "2011-01-01", nil, pub("Outlet X", "2014-04-01", "Baker, Bob"))
doc("", "p4", "REPORTED", "2012-03-01", nil, pub("Outlet Y", "2014-05-01"))
doc("", "p5", "REPORTED", "2012-09-01", nil, pub("Outlet Y", "2014-06-01"))
story := func(p map[string]interface{}, title, link string) map[string]interface{} {
p["title"], p["link"] = title, link
return p
}
partner := story(pub("Partner Z", "2014-07-02"), "Partner story", "https://example.org/partner")
partner["reporting_only"] = true
doc("", "p6", "REPORTED", "2013-02-01", nil, pub("Outlet Y", "2014-07-01"), partner)
doc("FOIA", "f1", "FOIA", "2009-07-01", []string{"Docket, Dan"},
story(pub("Outlet Y", "2014-08-01"), "FOIA story", "https://example.org/foia-story"))
writeFixtureFile(t, filepath.Join(root, "glossary.json"), `[]`)
return root
}
// Counts that turn on a publication entry's date or byline, or on the
// reader's scope, are still the numbers their links return.
func TestCountsTiedToEntriesAndScope(t *testing.T) {
withCorpus(t, writeEntryCorpus(t))
// A publication-year page files a document under its first publication,
// which no search asks for: p1, first published in 2013, would join the
// 2010 bar's search for 2014. So those bars carry no link. The
// document-year page's publication bars do, and they are exact.
body := get(t, "/browse/published/2014").Body.String()
if !strings.Contains(body, "By document date") || strings.Contains(body, `<a class="hist-col"`) {
t.Error("/browse/published/2014: want the document-date bars, and no link on them")
}
if n := followLinks(t, "/browse/docdate/2010", reBarLink, "publication bar"); n != 2 {
t.Errorf("/browse/docdate/2010: %d publication bars checked, want 2013 and 2014", n)
}
// Under a publication year a journalist counts for the documents that
// byline them on an entry of that year, because that is what the refine
// link's search matches: Able's byline on p1 is Outlet X's of 2013, not
// Outlet Y's of 2014.
for _, p := range []string{
"/browse/publisher/Outlet%20Y?year=2014",
"/search?q=alpha+published%3A2014",
// An OR row unties the dates, and the bylines with them.
"/search?do_search=1&fqv=alpha&fqf=TE&fqc=and&fqv=beta&fqf=TE&fqc=or&date_pub_start=2014&date_pub_end=2014",
} {
if n := followLinks(t, p, reFacetLink, "refine link"); n == 0 {
t.Errorf("%s: no refine links to check", p)
}
}
// A year bar narrows the search it stands under: in a range that starts
// in June 2012, the 2012 bar counts p5 and not p4 of March.
if n := followLinks(t, "/search?q=alpha+date%3A2012-06..2013-06", reBarLink, "year bar"); n != 2 {
t.Errorf("%d year bars checked, want 2012 and 2013", n)
}
// Docket is named only in the FOIA release and 2009 is only its year:
// neither is on the default reader's lists, nor in the boxes' counts.
for _, reader := range [][]*http.Cookie{nil, {foiaOn}} {
if n := checkCategoryBoxes(t, reader...); n < 10 {
t.Errorf("%d category boxes checked", n)
}
}
// The home page's publication chart counts first publications, as the
// year pages its bars link file them: p1 counts for 2013 and not 2014.
body = get(t, "/").Body.String()
for year, want := range map[string]string{"2013": "1", "2014": "5"} {
if !strings.Contains(body, `href="/browse/published/`+year+`?coll=news" title="`+want+` documents first published in `+year+`"`) {
t.Errorf("home: no bar of %s documents first published in %s linking to that year's page", want, year)
}
}
// In a search result a publication year links to its year page only
// when that page lists the document: p1 is filed under 2013, its first.
body = get(t, "/search?q=title%3Ap1").Body.String()
if !strings.Contains(body, `href="/browse/published/2013"`) || strings.Contains(body, `href="/browse/published/2014"`) {
t.Error("p1's result should link 2013, its first publication year, and not 2014")
}
// A partner that only reported has no publisher page for its story's
// byline to link.
body = get(t, "/article/"+articleKeyFor("https://example.org/partner")).Body.String()
if !strings.Contains(body, "Partner Z") || strings.Contains(body, "/browse/publisher/Partner") {
t.Error("the partner's article should name Partner Z without linking a publisher page")
}
// The outlets table of one collection counts the stories that have a
// document in it: Outlet Y's 2014 story published only the FOIA release.
stories := func(coll string) int {
table := corpus().outletTables[coll]
for _, row := range table.Rows {
if row.Name == "Outlet Y" {
return row.Cells[slices.Index(table.Years, "2014")].Articles
}
}
return -1
}
if all, news := stories(""), stories(collectionNews); all != 1 || news != 0 {
t.Errorf("Outlet Y's 2014 stories: %d over every collection and %d reported, want 1 and 0", all, news)
}
}
var reCategoryBox = regexp.MustCompile(`<a href="/browse/([a-z]+)" class="cat-box[^"]*"><strong>[^<]*</strong><span class="count">(\d+)</span></a>`)
// checkCategoryBoxes holds the number on each category box in the browse
// nav to the rows of the list its click opens, for the reader with these
// cookies.
func checkCategoryBoxes(t *testing.T, cookies ...*http.Cookie) int {
t.Helper()
boxes := reCategoryBox.FindAllStringSubmatch(get(t, "/browse", cookies...).Body.String(), -1)
for _, m := range boxes {
slug, shown := m[1], mustAtoi(t, m[2])
list := get(t, "/browse/"+slug, cookies...).Body.String()
if rows := strings.Count(list, `href="/browse/`+slug+`/`); rows != shown {
t.Errorf("the %s box says %d, its list has %d rows (%d cookies)", slug, shown, rows, len(cookies))
}
}
return len(boxes)
}
// -check reads the load counters, so the glossary's problems must still be
// in them when it does.
func TestGlossaryWarningsReachCheck(t *testing.T) {
root := writeConsistencyCorpus(t)
withCorpus(t, root)
clean := loadWarnings
writeFixtureFile(t, filepath.Join(root, "glossary.json"), `{not json`)
loadData()
if loadWarnings != clean+1 {
t.Errorf("malformed glossary.json: %d warnings, want %d", loadWarnings, clean+1)
}
}
// A 404 under /files must not carry the files' public cache policy.
func TestMissingFileIsNotCached(t *testing.T) {
w := get(t, "/files/prism-slides/no-such-file.pdf")
if w.Code != http.StatusNotFound {
t.Fatalf("status %d, want 404", w.Code)
}
if cc := w.Header().Get("Cache-Control"); strings.Contains(cc, "public") {
t.Errorf("404 carries Cache-Control %q", cc)
}
}
// -dev serves the on-disk assets so edits show without a restart: no
// startup ETag may pin the old content.
func TestDevModeStaticRevalidatesEdits(t *testing.T) {
buildFixture(t)
devMode = true
initAssets()
t.Cleanup(func() {
devMode = false
initAssets()
})
if staticHashes != nil {
t.Errorf("dev mode built %d static hashes", len(staticHashes))
}
w := httptest.NewRecorder()
handleStatic(w, httptest.NewRequest("GET", "/static/style.css", nil))
if w.Code != http.StatusOK || w.Header().Get("ETag") != "" || w.Header().Get("Last-Modified") == "" {
t.Errorf("dev static: status %d, ETag %q, Last-Modified %q - want 200, none, the file's",
w.Code, w.Header().Get("ETag"), w.Header().Get("Last-Modified"))
}
}
// What the search page offers to pin is what the pinned page can list.
func TestPinnableSearchesAreListed(t *testing.T) {
h, _ := buildFixture(t)
if label, ok := corpus().searchLabel(mustQuery(t, "/search?do_search=1&fqv=&fqf=TE&fqc=and&date_doc_start=2013&date_doc_end=2013")); !ok || label != "all documents · (doc 2013)" {
t.Errorf("date-only search label = %q, %v", label, ok)
}
if strings.Contains(get(t, "/search?do_search=1&fqv=&fqf=TE&fqc=and").Body.String(), "pin-search-btn") {
t.Error("a search with no terms and no filters offers a pin")
}
if !strings.Contains(get(t, "/search?q=date:2013").Body.String(), "pin-search-btn") {
t.Error("a date-only search offers no pin")
}
pin := func(qs string) *httptest.ResponseRecorder {
r := httptest.NewRequest("POST", "/pinsearch", strings.NewReader(url.Values{"qs": {qs}, "back": {"/pinned"}}.Encode()))
r.Header.Set("Content-Type", "application/x-www-form-urlencoded")
return record(h, r)
}
if w := pin("do_search=1&fqc=and&fqf=TE&fqv="); w.Code != http.StatusBadRequest {
t.Errorf("pinning an empty search: status %d, want 400", w.Code)
}
w := pin("date_doc_end=2013&date_doc_start=2013&do_search=1&fqc=and&fqf=TE&fqv=")
if w.Code != http.StatusSeeOther {
t.Fatalf("pinning a date-only search: status %d", w.Code)
}
body := get(t, "/pinned", w.Result().Cookies()...).Body.String()
if !strings.Contains(html.UnescapeString(body), "all documents · (doc 2013)") {
t.Error("the pinned date-only search is not listed on /pinned")
}
}

199
tests/corpus_test.go Normal file
View file

@ -0,0 +1,199 @@
package main
import (
"html"
"net/http"
"os"
"reflect"
"regexp"
"slices"
"strings"
"testing"
)
// Checks against a REAL corpus. Point $SNOWCHIVE_BENCH_DATA at one, as for
// the page benchmarks, and run: go test -run TestCorpus -v
//
// The fixtures are a handful of documents, each built to show one thing. A
// count and its link are computed by different code, and whether the two
// agree turns on what the documents happen to hold: a second publication in
// a later year, a byline on only one of them, a value found only in FOIA
// releases. The disagreements these tests walk the archive for were all
// found on the full corpus and on no fixture.
func TestCorpus(t *testing.T) {
root := os.Getenv("SNOWCHIVE_BENCH_DATA")
if root == "" {
t.Skip("set SNOWCHIVE_BENCH_DATA to a corpus directory to check a real corpus")
}
withCorpus(t, root)
t.Run("counts", corpusCounts)
t.Run("FOIA links", corpusFOIALinks)
t.Run("inflections", corpusInflections)
t.Run("snippets", corpusSnippets)
t.Run("index", corpusIndex)
}
// corpusCounts follows the counted links of the loaded corpus, as a reader
// who leaves FOIA releases out and as one who includes them.
func corpusCounts(t *testing.T) {
// The pages whose bars and refine links are followed: the year pages of
// both date categories, the three commonest values of every other
// category, each cell of the outlets table (a publisher's page for one
// publication year) and searches under each kind of date range.
var pages []string
for ci := range categories {
c := &categories[ci]
n := 3
if c.isDate() {
n = -1
}
for _, v := range rankCounts(c.Facet(corpus()), n) {
pages = append(pages, categoryValueURL(c.Slug, v))
}
}
for _, row := range corpus().outletTables[""].Rows {
for _, cell := range row.Cells {
if cell.URL != "" {
pages = append(pages, cell.URL)
}
}
}
pages = append(pages,
"/search?q=nsa",
"/search?q=nsa+date%3A2012-06..2013-06",
"/search?q=nsa+published%3A2016",
"/search?q=published%3A2014",
"/search?do_search=1&fqv=nsa&fqf=TE&fqc=and&date_pub_start=2015-01-01&date_pub_end=2016-06-30",
"/search?do_search=1&fqv=nsa&fqf=TE&fqc=and&fqv=gchq&fqf=TE&fqc=or&date_pub_start=2016&date_pub_end=2016",
)
for name, reader := range map[string][]*http.Cookie{"FOIA left out": nil, "FOIA included": {foiaOn}} {
t.Run(name, func(t *testing.T) {
bars, refines := 0, 0
for _, p := range pages {
bars += followLinks(t, p, reBarLink, "bar", reader...)
refines += followLinks(t, p, reFacetLink, "refine link", reader...)
}
if bars == 0 || refines == 0 {
t.Errorf("nothing exercised: %d bars, %d refine links", bars, refines)
}
boxes := checkCategoryBoxes(t, reader...)
t.Logf("%d pages: %d bars, %d refine links and %d category boxes checked", len(pages), bars, refines, boxes)
})
}
}
// corpusFOIALinks opens the page of every tenth FOIA release as a reader
// who leaves FOIA out, and follows each category value it links: the page
// behind the link has to list the release.
func corpusFOIALinks(t *testing.T) {
reValueLink := regexp.MustCompile(`href="(/browse/[a-z]+/[^"]+)"`)
links := 0
for i, d := range corpus().collectionDocs[collectionFOIA] {
if i%10 != 0 {
continue
}
// The release's own metadata: the cards under it are other documents.
page, _, _ := strings.Cut(get(t, docPath(d.ID)).Body.String(), `class="related-section"`)
for _, m := range reValueLink.FindAllStringSubmatch(page, -1) {
link := html.UnescapeString(m[1])
links++
if !strings.Contains(get(t, withQuery(link, "view=table")).Body.String(), `href="`+docPath(d.ID)+`"`) {
t.Errorf("%s links %s, which does not list it", d.ID, link)
}
}
}
if links == 0 {
t.Error("no FOIA release with a value link to follow")
}
t.Logf("%d value links followed", links)
}
// A word in -ies or -ied stems as the word in -y it is a form of, whatever
// rules that word goes through - checked over every such pair the corpus
// holds, since a suffix rule that reads one of the two forms and not the
// other splits them silently ("capability" from "capabilities").
func corpusInflections(t *testing.T) {
vocab := map[string]bool{}
for i := range corpus().archive.Docs {
for _, w := range splitWords(corpus().archive.Docs[i].SearchText) {
vocab[w] = true
}
}
pairs := 0
for w := range vocab {
if len(w) < 5 || !(strings.HasSuffix(w, "ies") || strings.HasSuffix(w, "ied")) {
continue
}
base := w[:len(w)-3] + "y"
if !vocab[base] {
continue
}
pairs++
if stem(w) != stem(base) {
t.Errorf("stem(%q) = %q, stem(%q) = %q; want equal", w, stem(w), base, stem(base))
}
}
if pairs == 0 {
t.Error("the corpus holds no -ies/-ied word next to its -y form")
}
t.Logf("%d pairs checked in a vocabulary of %d words", pairs, len(vocab))
}
// corpusSnippets holds every document's snippets off the fold kept at load
// to the ones folding its transcript again gives - the versions' lines sit
// behind that fold in the same string - and a page of "all" cards to its
// bound.
func corpusSnippets(t *testing.T) {
for _, q := range []string{"nsa", "cable tapping", `"upstream collection"`, "tap*", "the"} {
for i := range corpus().archive.Docs {
d := &corpus().archive.Docs[i]
if got, want := d.bodySnippets(q, 3), getSnippets(d.FullText, q, 3); !reflect.DeepEqual(got, want) {
t.Fatalf("%s, %q: %q from the loaded fold, %q from the text", d.ID, q, got, want)
}
}
}
all := &http.Cookie{Name: "pref_per_page", Value: "all"}
for _, c := range categories {
for _, v := range rankCounts(c.Facet(corpus()), 1) {
if c.Facet(corpus())[v] <= maxCardsPerPage {
continue
}
body := get(t, categoryValueURL(c.Slug, v), all, foiaOn).Body.String()
if n := strings.Count(body, `class="doc-card"`); n != maxCardsPerPage {
t.Errorf("%s=%s: %d cards on a page of all, want %d", c.Slug, v, n, maxCardsPerPage)
}
}
}
}
// corpusIndex holds the shortcut a one-word all-fields clause takes - it
// asks the document's entry in the index instead of walking its text - to
// the walk, for a spread of the vocabulary over every document.
func corpusIndex(t *testing.T) {
var vocab []string
for w := range corpus().tfidf.DocFreq {
vocab = append(vocab, w)
}
slices.Sort(vocab)
checked := 0
for i := 0; i < len(vocab); i += 131 {
w := vocab[i]
docs := 0
for di := range corpus().archive.Docs {
d := &corpus().archive.Docs[di]
_, indexed := d.terms[w]
if indexed != containsTokens(d.StemmedText, w) {
t.Fatalf("%q in %s: the index says %v, the all-fields stream the opposite", w, d.ID, indexed)
}
if indexed {
docs++
}
}
if docs != corpus().tfidf.DocFreq[w] {
t.Errorf("%q: in %d documents' entries, with a document frequency of %d", w, docs, corpus().tfidf.DocFreq[w])
}
checked++
}
t.Logf("%d terms checked over %d documents", checked, len(corpus().archive.Docs))
}

123
tests/data_test.go Normal file
View file

@ -0,0 +1,123 @@
package main
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
)
func TestAkaIDShape(t *testing.T) {
for _, s := range []string{
`{"title": "no aka_id key"}`,
`{"aka_id": []}`,
// null is "no akas": not a reason to drop the document.
`{"aka_id": null}`,
`{"aka_id": [{"id": "a"}, {"id": "b", "doc_source": "https://e.org/d", "doc_url": "https://e.org/b.pdf"}]}`,
} {
var d Document
if err := json.Unmarshal([]byte(s), &d); err != nil {
t.Errorf("%s: unexpected error: %v", s, err)
}
}
for _, s := range []string{`{"aka_id": "a"}`, `{"aka_id": {"id": "a"}}`, `{"aka_id": ["a", "b"]}`, `{"aka_id": [{"id": "a"}, 3]}`} {
var d Document
if err := json.Unmarshal([]byte(s), &d); err == nil {
t.Errorf("%s: decoded without an error", s)
}
}
}
func TestNormalizeAka(t *testing.T) {
d := Document{ID: "x", AkaID: []AkaEntry{
{ID: " first ", DocSource: " https://e.org/p "},
{ID: " "},
{ID: "FIRST"},
{ID: "second", DocURL: "https://e.org/s.pdf"},
}}
got := normalizeAka(&d)
if len(got) != 2 || got[0].ID != "first" || got[0].DocSource != "https://e.org/p" || got[1].ID != "second" {
t.Errorf("normalizeAka = %+v, want first (trimmed) and second, without the blank id and the repeat", got)
}
}
func writeTestFile(t *testing.T, path, content string) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
func TestLoadVersions(t *testing.T) {
dir := t.TempDir()
v := filepath.Join(dir, "versions")
writeTestFile(t, filepath.Join(v, "lemonde-2013", "copy.pdf"), "%PDF-1.4")
writeTestFile(t, filepath.Join(v, "lemonde-2013", "copy.md"), "# Slide 12\n\nOnly in Le Monde's copy.")
writeTestFile(t, filepath.Join(v, "lemonde-2013", "images", "page-10.jpg"), "")
writeTestFile(t, filepath.Join(v, "lemonde-2013", "images", "page-2.jpg"), "")
writeTestFile(t, filepath.Join(v, "guardian", "copy.pdf"), "%PDF-1.4") // no transcript or images: still a version
writeTestFile(t, filepath.Join(v, "no-pdf", "copy.md"), "text only") // skipped: no PDF
writeTestFile(t, filepath.Join(v, "bad name", "copy.pdf"), "%PDF-1.4") // skipped: name
writeTestFile(t, filepath.Join(v, "loose.pdf"), "%PDF-1.4") // skipped: not a folder
var d Document
if err := json.Unmarshal([]byte(`{"publication": [
{"publisher": "Le Monde", "version": "lemonde-2013"},
{"publisher": "The Guardian"},
{"publisher": "Dagbladet", "version": " lemonde-2013 "},
{"publisher": "Nobody", "version": "missing"}
]}`), &d); err != nil {
t.Fatal(err)
}
d.ID, d.Dir = "prism", dir
corpus().loadVersions(&d)
var slugs []string
for _, v := range d.Versions {
slugs = append(slugs, v.Slug)
}
if got := strings.Join(slugs, ","); got != "guardian,lemonde-2013" {
t.Fatalf("versions %q, want guardian,lemonde-2013", got)
}
lm := d.VersionFor("lemonde-2013")
if lm.PDFFile != "copy.pdf" || !strings.Contains(lm.FullText, "Only in Le Monde") || lm.HTMLContent == "" {
t.Errorf("lemonde-2013 not fully loaded: pdf %q, text %q, html %q", lm.PDFFile, lm.FullText, lm.HTMLContent)
}
if got := strings.Join(lm.Images, ","); got != "page-2.jpg,page-10.jpg" {
t.Errorf("images %q, want page-2.jpg,page-10.jpg (page order)", got)
}
if got := fmt.Sprint(lm.PubIndex); got != "[0 2]" {
t.Errorf("PubIndex %s, want [0 2]", got)
}
if got := d.Publication[2].Version; got != "lemonde-2013" {
t.Errorf("publication version not trimmed: %q", got)
}
if u := d.UnlinkedVersions(); len(u) != 1 || u[0].Slug != "guardian" {
t.Errorf("unlinked versions %+v, want only guardian", u)
}
if d.VersionFor("missing") != nil || d.VersionFor("") != nil {
t.Error("VersionFor returned a version that doesn't exist")
}
}
// TestBodyFormsAreTheCombinedTails: loadData slices FoldedFull and
// StemmedFull out of the ends of SearchText and StemmedText to save memory;
// they must still be exactly what setBodySearchText builds.
func TestBodyFormsAreTheCombinedTails(t *testing.T) {
buildFixture(t)
for i := range corpus().archive.Docs {
d := &corpus().archive.Docs[i]
c := *d
setBodySearchText(&c)
if c.FoldedFull != d.FoldedFull || c.StemmedFull != d.StemmedFull {
t.Errorf("%s: body forms differ from a fresh build", d.ID)
}
}
}

295
tests/fixture_test.go Normal file
View file

@ -0,0 +1,295 @@
package main
import (
"bytes"
"encoding/json"
"image"
"image/jpeg"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync"
"testing"
)
// A small corpus on disk plus a running mux: the harness every handler test
// uses. The fixture covers the shapes the handlers branch on - one document
// per collection, a SIDtoday issue with a series, another published copy in
// versions/, a joint investigation with a reporting-only partner, and a
// directory with no PDF (which the loader must skip).
//
// Built once for the package: loadData writes package globals, so tests
// share one corpus rather than racing to replace it.
var (
fixtureOnce sync.Once
fixtureMux http.Handler
fixtureDir string
)
type fixtureDoc struct {
id, collection string
meta map[string]interface{}
text string
pages []string
versions map[string][]string // slug -> page images
noPDF bool
}
// fixtureJPEG is a real 4x3 JPEG, so the loader can read each page image's size.
var fixtureJPEG = func() string {
var b bytes.Buffer
if err := jpeg.Encode(&b, image.NewGray(image.Rect(0, 0, 4, 3)), nil); err != nil {
panic(err)
}
return b.String()
}()
func writeFixtureFile(t *testing.T, path, content string) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
// TestMain removes the shared fixture corpus once every test has run.
func TestMain(m *testing.M) {
code := m.Run()
if fixtureDir != "" {
os.RemoveAll(fixtureDir)
}
os.Exit(code)
}
func buildFixture(t *testing.T) (http.Handler, string) {
t.Helper()
fixtureOnce.Do(func() {
root, err := os.MkdirTemp("", "snowchive-fixture")
if err != nil {
t.Fatal(err)
}
for _, d := range fixtureDocs() {
dir := root
switch d.collection {
case collectionFOIA:
dir = filepath.Join(root, "FOIA")
case collectionArchive:
dir = filepath.Join(root, "SIDtoday-archive")
}
dir = filepath.Join(dir, d.id)
b, err := json.MarshalIndent(d.meta, "", " ")
if err != nil {
t.Fatal(err)
}
writeFixtureFile(t, filepath.Join(dir, "metadata.json"), string(b)+"\n")
if !d.noPDF {
writeFixtureFile(t, filepath.Join(dir, d.id+".pdf"), "%PDF-1.4\nfixture\n")
}
if d.text != "" {
writeFixtureFile(t, filepath.Join(dir, d.id+".md"), d.text)
}
for _, p := range d.pages {
writeFixtureFile(t, filepath.Join(dir, "images", p), fixtureJPEG)
}
for slug, imgs := range d.versions {
vdir := filepath.Join(dir, "versions", slug)
writeFixtureFile(t, filepath.Join(vdir, slug+".pdf"), "%PDF-1.4\n")
writeFixtureFile(t, filepath.Join(vdir, slug+".md"),
"# Copy\n\nA line only this copy has: unredacted FAIRVIEW detail.\n")
for _, p := range imgs {
writeFixtureFile(t, filepath.Join(vdir, "images", p), fixtureJPEG)
}
}
}
writeFixtureFile(t, filepath.Join(root, "glossary.json"), `[
{"term": "XKEYSCORE", "definition": "A search system.", "aka": ["XKS"]}
]`)
dataDir, projectDir = root, root
devMode = false
apiEnabled = true
initAssets()
if err := parseTemplates(); err != nil {
t.Fatalf("templates: %v", err)
}
loadData()
fixtureDir = root
fixtureMux = withBaseHeaders(newMux())
})
if fixtureMux == nil {
t.Fatal("fixture not built")
}
return fixtureMux, fixtureDir
}
func fixtureDocs() []fixtureDoc {
return []fixtureDoc{
{
id: "prism-slides",
collection: collectionNews,
text: "# PRISM\n\nCollection directly from the servers. XKEYSCORE ingests it.\n" +
"Upstream collection feeds the corpus, and analysts were collecting metadata\n" +
"from RT-RG dashboards. Müller-Maguhn reviewed the São Paulo cable tapping.\n",
pages: []string{"page-1.jpg", "page-2.jpg"},
versions: map[string][]string{"lemonde": {"page-1.jpg"}},
meta: map[string]interface{}{
"title": "PRISM Collection Slides",
"type": "REPORTED",
"description": "Slides describing PRISM collection.",
"tags": []string{"PRISM", "collection", "cable tapping"},
"pdf_sha256": strings.Repeat("a", 64),
"aka_id": []map[string]string{{"id": "prism-slides-guardian"}},
"document": map[string]interface{}{
"creator": "NSA", "date_created": "2013-04-01",
"programs": []string{"PRISM", "XKEYSCORE"},
"classification": []string{"TOP SECRET"}, "dissemination": []string{"NOFORN"},
"countries": []string{"United States", "Brazil"},
"people": []string{"Snowden, Edward", "Müller-Maguhn, Andy"},
"banner": "TOP SECRET//SI//NOFORN", "genre": "slides",
},
"publication": []map[string]interface{}{
{"publisher": "The Guardian", "title": "NSA Prism program",
"link": "https://example.org/prism", "date_published": "2013-06-06",
"reporters": []string{"Greenwald, Glenn"}},
{"publisher": "Le Monde", "link": "https://example.org/prism-fr",
"date_published": "2013-06-08", "version": "lemonde"},
{"publisher": "NRK", "link": "https://example.org/prism",
"date_published": "2013-06-06", "reporting_only": true},
},
},
},
{
// Shares the Guardian link with prism-slides, which is what
// makes the "published in the same report" section exist.
id: "prism-companion", collection: collectionNews,
text: "# Companion\n\nA second document from the same story.\n",
pages: []string{"page-1.jpg"},
meta: map[string]interface{}{
"title": "PRISM Companion Slide", "type": "REPORTED",
"description": "A second slide from the same story.",
"tags": []string{"PRISM"},
"document": map[string]interface{}{
"creator": "NSA", "date_created": "2013-04-02",
"programs": []string{"PRISM"},
},
"publication": []map[string]interface{}{
{"publisher": "The Guardian", "title": "NSA Prism program",
"link": "https://example.org/prism", "date_published": "2013-06-06",
"reporters": []string{"Greenwald, Glenn"}},
},
},
},
{
id: "2003-05-29_SIDToday_-_Fixture_Issue",
collection: collectionArchive,
text: "# SIDtoday\n\nThe analyst cockpit and other matters. Upstream tapping was\n" +
"discussed at length by the directorate, and the collection of that traffic\n" +
"began only the following year.\n",
pages: []string{"page-1.jpg"},
meta: map[string]interface{}{
"title": "A Day in the Life of a Fixture", "type": "SIDTODAY",
"description": "An issue of SIDtoday.",
"tags": []string{"analysis"},
"document": map[string]interface{}{
"creator": "NSA", "date_created": "2003-05-29",
"classification": []string{"UNCLASSIFIED"}, "distribution": []string{},
"series": map[string]interface{}{"name": "(U) A Day in the Life of...", "part": 1},
},
"publication": []map[string]interface{}{
{"publisher": "The Intercept", "link": "https://example.org/sidtoday/1",
"date_published": "2016-05-16"},
},
},
},
{
id: "FISC_-_fixture_order",
collection: collectionFOIA,
text: "# Order\n\nThe Court finds the application satisfies the statute.\n",
meta: map[string]interface{}{
"title": "Fixture FISC Order", "type": "FOIA",
"description": "A court order released under FOIA.",
"document": map[string]interface{}{
"creator": "FISC", "date_created": "2013-10-11",
"classification": []string{"SECRET"},
},
"publication": []map[string]interface{}{{"publisher": "ODNI", "link": "https://example.org/foia/1"}},
},
},
{
id: "undated-fixture",
collection: collectionNews,
text: "# Undated\n\nNo document date at all.\n",
meta: map[string]interface{}{
"title": "Undated Fixture", "type": "REPORTED",
"description": "A document with no date.",
"document": map[string]interface{}{"creator": "GCHQ"},
"publication": []map[string]interface{}{{"publisher": "The Intercept", "link": "https://example.org/undated"}},
},
},
{
id: "broken-no-pdf",
collection: collectionNews,
noPDF: true,
meta: map[string]interface{}{"title": "Broken", "type": "REPORTED"},
},
}
}
// get runs one request through the real routing table and header wrapper.
func get(t *testing.T, path string, cookies ...*http.Cookie) *httptest.ResponseRecorder {
t.Helper()
mux, _ := buildFixture(t)
r := httptest.NewRequest("GET", path, nil)
for _, c := range cookies {
r.AddCookie(c)
}
w := httptest.NewRecorder()
mux.ServeHTTP(w, r)
return w
}
// newPost builds a form POST for the state-changing endpoints.
func newPost(path string) *http.Request {
r := httptest.NewRequest("POST", path, strings.NewReader("back=/browse"))
r.Header.Set("Content-Type", "application/x-www-form-urlencoded")
return r
}
func record(h http.Handler, r *http.Request) *httptest.ResponseRecorder {
w := httptest.NewRecorder()
h.ServeHTTP(w, r)
return w
}
// withCorpus loads another corpus for one test and puts the fixture back
// afterwards. Only possible because loadData is idempotent - every build*
// starts from empty (TestLoadDataIsIdempotent).
func withCorpus(t *testing.T, root string) {
t.Helper()
withDirs(t, root, root)
}
// withDirs is withCorpus with the corpus and the project files apart, as
// -root lays them out.
func withDirs(t *testing.T, data, project string) {
t.Helper()
buildFixture(t) // make sure templates and the fixture path exist first
prev := dataDir
t.Cleanup(func() {
dataDir, projectDir = prev, prev
loadData()
})
dataDir, projectDir = data, project
loadData()
}
// httptest_NewGet is a plain GET request for the tests that drive a mux
// directly rather than through get().
func httptest_NewGet(path string) *http.Request {
return httptest.NewRequest("GET", path, nil)
}

167
tests/flags_test.go Normal file
View file

@ -0,0 +1,167 @@
package main
// flags_test.go - what each command-line flag actually changes. The flags
// are parsed in main(), which no test can call, so these exercise the state
// the flags set and the behaviour that depends on it: the routes an
// instance serves, the URLs it writes, the warnings it prints.
import (
"encoding/json"
"net/http"
"path/filepath"
"strings"
"testing"
)
// -no-api: every /api/ path answers a JSON 403, and the UI stops offering
// the API. The kill switch is one registration, so a future endpoint
// inherits it - that is the property worth testing.
func TestFlagNoAPI(t *testing.T) {
buildFixture(t)
prev := apiEnabled
t.Cleanup(func() { apiEnabled = prev; fixtureMux = withBaseHeaders(newMux()) })
apiEnabled = false
mux := withBaseHeaders(newMux())
for _, p := range []string{"/api/docs", "/api/doc/prism-slides", "/api/search?q=x",
"/api/facets", "/api/something-added-next-year"} {
w := record(mux, httptest_NewGet(p))
if w.Code != http.StatusForbidden {
t.Errorf("%s with -no-api: status %d, want 403", p, w.Code)
}
var body map[string]string
if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil || body["error"] == "" {
t.Errorf("%s: body %q is not a JSON error", p, w.Body.String())
}
if ct := w.Header().Get("Content-Type"); !strings.HasPrefix(ct, "application/json") {
t.Errorf("%s: Content-Type %q", p, ct)
}
}
// The visitor's own export is NOT part of the API and stays available.
if w := record(mux, httptest_NewGet("/pinned/export")); w.Code != 200 {
t.Errorf("/pinned/export with -no-api: %d", w.Code)
}
}
// -base: the absolute URLs in robots.txt and the sitemap follow
// the canonical host instead of the request's Host header.
func TestFlagBase(t *testing.T) {
buildFixture(t)
prev, prevMap := canonicalBase, corpus().canonicalSitemap
t.Cleanup(func() {
canonicalBase, corpus().canonicalSitemap = prev, prevMap
corpus().buildCanonicalSitemap()
})
canonicalBase = "https://archive.example.org"
corpus().buildCanonicalSitemap()
for _, c := range []struct{ path, want string }{
{"/robots.txt", "Sitemap: https://archive.example.org/sitemap.xml"},
{"/sitemap.xml", "<loc>https://archive.example.org/doc/prism-slides</loc>"},
} {
if body := get(t, c.path).Body.String(); !strings.Contains(body, c.want) {
t.Errorf("%s does not carry %q", c.path, c.want)
}
}
// A hostile Host header cannot reach a response once -base is set.
r := httptest_NewGet("/robots.txt")
r.Host = "evil.example.net"
if body := record(fixtureMux, r).Body.String(); strings.Contains(body, "evil") {
t.Errorf("the Host header leaked into robots.txt: %s", body)
}
}
// -base is validated, not trusted: anything but scheme://host is refused,
// so the flag cannot produce URLs that do not resolve.
func TestFlagBaseValidation(t *testing.T) {
for _, ok := range []string{"https://a.org", "http://a.org", "https://a.org:8443", "https://a.org/"} {
if _, err := parseCanonicalBase(ok); err != nil {
t.Errorf("%s rejected: %v", ok, err)
}
}
for _, bad := range []string{"", "a.org", "ftp://a.org", "https://", "https://a.org/path",
"https://a.org?q=1", "https://a.org#f", "https://user@a.org", "://x"} {
if _, err := parseCanonicalBase(bad); err == nil {
t.Errorf("%q accepted", bad)
}
}
}
// -check counts what it reports: the fixture holds one document the loader
// refuses (no PDF) and one directory that looks like a document.
func TestFlagCheckCounts(t *testing.T) {
buildFixture(t)
if loadErrors == 0 {
t.Error("the fixture's PDF-less document should be an error")
}
if loadErrors+loadWarnings == 0 {
t.Error("-check would exit 0 on a corpus with problems")
}
}
// -listen warns for anything but loopback, because binding elsewhere puts
// the archive in front of the proxy instead of behind it.
func TestFlagListenWarning(t *testing.T) {
before := loadWarnings
for _, quiet := range []string{"127.0.0.1:8080", "localhost:8080", "[::1]:8080", "127.0.0.53:80"} {
warnNonLoopback(quiet)
if loadWarnings != before {
t.Errorf("%s warned", quiet)
before = loadWarnings
}
}
for _, loud := range []string{"0.0.0.0:8080", ":8080", "192.168.1.10:8080", "[::]:8080"} {
warnNonLoopback(loud)
if loadWarnings == before {
t.Errorf("%s did not warn", loud)
}
before = loadWarnings
}
}
// -dev: templates come from disk and nothing is pre-rendered, so every page
// must still render - the fallback path the pre-rendered blocks rely on.
func TestFlagDevMode(t *testing.T) {
buildFixture(t)
prevDev, prevPre, prevTmpl := devMode, corpus().prerendered, tmplSet
t.Cleanup(func() {
devMode, corpus().prerendered, tmplSet = prevDev, prevPre, prevTmpl
initAssets()
})
devMode = true
initAssets() // templates/ and static/ from the working directory
corpus().prerenderPages()
if len(corpus().prerendered) != 0 {
t.Error("dev mode pre-rendered pages; it must render them live")
}
for _, p := range []string{"/", "/browse", "/browse/program", "/explore/ids", "/glossary",
"/articles", "/explore/terms", "/doc/prism-slides"} {
if w := get(t, p); w.Code != 200 {
t.Errorf("%s in dev mode: %d", p, w.Code)
}
}
}
// -root: the documents are read from its data/ and the files about the
// archive from its project/.
func TestRootDir(t *testing.T) {
buildFixture(t)
prevData, prevProject := dataDir, projectDir
t.Cleanup(func() {
dataDir, projectDir = prevData, prevProject
loadData()
})
root := t.TempDir()
writeFixtureFile(t, filepath.Join(root, "data", "doc-in-root", "metadata.json"), `{"title": "In the root"}`)
writeFixtureFile(t, filepath.Join(root, "data", "doc-in-root", "doc-in-root.pdf"), "%PDF-1.4\n")
writeFixtureFile(t, filepath.Join(root, "project", "glossary.json"), `[{"term": "INPROJECT", "definition": "Read from project/."}]`)
setRoot(root)
loadData()
if corpus().docIndex["doc-in-root"] == nil {
t.Error("the document in data/ was not loaded")
}
if corpus().glossaryLookup("INPROJECT") == nil {
t.Error("glossary.json in project/ was not loaded")
}
}

272
tests/foia_test.go Normal file
View file

@ -0,0 +1,272 @@
package main
import (
"html"
"net/http"
"net/url"
"regexp"
"strconv"
"strings"
"testing"
)
// foiaOn is the cookie of a reader who opted in to FOIA releases.
var foiaOn = &http.Cookie{Name: "pref_include_foia", Value: "1"}
// In the consistency corpus "alpha" is in n1, n2 and the FOIA release f1.
func TestSearchLeavesOutFOIAByDefault(t *testing.T) {
withCorpus(t, writeConsistencyCorpus(t))
if got := resultTotal(t, "/search?q=alpha"); got != 2 {
t.Errorf("q=alpha, no preference: %d results, want 2 (FOIA left out)", got)
}
body := get(t, "/search?q=alpha").Body.String()
if strings.Contains(body, "left out") {
t.Error("the page counts the FOIA releases it left out")
}
// The page's own links name the collections it searched, so a copied
// link shows the same results to a reader with the other preference.
if !strings.Contains(body, "coll=news&amp;coll=sidtoday-archive") {
t.Error("the page's links do not carry the default collections")
}
if got := resultTotal(t, "/search?q=alpha&coll=news&coll=sidtoday-archive&coll=foia"); got != 3 {
t.Errorf("q=alpha in every collection: %d results, want 3", got)
}
if got := resultTotal(t, "/search?q=alpha", foiaOn); got != 3 {
t.Errorf("q=alpha with FOIA on: %d results, want 3", got)
}
if strings.Contains(get(t, "/search?q=alpha", foiaOn).Body.String(), "left out") {
t.Error("FOIA on, yet the page says something was left out")
}
// An explicit collection choice wins over the preference.
if got := resultTotal(t, "/search?q=alpha&coll=foia"); got != 1 {
t.Errorf("q=alpha&coll=foia, no preference: %d results, want 1", got)
}
if got := resultTotal(t, "/search?q=alpha&coll=news", foiaOn); got != 2 {
t.Errorf("q=alpha&coll=news with FOIA on: %d results, want 2", got)
}
}
// ZEBRA is on n1, n2, f1 (FOIA) and s1 (SIDtoday).
func TestCategoryPagesLeaveOutFOIAByDefault(t *testing.T) {
withCorpus(t, writeConsistencyCorpus(t))
if got := resultTotal(t, "/browse/program/ZEBRA"); got != 3 {
t.Errorf("ZEBRA, no preference: %d documents, want 3", got)
}
if strings.Contains(get(t, "/browse/program/ZEBRA").Body.String(), "left out") {
t.Error("the page still carries a FOIA notice")
}
if got := resultTotal(t, "/browse/program/ZEBRA", foiaOn); got != 4 {
t.Errorf("ZEBRA with FOIA on: %d documents, want 4", got)
}
if strings.Contains(get(t, "/browse/program/ZEBRA", foiaOn).Body.String(), "left out") {
t.Error("FOIA on, yet the page says FOIA releases are left out")
}
}
// Every total in a category table is the number its row's link lists, for
// either preference.
func TestCategoryTotalsMatchTheirPages(t *testing.T) {
withCorpus(t, writeConsistencyCorpus(t))
reRow := regexp.MustCompile(`(?s)<a href="([^"]+)">\s*([^<]+?)\s*</a>.*?</td>\s*<td class="num-col">(\d+)</td>`)
for _, cookies := range [][]*http.Cookie{nil, {foiaOn}} {
for _, slug := range []string{"program", "agency", "docdate", "published", "publisher"} {
body := get(t, "/browse/"+slug, cookies...).Body.String()
_, body, _ = strings.Cut(body, "<tbody>")
body, _, _ = strings.Cut(body, "</tbody>")
rows := reRow.FindAllStringSubmatch(body, -1)
if len(rows) == 0 {
t.Fatalf("/browse/%s: no rows", slug)
}
for _, row := range rows {
want, _ := strconv.Atoi(row[3])
if strings.Contains(row[1], "coll=") {
t.Errorf("/browse/%s %s: row links with a collection scope: %s", slug, row[2], row[1])
}
if got := resultTotal(t, html.UnescapeString(row[1]), cookies...); got != want {
t.Errorf("/browse/%s %s (FOIA on: %v): total %d, its page lists %d", slug, row[2], cookies != nil, want, got)
}
}
}
}
off, on := get(t, "/browse/program").Body.String(), get(t, "/browse/program", foiaOn).Body.String()
if strings.Contains(off, "Totals leave out") || strings.Contains(off, "sort=foia") {
t.Error("FOIA off: the table still has a FOIA notice or column")
}
if !strings.Contains(on, "sort=foia") {
t.Error("FOIA on: the table has no FOIA column")
}
}
func TestRandomAndRelatedLeaveOutFOIA(t *testing.T) {
withCorpus(t, writeConsistencyCorpus(t))
sawFOIA := false
for i := 0; i < 300; i++ {
if d := corpus().randomDoc(false); d == nil || d.Collection == collectionFOIA {
t.Fatalf("randomDoc(false) = %v", d)
}
if corpus().randomDoc(true).Collection == collectionFOIA {
sawFOIA = true
}
}
if !sawFOIA {
t.Error("randomDoc(true) never picked the FOIA release in 300 tries")
}
corpus().archive.RelatedDocs["n2"] = []string{"f1", "n1"}
corpus().archive.RelatedDocs["f1"] = []string{"n1"}
body := get(t, "/doc/n2").Body.String()
if strings.Contains(body, `href="/doc/f1"`) || !strings.Contains(body, `href="/doc/n1"`) {
t.Error("/doc/n2, no preference: want n1 related and f1 left out")
}
if !strings.Contains(get(t, "/doc/n2", foiaOn).Body.String(), `href="/doc/f1"`) {
t.Error("/doc/n2 with FOIA on: f1 is not related")
}
if !strings.Contains(get(t, "/doc/f1").Body.String(), `href="/doc/n1"`) {
t.Error("/doc/f1: its related documents are gone")
}
}
// The home page counts the Snowden documents apart from the FOIA releases and
// says how the preference treats them. The corpus: n1-n4, a1, d1 (news), s1
// (SIDtoday archive), f1 (FOIA).
func TestHomeStatesTheFOIAScope(t *testing.T) {
withCorpus(t, writeConsistencyCorpus(t))
body := get(t, "/").Body.String()
for _, want := range []string{"holds 7 published Snowden documents. 6 documents are", "1 documents are the NSA newsletter",
"also holds 1", "are left out by default but can be included via"} {
if !strings.Contains(body, want) {
t.Errorf("home page lacks %q", want)
}
}
if !strings.Contains(get(t, "/", foiaOn).Body.String(), "you have included via") {
t.Error("FOIA on: the home page does not say so")
}
}
func TestFOIAPreferenceIsSaved(t *testing.T) {
withCorpus(t, writeConsistencyCorpus(t))
if !strings.Contains(get(t, "/preferences").Body.String(), `name="include_foia"`) {
t.Error("/preferences has no FOIA checkbox")
}
if readPrefs(httptest_NewGet("/")).IncludeFOIA {
t.Error("FOIA releases are included by default")
}
}
// The form shows the scope the search ran with: FOIA left out by default
// is an unchecked box, so submitting the form keeps it out.
func TestSearchFormKeepsTheDefaultScope(t *testing.T) {
withCorpus(t, writeConsistencyCorpus(t))
checked := regexp.MustCompile(`value="foia" checked`)
if checked.MatchString(get(t, "/search?q=alpha").Body.String()) {
t.Error("FOIA left out, yet its box is checked")
}
if !checked.MatchString(get(t, "/search?q=alpha", foiaOn).Body.String()) {
t.Error("FOIA on, yet its box is not checked")
}
if got := resultTotal(t, "/search?q=alpha&coll=news&coll=sidtoday-archive&do_search=1"); got != 2 {
t.Errorf("the form's resubmit finds %d, want 2", got)
}
}
// A FOIA release's metadata links name every collection, so its value
// pages list it under the default scope.
func TestFOIADocLinksIncludeFOIA(t *testing.T) {
buildFixture(t)
// Every value link printed for the release - on its own page, on its
// card and its table row in the FOIA tab, in a search result - opens a
// page that lists the release.
const id = "FISC_-_fixture_order"
reValueLink := regexp.MustCompile(`href="(/browse/[a-z]+/[^"]+)"`)
for _, page := range []string{"/doc/" + id, "/browse?collection=foia", "/browse?collection=foia&view=table", "/search?q=court&coll=foia"} {
links := reValueLink.FindAllStringSubmatch(get(t, page).Body.String(), -1)
if len(links) < 2 {
t.Fatalf("%s: %d value links, want the release's agency and classification at least", page, len(links))
}
for _, m := range links {
link := html.UnescapeString(m[1])
if !strings.Contains(get(t, withQuery(link, "view=table")).Body.String(), `href="/doc/`+id+`"`) {
t.Errorf("%s links %s, which does not list the release", page, link)
}
}
}
// /explore/classification counts every collection, and so do its links.
for _, m := range regexp.MustCompile(`href="(/browse/[^"]+)"`).FindAllStringSubmatch(get(t, "/explore/classification").Body.String(), -1) {
if !strings.Contains(m[1], "coll=foia") {
t.Errorf("/explore/classification links %s without its scope", m[1])
}
}
body := get(t, "/doc/FISC_-_fixture_order").Body.String()
m := regexp.MustCompile(`href="(/browse/agency/FISC[^"]*)"`).FindStringSubmatch(body)
if m == nil || !strings.Contains(m[1], "coll=foia") {
t.Fatalf("agency link %v does not include FOIA", m)
}
if got := resultTotal(t, html.UnescapeString(m[1])); got != 1 {
t.Errorf("the agency link lists %d documents, want 1", got)
}
if strings.Contains(get(t, "/doc/FISC_-_fixture_order", foiaOn).Body.String(), "/browse/agency/FISC?") {
t.Error("FOIA on: the links need no scope")
}
}
// On a publisher page scoped to a publication year, the links built from
// its documents carry that year.
func TestPublisherYearLinksCarryTheYear(t *testing.T) {
buildFixture(t)
body := get(t, "/browse/publisher/The%20Intercept?year=2016").Body.String()
if !strings.Contains(body, "date_pub_start=2016") {
t.Error("no link on the year-scoped page carries the publication year")
}
}
// collScope is the one place a page's collection scope is decided: FOIA
// releases are opt-in when the URL names no collection, an explicit coll=
// wins, and only the search pins a defaulted scope into its links.
func TestCollScope(t *testing.T) {
off, on := Prefs{}, Prefs{IncludeFOIA: true}
none := url.Values{}
for _, c := range []struct {
name string
v url.Values
p Prefs
pin bool
foia, def bool
qs string
}{
{"default, browse", none, off, false, false, true, ""},
{"default, search", none, off, true, false, true, "coll=news&coll=sidtoday-archive"},
{"opted in, browse", none, on, false, true, true, ""},
{"opted in, search", none, on, true, true, true, "coll=news&coll=sidtoday-archive&coll=foia"},
{"explicit foia", url.Values{"coll": {"foia"}}, off, false, true, false, "coll=foia"},
{"explicit news over opt-in", url.Values{"coll": {"news"}}, on, false, false, false, "coll=news"},
{"explicit everything", url.Values{"coll": {"news", "foia", "sidtoday-archive"}}, off, false, true, false,
"coll=news&coll=sidtoday-archive&coll=foia"},
} {
s := pageScope(c.v, c.p, c.pin)
foia := s.shows(&Document{Collection: collectionFOIA})
if foia != c.foia || s.defaulted != c.def || s.qs() != c.qs || s.shown[collectionFOIA] != c.foia {
t.Errorf("%s: shows FOIA %v, defaulted %v, qs %q; want %v, %v, %q",
c.name, foia, s.defaulted, s.qs(), c.foia, c.def, c.qs)
}
}
// A FOIA release's links widen the default to the whole archive and
// say so; a reader who opted in needs no widening.
if got := pageScope(none, off, false).including(collectionFOIA).suffix(); got != "?coll=news&coll=sidtoday-archive&coll=foia" {
t.Errorf("widened default: %q", got)
}
if got := pageScope(none, on, false).including(collectionFOIA).suffix(); got != "" {
t.Errorf("opted in, widened: %q, want no scope", got)
}
// and adds to the links, not to the documents.
s := pageScope(none, off, false).and("date_pub_start=2014")
if s.link("/search?q=x") != "/search?q=x&date_pub_start=2014" || s.shows(&Document{Collection: collectionFOIA}) {
t.Errorf("and: %q", s.link("/search?q=x"))
}
}

5
tests/go.mod Normal file
View file

@ -0,0 +1,5 @@
// Keeps this directory out of the snowchive module's ./... patterns. The
// files here are package main's tests; tests/run.sh compiles them with it.
module snowchive/tests
go 1.22

53
tests/headings_test.go Normal file
View file

@ -0,0 +1,53 @@
package main
import (
"regexp"
"strconv"
"strings"
"testing"
)
var reHeadingTag = regexp.MustCompile(`<h([1-6])\b`)
func TestTranscriptHeadingsNestUnderThePage(t *testing.T) {
for src, want := range map[string]string{
"# A\n\n## B\n\n#### C\n\n## D": `<h2 class="md-h1">A</h2> <h3 class="md-h2">B</h3> <h4 class="md-h4">C</h4> <h3 class="md-h2">D</h3>`,
"### A\n\n### B\n\n#### C": `<h2 class="md-h3">A</h2> <h2 class="md-h3">B</h2> <h3 class="md-h4">C</h3>`,
"```\n# not a heading\n```\n\n## A": `<h2 class="md-h2">A</h2>`,
} {
var got []string
for _, line := range strings.Split(Render(src), "\n") {
if strings.HasPrefix(line, "<h") {
got = append(got, line)
}
}
if g := strings.Join(got, " "); g != want {
t.Errorf("Render(%q) headings:\n got %s\nwant %s", src, g, want)
}
}
}
func TestPagesHaveOneH1AndNoSkippedLevels(t *testing.T) {
for _, path := range []string{
"/", "/browse", "/browse?view=table", "/browse?collection=sidtoday-archive",
"/browse?collection=sidtoday-archive&view=card", "/browse?collection=foia",
"/browse/agency/NSA", "/search?q=prism", "/search?q=zzzqqq", "/doc/prism-slides",
"/doc/prism-slides/versions/lemonde", "/glossary", "/articles", "/about", "/pinned",
} {
body := get(t, path).Body.String()
h1, prev := 0, 0
for _, m := range reHeadingTag.FindAllStringSubmatch(body, -1) {
level, _ := strconv.Atoi(m[1])
if level == 1 {
h1++
}
if level > prev+1 {
t.Errorf("%s: <h%d> follows <h%d>", path, level, prev)
}
prev = level
}
if h1 != 1 {
t.Errorf("%s: %d <h1>, want 1", path, h1)
}
}
}

231
tests/http_test.go Normal file
View file

@ -0,0 +1,231 @@
package main
// http_test.go - protocol behaviour the browser and the proxy rely on:
// HEAD, byte ranges, conditional requests, and method handling. net/http
// implements most of it, but only if the handlers hand it the right thing -
// and a download path that loses Range support breaks resuming a 120 MB PDF
// and the page-by-page fetches a PDF viewer makes.
import (
"fmt"
"net/http"
"net/http/httptest"
"path/filepath"
"strings"
"testing"
)
func TestHTTPHead(t *testing.T) {
buildFixture(t)
for _, p := range []string{"/", "/browse", "/doc/prism-slides",
"/files/prism-slides/prism-slides.pdf", "/api/facets", "/static/style.css"} {
r := httptest.NewRequest("HEAD", p, nil)
w := httptest.NewRecorder()
fixtureMux.ServeHTTP(w, r)
if w.Code != 200 {
t.Errorf("HEAD %s: %d", p, w.Code)
continue
}
// net/http drops the body for HEAD at the transport layer, so the
// recorder still sees one; the headers are what matters here.
if w.Header().Get("Content-Type") == "" {
t.Errorf("HEAD %s: no Content-Type", p)
}
}
}
// Range on a document file. http.ServeFile implements it - this test is
// here so a future change to the download path (a buffered writer, a
// compressing wrapper, a manual io.Copy) cannot quietly remove it.
func TestHTTPRangeOnFiles(t *testing.T) {
buildFixture(t)
const path = "/files/prism-slides/prism-slides.pdf"
full := get(t, path)
body := full.Body.String()
if full.Header().Get("Accept-Ranges") != "bytes" {
t.Fatalf("no Accept-Ranges on a document file: %v", full.Header())
}
req := func(rangeHdr string) *httptest.ResponseRecorder {
r := httptest.NewRequest("GET", path, nil)
r.Header.Set("Range", rangeHdr)
w := httptest.NewRecorder()
fixtureMux.ServeHTTP(w, r)
return w
}
w := req("bytes=0-3")
if w.Code != http.StatusPartialContent {
t.Fatalf("Range bytes=0-3: status %d, want 206", w.Code)
}
if got, want := w.Body.String(), body[:4]; got != want {
t.Errorf("range body %q, want %q", got, want)
}
if cr := w.Header().Get("Content-Range"); cr != fmt.Sprintf("bytes 0-3/%d", len(body)) {
t.Errorf("Content-Range %q", cr)
}
// A suffix range: what a resumed download asks for.
w = req(fmt.Sprintf("bytes=%d-", len(body)-4))
if w.Code != http.StatusPartialContent || w.Body.String() != body[len(body)-4:] {
t.Errorf("suffix range: %d %q", w.Code, w.Body.String())
}
// Past the end is a 416 with the length, not a 200 of the whole file.
w = req(fmt.Sprintf("bytes=%d-%d", len(body)+10, len(body)+20))
if w.Code != http.StatusRequestedRangeNotSatisfiable {
t.Errorf("unsatisfiable range: status %d, want 416", w.Code)
}
// A malformed Range is refused with 416 - net/http's choice, not this
// server's; it is recorded here so a change in behaviour is noticed.
if w = req("bytes=abc"); w.Code != http.StatusRequestedRangeNotSatisfiable {
t.Errorf("malformed Range: status %d, want 416", w.Code)
}
// An unknown range unit is refused the same way. RFC 7233 says to
// ignore it and serve the whole representation; net/http does not, and
// nothing this archive serves sends one. Recorded, not worked around.
if w = req("pages=1-2"); w.Code != http.StatusRequestedRangeNotSatisfiable {
t.Errorf("unknown range unit: status %d, want net/http's 416", w.Code)
}
}
// Conditional requests: a document file and a static asset both carry
// validators, so a reader's second visit is a 304 instead of a re-download.
func TestHTTPConditional(t *testing.T) {
buildFixture(t)
// A document file comes off disk and carries a modification time.
const doc = "/files/prism-slides/prism-slides.pdf"
lm := get(t, doc).Header().Get("Last-Modified")
if lm == "" {
t.Fatalf("%s: no Last-Modified", doc)
}
r := httptest.NewRequest("GET", doc, nil)
r.Header.Set("If-Modified-Since", lm)
w := httptest.NewRecorder()
fixtureMux.ServeHTTP(w, r)
if w.Code != http.StatusNotModified || w.Body.Len() != 0 {
t.Errorf("%s with If-Modified-Since: status %d, %d bytes", doc, w.Code, w.Body.Len())
}
// An embedded asset has no modification time - embed.FS zeroes it - so
// it carries a content ETag instead. Without one, every page view
// re-downloads the stylesheet on an unstamped build.
for _, asset := range []string{"/static/style.css", "/static/main.js"} {
first := get(t, asset)
tag := first.Header().Get("ETag")
if tag == "" {
t.Errorf("%s: no ETag", asset)
continue
}
r := httptest.NewRequest("GET", asset, nil)
r.Header.Set("If-None-Match", tag)
w := httptest.NewRecorder()
fixtureMux.ServeHTTP(w, r)
if w.Code != http.StatusNotModified || w.Body.Len() != 0 {
t.Errorf("%s with If-None-Match: status %d, %d bytes (want an empty 304)",
asset, w.Code, w.Body.Len())
}
// A different asset's tag must not satisfy it.
r = httptest.NewRequest("GET", asset, nil)
r.Header.Set("If-None-Match", `"0000000000000000"`)
w = httptest.NewRecorder()
fixtureMux.ServeHTTP(w, r)
if w.Code != 200 {
t.Errorf("%s with a stale ETag: status %d, want the asset", asset, w.Code)
}
}
}
// Method handling: the routes declare GET or POST, so anything else is a
// 405 from the mux rather than a handler running on the wrong verb.
func TestHTTPMethods(t *testing.T) {
buildFixture(t)
for _, c := range []struct {
method, path string
want int
}{
// Methods the archive never answers stop at the wrapper.
{"PUT", "/api/docs", http.StatusMethodNotAllowed},
{"DELETE", "/doc/prism-slides", http.StatusMethodNotAllowed},
{"OPTIONS", "/", http.StatusMethodNotAllowed},
{"TRACE", "/browse", http.StatusMethodNotAllowed},
// A supported method on the wrong route is a 404: the catch-all
// route claims every path, which is what keeps unknown paths on the
// archive's own 404 page, and it claims them for POST too.
{"POST", "/browse", http.StatusNotFound},
{"GET", "/preferences/save", http.StatusNotFound},
{"GET", "/pin/prism-slides", http.StatusNotFound},
} {
r := httptest.NewRequest(c.method, c.path, nil)
w := httptest.NewRecorder()
fixtureMux.ServeHTTP(w, r)
if w.Code != c.want {
t.Errorf("%s %s: status %d, want %d", c.method, c.path, w.Code, c.want)
}
}
}
// Odd but legal requests must not break anything: repeated parameters,
// unknown parameters, a very long path, an empty query.
func TestHTTPOddRequests(t *testing.T) {
buildFixture(t)
for _, c := range []struct {
path string
want int
}{
{"/browse?sort=title&sort=date", 200},
{"/browse?unknown=1&another=2", 200},
{"/search?q=a&q=b&do_search=1", 200},
{"/browse/program/" + strings.Repeat("x", 2000), 200}, // an empty page, not a crash
{"/doc/" + strings.Repeat("x", 2000), 404},
{"/browse?", 200},
{"/search?do_search=1&fqv=&fqv=&fqv=", 200},
} {
if w := get(t, c.path); w.Code != c.want {
t.Errorf("%.60s: status %d, want %d", c.path, w.Code, c.want)
}
}
}
// With -accel-files the file is nginx's to send: the handler answers only
// the files it would have served itself, with the redirect and no body.
func TestAccelFiles(t *testing.T) {
buildFixture(t)
accelPrefix = "/_files/"
defer func() { accelPrefix = "" }()
w := get(t, "/files/prism-slides/prism-slides.pdf")
rel, err := filepath.Rel(dataDir, filepath.Join(corpus().docIndex["prism-slides"].Dir, "prism-slides.pdf"))
if err != nil {
t.Fatal(err)
}
if got, want := w.Header().Get("X-Accel-Redirect"), "/_files/"+filepath.ToSlash(rel); got != want {
t.Errorf("X-Accel-Redirect %q, want %q", got, want)
}
if w.Code != http.StatusOK || w.Body.Len() != 0 {
t.Errorf("accel answer: status %d, %d body bytes, want an empty 200", w.Code, w.Body.Len())
}
if w.Header().Get("Cache-Control") == "" {
t.Error("accel answer carries no Cache-Control for nginx to pass on")
}
for _, p := range []string{"/files/prism-slides/zz-stray.pdf", "/files/prism-slides/../glossary.json", "/files/no-such-doc/x.pdf"} {
w := get(t, p)
if w.Header().Get("X-Accel-Redirect") != "" || w.Code == http.StatusOK {
t.Errorf("%s: status %d, X-Accel-Redirect %q - a file the server does not serve was handed to nginx", p, w.Code, w.Header().Get("X-Accel-Redirect"))
}
}
}
func TestAccelPathEscapes(t *testing.T) {
old, oldPrefix := dataDir, accelPrefix
dataDir, accelPrefix = "/srv/data", "/_files/"
defer func() { dataDir, accelPrefix = old, oldPrefix }()
if got, ok := accelPath("/srv/data/FOIA/a doc/50%?.pdf"); !ok || got != "/_files/FOIA/a%20doc/50%25%3F.pdf" {
t.Errorf("accelPath = %q, %v", got, ok)
}
if got, ok := accelPath("/srv/other/x.pdf"); ok {
t.Errorf("a file outside the data directory mapped to %q", got)
}
}

99
tests/ids_test.go Normal file
View file

@ -0,0 +1,99 @@
package main
import (
"bytes"
"log"
"os"
"path/filepath"
"strings"
"testing"
)
// The /explore/ids table: one row per document in id order, versions nested
// under their document, and a checksum file that sha256sum -c can read from
// the corpus root.
func TestBuildIDRows(t *testing.T) {
oldDocs, oldDir := corpus().archive.Docs, dataDir
t.Cleanup(func() {
// The id tables are package state the routes serve: rebuild them
// from the restored corpus, or the fake rows below leak into
// every later test.
corpus().archive.Docs, dataDir = oldDocs, oldDir
corpus().buildIDRows()
})
dataDir = "/corpus"
corpus().archive.Docs = []Document{
{
ID: "zebra", Dir: filepath.Join(dataDir, "FOIA", "zebra"), Collection: "foia",
PDFFile: "zebra.pdf", PDFSize: 2 << 20, Pages: 3,
},
{
ID: "alpha", Dir: filepath.Join(dataDir, "alpha"), Collection: "news",
PDFFile: "alpha.pdf", PDFSize: 1500, Pages: 1,
PDFSha256: "abc123", AkaID: []AkaEntry{{ID: "alpha-lemonde"}},
Versions: []DocVersion{{Slug: "entier", PDFFile: "copy.pdf", PDFSize: 4096, Images: []string{"a", "b"}, SHA: "def456"}},
},
}
corpus().buildIDRows()
if len(corpus().idRows) != 2 || corpus().idRows[0].ID != "alpha" || corpus().idRows[1].ID != "zebra" {
t.Fatalf("rows out of order: %v", corpus().idRows)
}
if got := corpus().idRows[0].Size; got != "1 KB" {
t.Errorf("size %q, want 1 KB", got)
}
if v := corpus().idRows[0].Versions; len(v) != 1 || v[0].Pages != 2 || v[0].Size != "4 KB" || v[0].SHA != "def456" {
t.Errorf("version row %+v", v)
}
if len(corpus().idRows[1].Versions) != 0 {
t.Errorf("zebra has no versions, got %+v", corpus().idRows[1].Versions)
}
// sha256sum -c format: checksum, two spaces, path from the corpus root,
// a document's versions after it. Files without a checksum are absent.
if corpus().idChecksums != "abc123 alpha/alpha.pdf\ndef456 alpha/versions/entier/copy.pdf\n" {
t.Errorf("checksums %q", corpus().idChecksums)
}
}
func TestHumanSize(t *testing.T) {
for n, want := range map[int64]string{0: "", -1: "", 900: "900 B", 2048: "2 KB",
3 << 20: "3.0 MB", 5 << 30: "5.00 GB"} {
if got := humanSize(n); got != want {
t.Errorf("humanSize(%d) = %q, want %q", n, got, want)
}
}
if !strings.HasSuffix(humanSize(1<<20+1<<19), " MB") {
t.Errorf("1.5 MiB should render in MB")
}
}
// A version's PDF checksum comes from the metadata's
// "versions"/<slug>/pdf_sha256 only; a version without one is warned about
// and has no line in ids.txt.
func TestVersionChecksums(t *testing.T) {
root := assemblyCorpus(t, "")
writeAssemblyDoc(t, root, "listed", map[string]interface{}{
"title": "Listed",
"versions": map[string]interface{}{"copy": map[string]interface{}{"pdf_sha256": " ABC999 "}},
}, []string{"page-1.jpg"}, map[string][]string{"copy": {"page-1.jpg"}})
var logged bytes.Buffer
log.SetOutput(&logged)
withCorpus(t, root)
log.SetOutput(os.Stderr)
if v := corpus().docIndex["listed"].VersionFor("copy"); v == nil || v.SHA != "abc999" {
t.Errorf("metadata version checksum: %+v, want abc999 from the metadata", v)
}
if v := corpus().docIndex["deck"].VersionFor("fuller"); v == nil || v.SHA != "" {
t.Errorf("a version without pdf_sha256 in the metadata got a checksum: %+v", v)
}
if !strings.Contains(logged.String(), "deck: versions/fuller has no pdf_sha256") {
t.Errorf("no warning for the version without a checksum:\n%s", logged.String())
}
if !strings.Contains(corpus().idChecksums, "abc999 listed/versions/copy/copy.pdf\n") ||
strings.Contains(corpus().idChecksums, "deck/versions/fuller/") {
t.Errorf("ids.txt version lines:\n%s", corpus().idChecksums)
}
}

53
tests/image_sizes_test.go Normal file
View file

@ -0,0 +1,53 @@
package main
import (
"bytes"
"log"
"os"
"path/filepath"
"strings"
"testing"
)
func TestPageImagesCarryTheirSize(t *testing.T) {
for _, path := range []string{
"/doc/prism-slides",
"/doc/prism-slides/pages",
"/doc/prism-slides/versions/lemonde",
"/doc/prism-slides/versions/lemonde/pages",
} {
body := get(t, path).Body.String()
if n := strings.Count(body, `page-1.jpg" width="4" height="3"`); n < 2 {
t.Errorf("%s: %d page-1 images with their size, want the page and its full-screen view", path, n)
}
}
}
func TestAssemblySlidesCarryTheirSize(t *testing.T) {
root := assemblyCorpus(t, `[{"id": "sized", "title": "Sized", "slides": [{"doc": "deck", "pages": "1"}]}]`)
withCorpus(t, root)
for _, path := range []string{"/explore/assemblies", "/explore/assemblies/sized"} {
if body := get(t, path).Body.String(); !strings.Contains(body, `page-1.jpg" width="4" height="3"`) {
t.Errorf("%s: slide image without its size", path)
}
}
}
func TestUnreadablePageImageWarns(t *testing.T) {
root := assemblyCorpus(t, "")
writeAssemblyDoc(t, root, "broken", map[string]interface{}{"title": "Broken"}, []string{"page-1.jpg"}, nil)
writeFixtureFile(t, filepath.Join(root, "broken", "images", "page-1.jpg"), "not an image")
var logged bytes.Buffer
log.SetOutput(&logged)
withCorpus(t, root)
log.SetOutput(os.Stderr)
if !strings.Contains(logged.String(), "page image unreadable, served without its size: broken/images/page-1.jpg") {
t.Errorf("no warning for the unreadable image:\n%s", logged.String())
}
body := get(t, "/doc/broken").Body.String()
if !strings.Contains(body, `<img src="/files/broken/images/page-1.jpg"
`) || strings.Contains(body, `page-1.jpg" width=`) {
t.Errorf("unreadable image: want it served without width/height")
}
}

277
tests/invariants_test.go Normal file
View file

@ -0,0 +1,277 @@
package main
import (
"encoding/json"
"html"
"net/http"
"net/url"
"regexp"
"strconv"
"strings"
"testing"
)
// The archive's central promise: a number on a link is the number the click
// returns. Facet counts, refine boxes and histogram bars all assert it, and
// every one of them is computed by different code from the page it opens.
// These tests walk the real pages and follow their own links.
var (
reFacetLink = regexp.MustCompile(`<a class="facet-link" href="([^"]+)">.*?<span class="facet-count">(\d+)</span></a>`)
reBarLink = regexp.MustCompile(`<a class="hist-col" href="([^"]+)"[^>]*>\s*<span class="hist-count">(\d+)</span>`)
reTotal = regexp.MustCompile(`([\d,]+)\s+results?`)
reListTotal = regexp.MustCompile(`([\d,]+)\s+documents?`)
)
// resultTotal reads the count a /search or a list page reports.
func resultTotal(t *testing.T, path string, cookies ...*http.Cookie) int {
t.Helper()
body := get(t, path, cookies...).Body.String()
re := reTotal
if !strings.HasPrefix(path, "/search") {
re = reListTotal
}
m := re.FindStringSubmatch(body)
if m == nil {
t.Fatalf("%s: no result count in the page", path)
}
n, err := strconv.Atoi(strings.ReplaceAll(m[1], ",", ""))
if err != nil {
t.Fatalf("%s: %v", path, err)
}
return n
}
// followLinks checks every (link, count) pair a page prints, as the reader
// with these cookies sees it.
func followLinks(t *testing.T, page string, re *regexp.Regexp, what string, cookies ...*http.Cookie) int {
t.Helper()
body := get(t, page, cookies...).Body.String()
pairs := re.FindAllStringSubmatch(body, -1)
for _, p := range pairs {
link := html.UnescapeString(p[1])
shown, _ := strconv.Atoi(p[2])
if got := resultTotal(t, link, cookies...); got != shown {
t.Errorf("%s: %s says %d, its link returns %d\n %s", page, what, shown, got, link)
}
}
return len(pairs)
}
// A category page lists exactly as many documents as its facet counts.
func TestInvariantFacetCountsMatchPages(t *testing.T) {
buildFixture(t)
checked := 0
for ci := range categories {
c := &categories[ci]
for val, want := range c.Facet(corpus()) {
if got := len(corpus().categoryMatches(c.Slug, val)); got != want {
t.Errorf("%s=%q: facet says %d, the page lists %d", c.Slug, val, want, got)
}
checked++
}
}
if checked == 0 {
t.Fatal("no facet values in the fixture")
}
}
// Refine links and histogram bars open searches that must return what they
// promise - the two places where a count and its click are computed by
// different code paths.
func TestInvariantRefineAndBarCounts(t *testing.T) {
pages := []string{
"/browse/agency/NSA",
"/browse/program/PRISM",
"/browse/classification/TOP%20SECRET",
"/search?q=prism&do_search=1",
"/search?do_search=1&fqv=NSA&fqf=CR&fqc=and",
}
facets, bars := 0, 0
for _, p := range pages {
facets += followLinks(t, p, reFacetLink, "refine link")
bars += followLinks(t, p, reBarLink, "histogram bar")
}
if facets == 0 || bars == 0 {
t.Fatalf("nothing exercised: %d refine links, %d bars", facets, bars)
}
t.Logf("checked %d refine links and %d histogram bars", facets, bars)
}
// The home page is scoped to the reported collection while the category
// pages follow the reader's scope, so its bars and lists link to those
// pages with the collection named. Every one of them has to be the number
// its click returns.
func TestInvariantHomeCounts(t *testing.T) {
body := get(t, "/").Body.String()
pairs := regexp.MustCompile(`<a class="pill" href="([^"]+)">[^<]*<span class="pill-count">(\d+)</span>`).FindAllStringSubmatch(body, -1)
lists := regexp.MustCompile(`<li><a href="(/browse/[^"]+)">[^<]*</a> <span class="muted">(\d+)</span>`).FindAllStringSubmatch(body, -1)
if len(lists) < 5 {
t.Fatalf("only %d counted list links on the home page - the stat lists should link to their browse pages", len(lists))
}
pairs = append(pairs, lists...)
pairs = append(pairs, regexp.MustCompile(`<a class="hist-col" href="([^"]+)" title="(\d+) documents`).FindAllStringSubmatch(body, -1)...)
if len(pairs) < 5 {
t.Fatalf("only %d counted links on the home page", len(pairs))
}
for _, m := range pairs {
link, shown := html.UnescapeString(m[1]), m[2]
if got := resultTotal(t, link); got != mustAtoi(t, shown) {
t.Errorf("home says %s, its link returns %d: %s", shown, got, link)
}
}
}
func mustAtoi(t *testing.T, s string) int {
t.Helper()
n, err := strconv.Atoi(s)
if err != nil {
t.Fatalf("%q: %v", s, err)
}
return n
}
// The type pills follow the same rule: the count is over the full match set,
// and clicking one narrows to exactly that many.
func TestInvariantTypePillCounts(t *testing.T) {
body := get(t, "/browse").Body.String()
re := regexp.MustCompile(`href="([^"]*type=[^"]*)"[^>]*>\s*([A-Z]+)\s*<span[^>]*>\(?(\d+)\)?</span>`)
for _, m := range re.FindAllStringSubmatch(body, -1) {
link := html.UnescapeString(m[1])
shown, _ := strconv.Atoi(m[3])
if got := resultTotal(t, link); got != shown {
t.Errorf("type pill %s says %d, its link returns %d (%s)", m[2], shown, got, link)
}
}
}
// Everything the sitemap advertises must exist: those URLs are built in Go,
// away from the templates that normally escape ids.
func TestInvariantAdvertisedURLsResolve(t *testing.T) {
for _, src := range []string{"/sitemap.xml"} {
body := get(t, src).Body.String()
locs := regexp.MustCompile(`<loc>([^<]+)</loc>|<link[^>]+href="([^"]+)"`).FindAllStringSubmatch(body, -1)
if len(locs) == 0 {
t.Fatalf("%s advertises nothing", src)
}
seen := 0
for _, m := range locs {
raw := m[1]
if raw == "" {
raw = m[2]
}
u, err := url.Parse(html.UnescapeString(raw))
if err != nil || u.Path == "" {
continue
}
p := u.Path
if u.RawQuery != "" {
p += "?" + u.RawQuery
}
if w := get(t, p); w.Code != 200 {
t.Errorf("%s advertises %s, which answers %d", src, p, w.Code)
}
seen++
}
t.Logf("%s: %d URLs resolve", src, seen)
}
}
// The API's pagination envelope has to describe the page it returned, at
// every page size, and per_page=all is the export mode that skips snippets.
func TestInvariantAPIPagination(t *testing.T) {
type envelope struct {
Total int `json:"total"`
Page int `json:"page"`
PerPage int `json:"per_page"`
TotalPages int `json:"total_pages"`
Omitted bool `json:"snippets_omitted"`
Docs []struct {
ID string `json:"id"`
} `json:"docs"`
Results []struct {
ID string `json:"id"`
Snippet string `json:"snippet"`
} `json:"results"`
}
read := func(path string) envelope {
var e envelope
w := get(t, path)
if w.Code != 200 {
t.Fatalf("%s: status %d", path, w.Code)
}
if err := json.Unmarshal(w.Body.Bytes(), &e); err != nil {
t.Fatalf("%s: %v", path, err)
}
return e
}
for _, path := range []string{"/api/docs", "/api/docs?per_page=1", "/api/docs?per_page=all",
"/api/search?q=prism", "/api/search?q=prism&per_page=1", "/api/search?q=prism&per_page=all"} {
e := read(path)
n := len(e.Docs) + len(e.Results)
switch {
case e.Page < 1:
t.Errorf("%s: page %d", path, e.Page)
case e.PerPage < 1:
t.Errorf("%s: per_page %d", path, e.PerPage)
case n > e.PerPage:
t.Errorf("%s: returned %d rows for a page of %d", path, n, e.PerPage)
case e.TotalPages < 1:
t.Errorf("%s: total_pages %d", path, e.TotalPages)
}
if want := (e.Total + e.PerPage - 1) / e.PerPage; e.Total > 0 && e.TotalPages != want {
t.Errorf("%s: total_pages %d, want %d for %d/%d", path, e.TotalPages, want, e.Total, e.PerPage)
}
}
// An out-of-range page clamps rather than 500s or returns junk.
if e := read("/api/docs?p=9999"); e.Page != e.TotalPages {
t.Errorf("p=9999 gave page %d of %d", e.Page, e.TotalPages)
}
// Export mode returns everything and says why there are no snippets.
all := read("/api/search?q=prism&per_page=all")
if !all.Omitted || len(all.Results) != all.Total {
t.Errorf("per_page=all: omitted=%v, %d of %d results", all.Omitted, len(all.Results), all.Total)
}
for _, r := range all.Results {
if r.Snippet != "" {
t.Errorf("per_page=all returned a snippet for %s", r.ID)
}
}
// A normal page does carry them.
paged := read("/api/search?q=prism")
if paged.Omitted {
t.Error("a paged search should not omit snippets")
}
}
// Snippet work is bounded by maxSnippetDocs, not by the page size, and the
// page says so when the two differ.
func TestInvariantSnippetCap(t *testing.T) {
buildFixture(t)
doc := corpus().docIndex["prism-slides"]
if doc == nil {
t.Fatal("fixture document missing")
}
docs := make([]*Document, maxSnippetDocs+5)
for i := range docs {
docs[i] = doc
}
page, capped := decorateResults(docs, []string{"collection"}, []string{"TE"}, []string{"and"})
if !capped {
t.Error("a page past the budget should report itself capped")
}
withSnippets := 0
for i := range page {
if len(page[i].Snippets) > 0 {
withSnippets++
}
if page[i].Title == "" {
t.Fatalf("row %d lost its title highlighting", i)
}
}
if withSnippets != maxSnippetDocs {
t.Errorf("%d rows got snippets, want %d", withSnippets, maxSnippetDocs)
}
}

166
tests/loader_test.go Normal file
View file

@ -0,0 +1,166 @@
package main
import (
"os"
"path/filepath"
"testing"
)
// What the loader does with a corpus that is wrong in the ways corpora are
// wrong in practice. Every case must leave the rest of the archive intact -
// a broken document is skipped, never a reason to serve nothing - and say
// so in the tally that -check reads.
func TestLoaderRejectsBadDocuments(t *testing.T) {
root := t.TempDir()
write := func(path, content string) {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
good := func(id string) {
write(filepath.Join(root, id, "metadata.json"), `{"title": "`+id+`", "type": "REPORTED"}`)
write(filepath.Join(root, id, id+".pdf"), "%PDF-1.4")
}
good("keeper-one")
good("keeper-two")
// No PDF: this is an archive OF the PDFs, so the document is skipped.
write(filepath.Join(root, "no-pdf", "metadata.json"), `{"title": "No PDF"}`)
// Invalid JSON: skipped, everything else still loads.
write(filepath.Join(root, "bad-json", "metadata.json"), `{"title": "oops"`)
write(filepath.Join(root, "bad-json", "x.pdf"), "%PDF-1.4")
// aka_id that is not a list of objects: refused, not silently half-read.
write(filepath.Join(root, "bad-aka", "metadata.json"), `{"title": "Aka", "aka_id": "other-id"}`)
write(filepath.Join(root, "bad-aka", "x.pdf"), "%PDF-1.4")
// A directory that is not a document at all: skipped without a warning.
write(filepath.Join(root, "notes", "README.txt"), "just notes")
// A directory holding a PDF and nothing else: the shape worth warning
// about, since it looks like a document nobody described.
write(filepath.Join(root, "loose", "orphan.pdf"), "%PDF-1.4")
withCorpus(t, root)
clean := loadWarnings
// A document in the SIDtoday archive that is not typed SIDTODAY is on
// no browse tab: served, and warned about.
write(filepath.Join(root, "SIDtoday-archive", "mistyped", "metadata.json"), `{"title": "Mistyped", "type": "REPORTED"}`)
write(filepath.Join(root, "SIDtoday-archive", "mistyped", "x.pdf"), "%PDF-1.4")
loadData()
if corpus().docIndex["mistyped"] == nil || loadWarnings != clean+1 {
t.Errorf("a REPORTED document in SIDtoday-archive: loaded %v, %d warnings, want it loaded with %d", corpus().docIndex["mistyped"] != nil, loadWarnings, clean+1)
}
if err := os.RemoveAll(filepath.Join(root, "SIDtoday-archive")); err != nil {
t.Fatal(err)
}
loadData()
if len(corpus().archive.Docs) != 2 {
ids := make([]string, 0, len(corpus().archive.Docs))
for _, d := range corpus().archive.Docs {
ids = append(ids, d.ID)
}
t.Fatalf("loaded %v, want the two good documents", ids)
}
if loadErrors < 3 { // no-pdf, bad-aka and bad JSON are errors: not served
t.Errorf("loadErrors = %d, want at least 3", loadErrors)
}
if loadWarnings < 1 { // the loose PDF directory: served nothing, lost nothing
t.Errorf("loadWarnings = %d, want at least 1", loadWarnings)
}
for _, id := range []string{"no-pdf", "bad-json", "bad-aka", "notes", "loose"} {
if corpus().docIndex[id] != nil {
t.Errorf("%s should not have loaded", id)
}
}
// The archive is coherent: facets, search and the sitemap all see the
// same two documents.
if got := corpus().archive.CollectionCounts[collectionNews]; got != 2 {
t.Errorf("news collection count %d, want 2", got)
}
if got, _ := corpus().executeSearch(searchParams{Queries: []string{""}, Fields: []string{"TE"}, Conjs: []string{"and"}}); len(got) != 2 {
t.Errorf("match-all search found %d, want 2", len(got))
}
}
// An empty corpus is a corpus: the server loads, every page renders, and
// nothing divides by zero.
func TestLoaderEmptyCorpus(t *testing.T) {
withCorpus(t, t.TempDir())
if len(corpus().archive.Docs) != 0 {
t.Fatalf("loaded %d documents from an empty directory", len(corpus().archive.Docs))
}
for _, p := range []string{"/", "/browse", "/search?q=x&do_search=1", "/explore", "/explore/ids",
"/articles", "/api/docs", "/api/facets", "/sitemap.xml"} {
if w := get(t, p); w.Code != 200 {
t.Errorf("%s on an empty corpus: %d", p, w.Code)
}
}
}
// Values are tidied before anything reads them, and a copy the loader skips
// leaves no trace - not even a claim on its ID.
func TestLoaderTidiesBeforeUse(t *testing.T) {
root := t.TempDir()
write := func(path, content string) {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
// FOIA/ sorts before the flat root, so the PDF-less copy is seen first.
write(filepath.Join(root, "FOIA", "dup-doc", "metadata.json"), `{"title": "Shadow"}`)
write(filepath.Join(root, "dup-doc", "metadata.json"), `{"title": "Real", "type": "REPORTED"}`)
write(filepath.Join(root, "dup-doc", "x.pdf"), "%PDF-1.4")
write(filepath.Join(root, "padded", "metadata.json"), `{
"title": "Padded", "type": "REPORTED",
"document": {"creator": " NSA ", "series": {"name": "Series X ", "part": 1}},
"publication": [{"publisher": "Le Monde", "date_published": "June 2014", "version": "entier"}]
}`)
write(filepath.Join(root, "padded", "x.pdf"), "%PDF-1.4")
write(filepath.Join(root, "padded", "versions", "entier", "entier.pdf"), "%PDF-1.4")
withCorpus(t, root)
if d := corpus().docIndex["dup-doc"]; d == nil || d.Title != "Real" {
t.Errorf("dup-doc = %+v, want the copy with a PDF", d)
}
d := corpus().docIndex["padded"]
if d == nil {
t.Fatal("padded not loaded")
}
if d.DocMeta.Creator != "NSA" || d.DocMeta.Series.Name != "Series X" {
t.Errorf("creator %q, series %q: want both trimmed", d.DocMeta.Creator, d.DocMeta.Series.Name)
}
// The malformed date is dropped everywhere, the version label included.
if v := d.VersionFor("entier"); v == nil || v.Label != "Le Monde" {
t.Errorf("version label %+v, want \"Le Monde\" without the invalid date", v)
}
}
// Two page images with one page number show that page twice; the loader
// says so instead of linking changed pages to whichever sorted first.
func TestLoadPageImagesWarnsOnDuplicatePageNumbers(t *testing.T) {
dir := t.TempDir()
if err := os.MkdirAll(filepath.Join(dir, "images"), 0o755); err != nil {
t.Fatal(err)
}
for _, name := range []string{"page-1.jpg", "page-2.jpg", "page-02.png"} {
if err := os.WriteFile(filepath.Join(dir, "images", name), nil, 0o644); err != nil {
t.Fatal(err)
}
}
before := loadWarnings
imgs := loadPageImages(dir, "doc")
if len(imgs) != 3 || imgs[0] != "page-1.jpg" {
t.Errorf("images %v, want all three in page order", imgs)
}
if loadWarnings != before+1 {
t.Errorf("%d warnings for one duplicated page number, want 1", loadWarnings-before)
}
}

74
tests/main_test.go Normal file
View file

@ -0,0 +1,74 @@
package main
import (
"bytes"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"time"
)
// TestFileWriteBudget: a download slower than the server's WriteTimeout still
// completes, because handleFiles replaces that page-sized budget with
// fileWriteBudget. The control - the same slow download with the budget cut
// short - must be truncated, or the test would not be exercising the deadline
// at all.
func TestFileWriteBudget(t *testing.T) {
const size = 64 << 20 // well past the loopback socket buffers
path := filepath.Join(t.TempDir(), "big.pdf")
if err := os.WriteFile(path, bytes.Repeat([]byte("snowchive"), size/9+1)[:size], 0o644); err != nil {
t.Fatal(err)
}
old := fileWriteBudget
defer func() { fileWriteBudget = old }()
// The same two lines handleFiles uses, over a server whose page budget is
// far shorter than the download takes.
ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_ = http.NewResponseController(w).SetWriteDeadline(time.Now().Add(fileWriteBudget))
http.ServeFile(w, r, path)
}))
ts.Config.WriteTimeout = 300 * time.Millisecond
ts.Start()
defer ts.Close()
// fetch reads 1 MB at a time with a pause after each chunk, returning the
// bytes received before the body ended.
fetch := func(pause time.Duration) (int64, time.Duration) {
start := time.Now()
resp, err := http.Get(ts.URL)
if err != nil {
t.Fatal(err)
}
defer resp.Body.Close()
var got int64
buf := make([]byte, 1<<20)
for {
n, err := io.ReadFull(resp.Body, buf)
got += int64(n)
if err != nil {
return got, time.Since(start)
}
time.Sleep(pause)
}
}
fileWriteBudget = 30 * time.Second
got, elapsed := fetch(40 * time.Millisecond)
if got != size {
t.Errorf("slow download got %d of %d bytes", got, size)
}
if elapsed < 2*ts.Config.WriteTimeout {
t.Errorf("download took %v: too fast to prove it outlasted the %v WriteTimeout",
elapsed, ts.Config.WriteTimeout)
}
fileWriteBudget = 200 * time.Millisecond
if got, _ := fetch(40 * time.Millisecond); got >= size {
t.Errorf("control: a %v budget did not cut the download (got all %d bytes)",
fileWriteBudget, got)
}
}

145
tests/mdparser_test.go Normal file
View file

@ -0,0 +1,145 @@
package main
import "testing"
// Emphasis follows CommonMark's flanking rules, so the loose asterisks and
// underscores of OCR - footnote marks, flattened bullets, star pages,
// form blanks - stay text, and real emphasis nests properly.
func TestParseInlineEmphasis(t *testing.T) {
for in, want := range map[string]string{
// Real emphasis.
"*it* and **bold**": "<em>it</em> and <strong>bold</strong>",
"***bold italic***": "<em><strong>bold italic</strong></em>",
"**2** | **61**": "<strong>2</strong> | <strong>61</strong>",
"An *n*-clique": "An <em>n</em>-clique",
"*a **b** c*": "<em>a <strong>b</strong> c</em>",
"**bold *nested* x**": "<strong>bold <em>nested</em> x</strong>",
"*`code`*": "<em><code>code</code></em>",
"_em_ and __init__.py": "<em>em</em> and <strong>init</strong>.py",
`"On Language,"* a *New York Times*`: "&#34;On Language,&#34;* a <em>New York Times</em>",
// A closer takes the nearest opener, not the first one on the line.
"Inc.*, 321 F.3d; see also *North v. Ashcroft*, 308": "Inc.*, 321 F.3d; see also <em>North v. Ashcroft</em>, 308",
// Not emphasis.
"PINWALE (Sweet* and Sour* partitions)": "PINWALE (Sweet* and Sour* partitions)",
"NRO*, CIA, SCEs* and field sites": "NRO*, CIA, SCEs* and field sites",
"works with: * NCSC * TAO": "works with: * NCSC * TAO",
"*** Currently Inactive ***": "*** Currently Inactive ***",
"a Driving Force ******": "a Driving Force ******",
"2013 WL 5741573, at *3-*9": "2013 WL 5741573, at *3-*9",
"2 * 3 * 4": "2 * 3 * 4",
"snake_case_name": "snake_case_name",
"D/DIR____,IG_____,OGC___": "D/DIR____,IG_____,OGC___",
"SO ORDERED, this _____ day of July": "SO ORDERED, this _____ day of July",
"datastore<br>_ target user<br>_ time": "datastore<br>_ target user<br>_ time",
`\*not em\*`: "*not em*",
} {
if got := parseInline(in); got != want {
t.Errorf("parseInline(%q)\n got %q\nwant %q", in, got, want)
}
}
}
// A list never mixes bullets and numbers: a number kept in a <ul> would
// lose itself to a bullet, a bullet kept in an <ol> would gain a number.
func TestRenderListKindChange(t *testing.T) {
for in, want := range map[string]string{
"1. Fact:\n* South Korea\n* Japan\n\n2. Existence": "<ol>\n<li>Fact:</li>\n</ol>\n" +
"<ul>\n<li>South Korea</li>\n<li>Japan</li>\n</ul>\n" +
"<ol>\n<li value=\"2\">Existence</li>\n</ol>\n",
"* Insider claims\n1. ↑ one\n2. ↑ two": "<ul>\n<li>Insider claims</li>\n</ul>\n" +
"<ol>\n<li>↑ one</li>\n<li>↑ two</li>\n</ol>\n",
// Both nested lists stay inside the item they hang from.
"- a\n - b\n 1. c": "<ul>\n<li>a\n<ul>\n<li>b</li>\n</ul>\n<ol>\n<li>c</li>\n</ol>\n</li>\n</ul>\n",
// Unindented sub-bullets: "5." picks up where "4." left off.
"4. Demonstrate:\n* one\n* two\n5. Thank": "<ol>\n<li value=\"4\">Demonstrate:</li>\n</ol>\n" +
"<ul>\n<li>one</li>\n<li>two</li>\n</ul>\n" +
"<ol>\n<li value=\"5\">Thank</li>\n</ol>\n",
} {
if got := Render(in); got != want {
t.Errorf("Render(%q)\n got %q\nwant %q", in, got, want)
}
}
}
// A number only starts a list right under a line of text when it is a 1
// with content (CommonMark), so OCR runover - a year, a section number, a
// page cite wrapped onto the next line - stays part of the sentence.
func TestRenderRunoverNumbers(t *testing.T) {
for in, want := range map[string]string{
"as added by the FISA Amendments Act of\n2008.": "<p>as added by the FISA Amendments Act of<br>\n2008.</p>\n",
"extended until August\n2013. He will be joined": "<p>extended until August<br>\n2013. He will be joined</p>\n",
"* last visited NSA/CSS in May\n2012.": "<ul>\n<li>last visited NSA/CSS in May<br>\n2012.</li>\n</ul>\n",
"See Alexander Decl. at 39-\n42. While terrorists": "<p>See Alexander Decl. at 39-<br>\n42. While terrorists</p>\n",
"First Events\n+\nLast Events": "<p>First Events<br>\n+<br>\nLast Events</p>\n",
// These still start a list.
"Steps:\n1. one\n2. two": "<p>Steps:</p>\n<ol>\n<li>one</li>\n<li>two</li>\n</ol>\n",
"Steps:\n- one": "<p>Steps:</p>\n<ul>\n<li>one</li>\n</ul>\n",
"\n2008. after a blank": "<ol>\n<li value=\"2008\">after a blank</li>\n</ol>\n",
"5. five\n6.\n7. seven": "<ol>\n<li value=\"5\">five</li>\n<li></li>\n<li>seven</li>\n</ol>\n",
"1. one\n3. gap\n10. ten": "<ol>\n<li>one</li>\n<li value=\"3\">gap</li>\n<li value=\"10\">ten</li>\n</ol>\n",
} {
if got := Render(in); got != want {
t.Errorf("Render(%q)\n got %q\nwant %q", in, got, want)
}
}
}
// "0." is a number like any other; it used to double as "not ordered" and
// render as whatever the running count was.
func TestRenderZeroNumber(t *testing.T) {
for in, want := range map[string]string{
"0. zero\n1. one": "<ol>\n<li value=\"0\">zero</li>\n<li>one</li>\n</ol>\n",
"20. a\n10. b\n0.": "<ol>\n<li value=\"20\">a</li>\n<li value=\"10\">b</li>\n<li value=\"0\"></li>\n</ol>\n",
} {
if got := Render(in); got != want {
t.Errorf("Render(%q)\n got %q\nwant %q", in, got, want)
}
}
// Ten digits is text, not a list item.
if got := Render("1234567890. x"); got != "<p>1234567890. x</p>\n" {
t.Errorf("Render ten-digit number = %q", got)
}
}
// Thematic breaks may space out their characters, as the court filings'
// asterisms do; such a line is a rule, never a bullet.
func TestRenderSpacedRule(t *testing.T) {
for _, in := range []string{"* * *", "- - - - -", "_ _ _", " * *\t* ", "***", "---"} {
if got := Render(in); got != "<hr>\n" {
t.Errorf("Render(%q) = %q, want a rule", in, got)
}
}
if got := Render("1. a\n* * *\n2. b"); got != "<ol>\n<li>a</li>\n</ol>\n<hr>\n<ol>\n<li value=\"2\">b</li>\n</ol>\n" {
t.Errorf("a spaced rule inside a list = %q", got)
}
for _, in := range []string{"* * x", "--", "- -", "*-*"} {
if got := Render(in); got == "<hr>\n" {
t.Errorf("Render(%q) is a rule", in)
}
}
}
// Only the one outer pipe at each end of a row is dropped, so empty edge
// cells keep their columns, and "\|" is a pipe inside a cell.
func TestSplitCells(t *testing.T) {
for in, want := range map[string][]string{
"| a | b |": {"a", "b"},
"||||| ECIs. |": {"", "", "", "", "ECIs."},
"| B. (U) ASSOCIATIONS|||||": {"B. (U) ASSOCIATIONS", "", "", "", ""},
"| | |": {"", ""},
`| Service \| Rights | x |`: {"Service | Rights", "x"},
`| a \|`: {"a |"},
} {
got := splitCells(in)
if len(got) != len(want) {
t.Errorf("splitCells(%q) = %q, want %q", in, got, want)
continue
}
for i := range got {
if got[i] != want[i] {
t.Errorf("splitCells(%q) = %q, want %q", in, got, want)
break
}
}
}
}

54
tests/outlets_test.go Normal file
View file

@ -0,0 +1,54 @@
package main
import (
"strings"
"testing"
)
// A document counts for an outlet in the years of THAT outlet's entries: The
// Guardian's 2013 publication must not put The Intercept's 2014 copy in
// The Intercept's 2013 column, and the cell links a page listing exactly the
// documents it counts.
func TestOutletYearsArePerPublisher(t *testing.T) {
root := assemblyCorpus(t, "")
writeAssemblyDoc(t, root, "shared", map[string]interface{}{
"title": "Shared",
"publication": []map[string]interface{}{
{"publisher": "The Guardian", "date_published": "2013-06-06"},
{"publisher": "The Intercept", "date_published": "2014-02-10"},
{"publisher": "Partner", "date_published": "2012-01-01", "reporting_only": true},
},
}, []string{"page-1.jpg"}, nil)
withCorpus(t, root)
cells := map[string]outletCell{}
table := corpus().outletTables[""]
for _, row := range table.Rows {
for i, y := range table.Years {
cells[row.Name+" "+y] = row.Cells[i]
}
}
for k, want := range map[string]int{
"The Guardian 2013": 1, "The Guardian 2014": 0,
"The Intercept 2013": 0, "The Intercept 2014": 1,
} {
if got := cells[k].Docs; got != want {
t.Errorf("%s: %d documents, want %d", k, got, want)
}
}
for _, y := range table.Years {
if y == "2012" {
t.Errorf("a reporting-only entry made a 2012 column")
}
}
link := cells["The Intercept 2014"].URL
if link != "/browse/publisher/The%20Intercept?year=2014&coll=news&coll=sidtoday-archive&coll=foia" {
t.Errorf("cell link %q", link)
}
if got := resultTotal(t, link); got != 1 {
t.Errorf("%s lists %d documents, want 1", link, got)
}
if body := get(t, "/browse/publisher/The%20Intercept?year=2013").Body.String(); !strings.Contains(body, "0 documents") {
t.Errorf("The Intercept's 2013 page is not empty")
}
}

119
tests/portions_test.go Normal file
View file

@ -0,0 +1,119 @@
package main
import (
"reflect"
"testing"
)
func TestParsePortion(t *testing.T) {
cases := []struct {
in string
canonical string
level string
ok bool
}{
{"U", "U", "U", true},
{"U//FOUO", "U//FOUO", "U", true},
{"TS//SI//NF", "TS//SI//NF", "TS", true},
{"S//REL TO USA, FVEY", "S//REL TO USA, FVEY", "S", true},
// OCR writes the separator every way; one marking, one canonical form.
{"S/NF", "S//NF", "S", true},
{"U/FOUO", "U//FOUO", "U", true},
{"TS/SI//REL", "TS//SI//REL", "TS", true},
{"TS//SI/NF", "TS//SI//NF", "TS", true},
{"TS//STLW//SI//OC/NF", "TS//STLW//SI//OC//NF", "TS", true},
{"TS//SI// ECI SOL", "TS//SI//ECI SOL", "TS", true},
// FOUO alone is a control on unclassified material.
{"FOUO", "U//FOUO", "U", true},
// Spelled out.
{"TOP SECRET//COMINT", "TS//COMINT", "TS", true},
// A level with a part no rule covers: recognised as a marking, but
// not counted - the Explore page lists these separately.
{"S//SI) (DNI collection", "", "S", false},
{"S//WIBBLE", "S//WIBBLE", "S", false},
}
for _, c := range cases {
got := parsePortion(c.in)
if got.Level != c.level {
t.Errorf("parsePortion(%q).Level = %q, want %q", c.in, got.Level, c.level)
}
if got.OK != c.ok {
t.Errorf("parsePortion(%q).OK = %v, want %v", c.in, got.OK, c.ok)
}
if c.ok && got.Canonical != c.canonical {
t.Errorf("parsePortion(%q).Canonical = %q, want %q", c.in, got.Canonical, c.canonical)
}
}
}
// The things that merely look like markings: list enumerators, FOIA
// exemption citations, asides. Each of these appears hundreds of times.
func TestParsePortionRejectsNonMarkings(t *testing.T) {
for _, s := range []string{"1", "2", "a", "b", "iii", "d", "Cont'd.", "continued...",
"Dollars in Thousands", "b)(1), (b)(3", "A", "", " "} {
if got := parsePortion(s); got.Level != "" {
t.Errorf("parsePortion(%q) read a level %q - it is not a marking", s, got.Level)
}
}
}
func TestExtractPortions(t *testing.T) {
text := "" +
"TOP SECRET//COMINT\n" +
"\n" +
"(U) Some Tips on Portion Marking\n" +
"\n" +
"(U//FOUO) The lead-in sentence MUST ALWAYS be portion-marked.\n" +
"* (S//SI) a bullet carrying a complete thought\n" +
"* an unmarked bullet, covered by its lead-in\n" +
"## (TS//SI//NF) A marked heading\n" +
"1. (C) a numbered portion\n" +
"(1) not a marking, a list number\n" +
"text mentioning (U//FOUO) mid-line is not a portion mark\n"
got := make([]string, 0, 5)
for _, m := range extractPortions(text) {
got = append(got, m.Canonical)
}
want := []string{"U", "U//FOUO", "S//SI", "TS//SI//NF", "C"}
if !reflect.DeepEqual(got, want) {
t.Errorf("extractPortions = %v, want %v", got, want)
}
}
// A document quoting a marking is not marked by it. Fenced blocks and block
// quotes are where a transcript puts an example, and they are the curator's
// correction surface: wrap the example, and it stops being counted.
func TestExtractPortionsSkipsQuotedExamples(t *testing.T) {
text := "" +
"(U) Wrong way:\n" +
"\n" +
"> (U) All good men and women should come to the aid of their country through:\n" +
"> * (U//FOUO) volunteering for a tdy to the Middle East\n" +
"\n" +
"```\n" +
"(TS//SI//NF) an example inside a fenced block\n" +
"```\n" +
"(S) a real portion after the block\n"
var got []string
for _, m := range extractPortions(text) {
got = append(got, m.Canonical)
}
want := []string{"U", "S"}
if !reflect.DeepEqual(got, want) {
t.Errorf("extractPortions = %v, want %v (quoted and fenced examples must not count)", got, want)
}
}
// Markings are written in capitals. Court filings enumerate subsections
// "(a)", "(b)", "(c)" at the start of a line, and "(c)" is not CONFIDENTIAL.
func TestParsePortionNeedsCapitals(t *testing.T) {
for _, s := range []string{"c", "s", "u", "ts", "s//si", "Ts", "secret"} {
if got := parsePortion(s); got.Level != "" {
t.Errorf("parsePortion(%q) read level %q from a lower-case label", s, got.Level)
}
}
// The controls after a capital level are read whatever their case.
if got := parsePortion("S//si"); got.Canonical != "S//SI" || !got.OK {
t.Errorf("parsePortion(S//si) = %+v, want S//SI", got)
}
}

153
tests/prefs_matrix_test.go Normal file
View file

@ -0,0 +1,153 @@
package main
// prefs_matrix_test.go - every preference, asserted against the markup it
// is supposed to change. The cookies are documented in prefs.go and their
// effects are spread across four templates; before this, nothing checked
// that setting one did anything at all.
import (
"net/http"
"strings"
"testing"
)
// prefCase is one preference: the cookie that turns it on, a page that
// shows it, and a marker that must be present in exactly one of the two
// states. present says which state the marker belongs to.
type prefCase struct {
name string
cookie string
value string
path string
marker string
present bool // true: the marker appears when the cookie is SET
}
func TestPreferenceMatrix(t *testing.T) {
buildFixture(t)
cases := []prefCase{
// --- card display switches (browse lists, search results) -------
{"hide thumbnails", "pref_hide_thumbs", "1", "/browse", `class="thumb-container"`, false},
{"hide thumbnails (search)", "pref_hide_thumbs", "1", "/search?q=prism&do_search=1", `class="result-thumb"`, false},
{"hide summaries", "pref_hide_summary", "1", "/browse", `class="card-summary"`, false},
{"hide classification", "pref_hide_classification", "1", "/browse", `class="doc-classification"`, false},
{"hide programs", "pref_hide_programs", "1", "/browse", `class="doc-programs"`, false},
{"hide people", "pref_hide_people", "1", "/browse", `class="doc-people"`, false},
{"hide journalists", "pref_hide_journalists", "1", "/browse", `class="doc-journalist`, false},
{"page preview hook", "pref_page_preview", "1", "/browse", "data-pp-id=", true},
// --- category tables --------------------------------------------
{"collection columns", "pref_show_coll_cols", "1", "/browse/program", `id="coll-cols" class="coll-cols-checkbox" checked`, true},
// --- the document page ------------------------------------------
{"info sidebar", "pref_info_position", "beside", "/doc/prism-slides", `class="doc-layout info-sidebar"`, true},
{"scroll panes", "pref_scroll_doc", "1", "/doc/prism-slides", "scroll-panes", true},
{"expand more info", "pref_open_info", "1", "/doc/prism-slides", `id="meta-toggle" class="meta-checkbox" checked`, true},
{"programs one per line", "pref_programs_list", "1", "/doc/prism-slides", `<ul class="meta-value-list">`, true},
{"tags one per line", "pref_tags_list", "1", "/doc/prism-slides", `<ul class="meta-value-list">`, true},
{"collapse published-together", "pref_collapse_pub", "1", "/doc/prism-slides", `<details class="collapse-section" open>`, false},
// --- the viewport picker ----------------------------------------
{"default view: text", "pref_doc_view", "text", "/doc/prism-slides", `id="view-text" aria-controls="doc-viewport" checked`, true},
{"default view: split", "pref_doc_view", "split", "/doc/prism-slides", `id="view-split" aria-controls="doc-viewport" checked`, true},
// --- theme -------------------------------------------------------
{"dark theme", "pref_theme", "dark", "/", `data-theme="dark"`, true},
{"light theme", "pref_theme", "light", "/browse", `data-theme="light"`, true},
}
for _, c := range cases {
off := get(t, c.path).Body.String()
on := get(t, c.path, &http.Cookie{Name: c.cookie, Value: c.value}).Body.String()
hasOff, hasOn := strings.Contains(off, c.marker), strings.Contains(on, c.marker)
wantOn, wantOff := c.present, !c.present
if hasOn != wantOn || hasOff != wantOff {
t.Errorf("%s (%s=%s on %s): marker %q present without the cookie=%v, with it=%v; wanted %v / %v",
c.name, c.cookie, c.value, c.path, c.marker, hasOff, hasOn, wantOff, wantOn)
}
}
}
// Text-only mode is the archive's promise to readers on Tor and metered
// links: NO page image, thumbnail or preview, anywhere, whatever else is
// set. It also overrides the thumbnail preference rather than being
// overridden by it.
func TestPreferenceTextOnly(t *testing.T) {
buildFixture(t)
textOnly := &http.Cookie{Name: "pref_text_only", Value: "1"}
thumbsOn := &http.Cookie{Name: "pref_hide_thumbs", Value: "0"}
preview := &http.Cookie{Name: "pref_page_preview", Value: "1"}
for _, p := range []string{"/", "/browse", "/browse?collection=sidtoday-archive&view=card",
"/doc/prism-slides", "/doc/prism-slides/pages", "/doc/prism-slides/versions/lemonde",
"/search?q=prism&do_search=1", "/pinned", "/articles"} {
body := get(t, p, textOnly, thumbsOn, preview).Body.String()
if i := strings.Index(body, "<img"); i >= 0 {
t.Errorf("%s in text-only mode emits an image: %.80s", p, body[i:])
}
if strings.Contains(body, "data-pp-imgs") {
t.Errorf("%s in text-only mode emits the page-preview payload", p)
}
}
// The document page says so and offers the pages as links instead.
doc := get(t, "/doc/prism-slides", textOnly).Body.String()
for _, want := range []string{"Text-only mode is on", `class="page-link-list"`} {
if !strings.Contains(doc, want) {
t.Errorf("the document page in text-only mode lacks %q", want)
}
}
// Without it, that same page does show images.
if !strings.Contains(get(t, "/doc/prism-slides").Body.String(), "<img") {
t.Error("the document page shows no images even with images on")
}
}
// Documents per page drives the list length and the pagination, on browse
// and on search alike.
func TestPreferencePerPage(t *testing.T) {
buildFixture(t)
count := func(body string) int { return strings.Count(body, `class="doc-card"`) }
all := count(get(t, "/browse", &http.Cookie{Name: "pref_per_page", Value: "all"}).Body.String())
if all < 2 {
t.Fatalf("the fixture's news collection has %d cards; the test needs at least 2", all)
}
one := get(t, "/browse", &http.Cookie{Name: "pref_per_page", Value: "10"}).Body.String()
if n := count(one); n > 10 {
t.Errorf("per_page=10 rendered %d cards", n)
}
// An unknown value falls back to the default rather than to everything.
def := count(get(t, "/browse", &http.Cookie{Name: "pref_per_page", Value: "7"}).Body.String())
if def > 20 {
t.Errorf("an invalid per_page rendered %d cards, want at most the default 20", def)
}
}
// The two history preferences are opt-in: nothing is recorded until they
// are on, and turning one off deletes what it recorded.
func TestPreferenceHistoryOptIn(t *testing.T) {
buildFixture(t)
// Off by default: viewing a document sets no history cookie.
w := get(t, "/doc/prism-slides")
for _, c := range w.Result().Cookies() {
if c.Name == recentDocsCookie && c.Value != "" {
t.Errorf("a document view recorded history without the preference: %v", c)
}
}
// On: the view is recorded.
w = get(t, "/doc/prism-slides", &http.Cookie{Name: "pref_track_recent", Value: "1"})
found := false
for _, c := range w.Result().Cookies() {
if c.Name == recentDocsCookie && strings.Contains(c.Value, "prism-slides") {
found = true
}
}
if !found {
t.Error("with tracking on, a document view was not recorded")
}
// A page showing history must never be stored by a shared cache.
hist := get(t, "/", &http.Cookie{Name: "pref_track_recent", Value: "1"},
&http.Cookie{Name: recentDocsCookie, Value: "prism-slides"})
if cc := hist.Header().Get("Cache-Control"); !strings.Contains(cc, "no-store") {
t.Errorf("the home page showing history: Cache-Control %q", cc)
}
}

117
tests/prefs_test.go Normal file
View file

@ -0,0 +1,117 @@
package main
import (
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
)
func reqWithCookies(pairs ...string) *http.Request {
r := httptest.NewRequest("GET", "/", nil)
for i := 0; i+1 < len(pairs); i += 2 {
r.AddCookie(&http.Cookie{Name: pairs[i], Value: pairs[i+1]})
}
return r
}
// Defaults apply when a cookie is absent, and a value the preference does
// not accept is a value it does not take.
func TestReadPrefsValues(t *testing.T) {
p := readPrefs(reqWithCookies())
if p.DocView != "images" || p.Theme != "auto" || p.PerPage != 20 || p.InfoSidebar {
t.Errorf("defaults wrong: %+v", p)
}
if !p.ShowThumbs || !p.ShowClassification || p.TrackRecent {
t.Errorf("boolean defaults wrong: %+v", p)
}
p = readPrefs(reqWithCookies(
"pref_doc_view", "split",
"pref_theme", "dark",
"pref_per_page", "all",
"pref_info_position", "beside",
"pref_hide_thumbs", "1",
"pref_track_recent", "1",
))
if p.DocView != "split" || p.Theme != "dark" || p.PerPage != 0 || !p.InfoSidebar {
t.Errorf("cookie values not applied: %+v", p)
}
if p.ShowThumbs || !p.TrackRecent {
t.Errorf("boolean cookies not applied: %+v", p)
}
if p.ThemeAttr() != "dark" {
t.Errorf("ThemeAttr = %q", p.ThemeAttr())
}
// Junk in the jar: every value pref falls back to its default.
p = readPrefs(reqWithCookies(
"pref_doc_view", "sideways",
"pref_theme", "neon",
"pref_per_page", "7",
"pref_info_position", "under",
))
if p.DocView != "images" || p.Theme != "auto" || p.PerPage != 20 || p.InfoSidebar {
t.Errorf("junk cookies not rejected: %+v", p)
}
if p.ThemeAttr() != "" {
t.Errorf("auto must render no data-theme, got %q", p.ThemeAttr())
}
}
// Saving writes the cookies a non-default choice needs and clears the rest,
// so a default-settings visitor stays cookieless (and their pages stay
// publicly cacheable).
func TestPreferencesSaveRoundTrip(t *testing.T) {
form := url.Values{
"doc_view": {"text"}, "theme": {"light"}, "per_page": {"100"},
"info_position": {"beside"}, "hide_thumbs": {"1"},
}
r := httptest.NewRequest("POST", "/preferences/save", strings.NewReader(form.Encode()))
r.Header.Set("Content-Type", "application/x-www-form-urlencoded")
r.Header.Set("Sec-Fetch-Site", "same-origin")
w := httptest.NewRecorder()
handlePreferencesSave(w, r)
set := map[string]string{}
for _, c := range w.Result().Cookies() {
set[c.Name] = c.Value
if c.MaxAge < 0 {
set[c.Name] = "" // deleted
}
}
for name, want := range map[string]string{
"pref_doc_view": "text", "pref_theme": "light", "pref_per_page": "100",
"pref_info_position": "beside", "pref_hide_thumbs": "1",
"pref_track_recent": "", // unchecked: deleted
} {
if got, ok := set[name]; !ok || got != want {
t.Errorf("%s = %q (present %v), want %q", name, got, ok, want)
}
}
// Defaults delete rather than store: nothing in the jar means nothing
// to vary the cache on.
form = url.Values{"doc_view": {"images"}, "theme": {"auto"}, "per_page": {"20"}, "info_position": {"above"}}
r = httptest.NewRequest("POST", "/preferences/save", strings.NewReader(form.Encode()))
r.Header.Set("Content-Type", "application/x-www-form-urlencoded")
r.Header.Set("Sec-Fetch-Site", "same-origin")
w = httptest.NewRecorder()
handlePreferencesSave(w, r)
for _, c := range w.Result().Cookies() {
if c.MaxAge >= 0 {
t.Errorf("default settings wrote cookie %s=%q", c.Name, c.Value)
}
}
}
// "All" per page is every card up to maxCardsPerPage: a list longer than
// that is paged, however large the reader's page size.
func TestCardsPerPage(t *testing.T) {
for perPage, want := range map[int]int{0: maxCardsPerPage, 20: 20, 100: 100, maxCardsPerPage + 1: maxCardsPerPage} {
if got := (Prefs{PerPage: perPage}).cardsPerPage(); got != want {
t.Errorf("PerPage %d: %d cards a page, want %d", perPage, got, want)
}
}
}

126
tests/probes_test.go Normal file
View file

@ -0,0 +1,126 @@
package main
// probes_test.go - the instruments that catch what reading the code cannot:
// data races, the parameter space behind each route, and header injection.
// Each of these exists because a review that only read the source missed
// something a probe found in seconds.
import (
"fmt"
"net/http"
"net/http/httptest"
"os"
"sort"
"strings"
"sync"
"testing"
"time"
)
// Concurrency: the archive claims to be read-only while serving. Hammer the
// handlers from many goroutines with -race and see.
func TestProbeConcurrent(t *testing.T) {
if os.Getenv("SNOWCHIVE_BENCH_DATA") == "" {
buildFixture(t)
}
paths := []string{"/", "/browse", "/browse/program", "/browse/tag?sort=count",
"/search?q=prism&do_search=1", "/explore/ids", "/api/facets", "/glossary",
"/doc/prism-slides", "/articles", "/sitemap.xml"}
mux, _ := buildFixture(t)
var wg sync.WaitGroup
for i := 0; i < 24; i++ {
wg.Add(1)
go func(n int) {
defer wg.Done()
for j := 0; j < 20; j++ {
p := paths[(n+j)%len(paths)]
r := httptest.NewRequest("GET", p, nil)
if j%3 == 0 {
r.AddCookie(&http.Cookie{Name: "pref_theme", Value: "dark"})
}
w := httptest.NewRecorder()
mux.ServeHTTP(w, r)
if w.Code != 200 {
t.Errorf("%s: %d", p, w.Code)
return
}
}
}(i)
}
wg.Wait()
}
// Input space: every route crossed with the parameters it accepts. The
// 80 ms page hid behind ?sort=count - a path-only sweep never saw it.
// Input space: every route crossed with the parameters it accepts. The
// 80 ms page hid behind ?sort=count - a path-only sweep never saw it.
func TestProbeParamSweep(t *testing.T) {
buildFixture(t)
base := []string{"/", "/browse", "/browse/program", "/browse/tag", "/browse/docdate",
"/browse/program/PRISM", "/browse/docdate/2013", "/search", "/explore/ids",
"/explore/terms", "/explore/pages", "/explore/outlets", "/articles",
"/api/docs", "/api/search", "/glossary", "/pinned", "/preferences"}
params := []string{"", "?sort=count", "?sort=count&order=asc", "?sort=zzz", "?order=desc",
"?view=table", "?view=card", "?p=2", "?p=-1", "?p=99999999999999999999",
"?per_page=all", "?per_page=0", "?per_page=-5", "?type=REPORTED", "?type=NOPE",
"?coll=news", "?coll=news&coll=foia", "?missing=1", "?q=prism&do_search=1",
"?q=%22a+b%22~5&do_search=1", "?q=-prism&do_search=1", "?hist=foia,sidtoday-archive",
"?date_doc_start=2013&date_doc_end=2013", "?fqv=x&fqf=ZZ&fqc=xx&do_search=1",
"?sort=name&order=desc&p=1&type=FOIA&coll=foia"}
type slow struct {
path string
d time.Duration
}
var slows []slow
bad := 0
for _, b := range base {
for _, p := range params {
u := b + p
start := time.Now()
w := get(t, u)
d := time.Since(start)
if w.Code != 200 && w.Code != 400 && w.Code != 404 {
bad++
t.Errorf("%s: status %d", u, w.Code)
}
slows = append(slows, slow{u, d})
}
}
sort.Slice(slows, func(i, j int) bool { return slows[i].d > slows[j].d })
fmt.Printf("%d combinations, %d unexpected statuses; slowest on the fixture:\n", len(slows), bad)
for _, s := range slows[:5] {
fmt.Printf(" %8s %s\n", s.d.Round(time.Microsecond), s.path)
}
}
// Header injection: the Onion-Location header echoes the request URI.
// Header injection: the Onion-Location header echoes the request URI.
func TestProbeHeaderInjection(t *testing.T) {
buildFixture(t)
for _, raw := range []string{
"/browse?x=1%0d%0aX-Injected:+yes",
"/browse?x=%0aSet-Cookie:+a=b",
"/doc/" + strings.Repeat("a", 200),
} {
r := httptest.NewRequest("GET", raw, nil)
w := httptest.NewRecorder()
mux, _ := buildFixture(t)
mux.ServeHTTP(w, r)
h := w.Header()
if h.Get("X-Injected") != "" || h.Get("Set-Cookie") != "" {
t.Errorf("%s: header injected: %v", raw, h)
}
for k, vs := range h {
for _, v := range vs {
if strings.ContainsAny(v, "\r\n") {
t.Errorf("%s: header %s carries a newline: %q", raw, k, v)
}
}
}
}
}
// Pathological documents: the biggest thing in the corpus is what breaks
// first. What IS the biggest thing?

View file

@ -0,0 +1,43 @@
package main
import (
"encoding/json"
"testing"
)
// The "published in the same report" box is capped at pubTogetherCap, and
// the overflow links point at the article pages: one per article that holds
// other documents, in publication order, so each link's number is the
// number its page lists.
func TestDocArticles(t *testing.T) {
saved := corpus().articleByLink
defer func() { corpus().articleByLink = saved }()
small := &Article{Key: "aaaaaaaaaaaa", DocIDs: []string{"a", "b"}}
big := &Article{Key: "bbbbbbbbbbbb", DocIDs: []string{"a", "b", "c", "d"}}
alone := &Article{Key: "cccccccccccc", DocIDs: []string{"a"}}
corpus().articleByLink = map[string]*Article{
"https://small.example/story": small,
"https://big.example/appendix": big,
"https://alone.example/x": alone,
}
var doc Document
if err := json.Unmarshal([]byte(`{"publication":[
{"link": "https://small.example/story"},
{"link": "https://alone.example/x"},
{"link": "https://big.example/appendix"},
{"link": "https://small.example/story"}]}`), &doc); err != nil {
t.Fatal(err)
}
got := corpus().docArticles(&doc)
if len(got) != 2 || got[0] != small || got[1] != big {
t.Fatalf("docArticles = %+v, want the two articles with other documents, once each, in order", got)
}
var orphan Document
if err := json.Unmarshal([]byte(`{"publication":[{"link": "https://nowhere.example/x"}]}`), &orphan); err != nil {
t.Fatal(err)
}
if a := corpus().docArticles(&orphan); a != nil {
t.Error("a document whose links have no article must produce no overflow link")
}
}

193
tests/query_test.go Normal file
View file

@ -0,0 +1,193 @@
package main
// query_test.go - what the search engine promises, as queries and the
// documents they must return. The unit tests cover the pieces (tokenizer,
// wildcards, ranges); this covers the sentence a reader types.
//
// Results come from /api/search, which runs the same engine as the page, so
// a refactor that changes what search FINDS fails here even when every page
// still renders.
import (
"encoding/json"
"net/url"
"sort"
"strings"
"testing"
)
func searchIDs(t *testing.T, q string) []string {
t.Helper()
w := get(t, "/api/search?per_page=all&q="+url.QueryEscape(q))
if w.Code != 200 {
t.Fatalf("%q: status %d", q, w.Code)
}
var body struct {
Results []struct {
ID string `json:"id"`
} `json:"results"`
}
if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil {
t.Fatalf("%q: %v", q, err)
}
ids := make([]string, 0, len(body.Results))
for _, r := range body.Results {
ids = append(ids, r.ID)
}
sort.Strings(ids)
return ids
}
func TestQuerySuite(t *testing.T) {
buildFixture(t)
const (
prism = "prism-slides"
companion = "prism-companion"
sid = "2003-05-29_SIDToday_-_Fixture_Issue"
foia = "FISC_-_fixture_order"
undated = "undated-fixture"
)
cases := []struct {
query string
want []string
why string
}{
// --- words, stems and case --------------------------------------
{"xkeyscore", []string{prism}, "a plain word in the transcript"},
{"XKEYSCORE", []string{prism}, "matching ignores case"},
{"collections", []string{prism, sid}, "stemming: plural finds the singular"},
{"analysts", []string{prism, sid}, "plural finds singular"},
// The stemmer deliberately leaves -tion alone (see stem), so
// "collecting" and "collection" are different words here. This is
// the behaviour, recorded so a change to the stemmer is a decision
// rather than a surprise.
{"collecting", []string{prism}, "-ing does not reach -tion"},
{"zzzznotacorpusword", nil, "a word nobody wrote"},
// --- phrases and proximity ---------------------------------------
{`"upstream collection"`, []string{prism}, "a quoted phrase is adjacency, and SIDtoday has the words apart"},
{`"upstream collection"~12`, []string{prism, sid}, "proximity widens it to a window"},
{`"cable tapping"`, []string{prism}, "a phrase that is also a tag"},
// --- punctuation and accents --------------------------------------
{"RT-RG", []string{prism}, "a hyphenated term matches its parts in sequence"},
{"muller", []string{prism}, "accent folding: muller finds Müller-Maguhn"},
{"Müller-Maguhn", []string{prism}, "and the accented form finds itself"},
{"sao paulo", []string{prism}, "folding works across a phrase too"},
// --- wildcards -----------------------------------------------------
{"*keyscore", []string{prism}, "a leading wildcard"},
{"tapp*", []string{prism, sid}, "a trailing wildcard reaches tapping"},
{"*ollectio*", []string{prism, sid}, "both ends open"},
// --- exact values vs substrings -------------------------------------
{`classification:="TOP SECRET"`, []string{prism}, "an exact value is the whole value"},
{`classification:="SECRET"`, []string{foia}, "so SECRET does not match TOP SECRET"},
{"classification:secret", []string{foia, prism}, "unquoted, it is a word match on both"},
// --- fielded search ---------------------------------------------------
{"program:PRISM", []string{companion, prism}, "a metadata field"},
{"agency:FISC", []string{foia}, "creator"},
{"agency:NSA", []string{companion, prism, sid}, "creator across collections"},
{`journalist:"Greenwald, Glenn"`, []string{companion, prism}, "a quoted byline"},
{"publisher:NRK", nil, "a reporting-only partner is not a publisher"},
{"publisher:\"The Guardian\"", []string{companion, prism}, "the outlet that published it"},
{"tag:collection", []string{prism}, "tags are their own field"},
{"country:Brazil", []string{prism}, "countries"},
{"people:Snowden", []string{prism}, "people, by surname"},
{"type:FOIA", []string{foia}, "document type"},
{"series:\"(U) A Day in the Life of...\"", []string{sid}, "a series name with punctuation"},
// --- boolean and negation ---------------------------------------------
{"prism -companion", []string{prism}, "inline negation excludes"},
{"upstream -xkeyscore", []string{sid}, "negation against a word in the other document"},
{"prism -x", []string{companion, prism}, "a single letter is a literal word, not an empty clause that excludes everything"},
{"u.s.", nil, "a dotted abbreviation is matched as written, not dropped"},
// --- a phrase stays inside one value ------------------------------------
{`"united states"`, []string{prism}, "a phrase inside one country"},
{`"andy united"`, nil, "a phrase cannot run from one metadata value into the next"},
{`"slides slides describing"`, nil, "nor from the title into the description"},
{"program:PRISM tag:collection", []string{prism}, "two fields are an AND"},
// --- dates -------------------------------------------------------------
{"date:2013", []string{companion, foia, prism}, "document year, across collections"},
{"date:2013-04-02", []string{companion}, "an exact document date"},
{"date:2003..2013", []string{companion, foia, prism, sid}, "a range"},
{"published:2016", []string{sid}, "publication year, not document year"},
// --- the corner the archive cares about ---------------------------------
{"fairview", []string{prism}, "text that exists only in another published copy"},
{"guardian", []string{companion, prism}, "an outlet name is searchable as text"},
{"prism-slides", []string{prism}, "the document id itself"},
{"prism-slides-guardian", []string{prism}, "and the name the same file was published under"},
}
for _, c := range cases {
got := searchIDs(t, c.query)
want := append([]string(nil), c.want...)
sort.Strings(want)
if strings.Join(got, ",") != strings.Join(want, ",") {
t.Errorf("%-34s got %v, want %v (%s)", c.query, got, want, c.why)
}
}
}
// Relevance is not just membership: a title hit must outrank a body hit,
// and the phrase bonus must beat loose words.
func TestQueryRanking(t *testing.T) {
buildFixture(t)
first := func(q string) string {
ids := func() []string {
w := get(t, "/api/search?q="+url.QueryEscape(q))
var body struct {
Results []struct {
ID string `json:"id"`
} `json:"results"`
}
json.Unmarshal(w.Body.Bytes(), &body)
out := make([]string, 0, len(body.Results))
for _, r := range body.Results {
out = append(out, r.ID)
}
return out
}()
if len(ids) == 0 {
t.Fatalf("%q returned nothing", q)
}
return ids[0]
}
if got := first("collection"); got != "prism-slides" {
t.Errorf("collection: first result %s, want the document with it in the title and the tags", got)
}
if got := first("fixture"); got == "" {
t.Error("fixture: no results")
}
}
// TestQueryRowParts: a structured row is read as ?q= reads its text - each
// quoted phrase is a part of its own and the bare words are one more, all
// ANDed. A row with two phrases used to be one substring search for the
// text between the outer quotes, quotes included, and found nothing.
func TestQueryRowParts(t *testing.T) {
buildFixture(t)
for _, c := range []struct {
row string
want int
}{
{`"upstream collection" "cable tapping"`, 1},
{`"upstream collection" xkeyscore`, 1},
{`"collection upstream" xkeyscore`, 0},
} {
w := get(t, "/api/search?per_page=all&fqf=TE&fqc=and&fqv="+url.QueryEscape(c.row))
var body struct {
Total int `json:"total"`
}
if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil {
t.Fatalf("%s: %v", c.row, err)
}
if body.Total != c.want {
t.Errorf("row %s: %d results, want %d", c.row, body.Total, c.want)
}
}
}

77
tests/related_test.go Normal file
View file

@ -0,0 +1,77 @@
package main
import (
"fmt"
"strings"
"testing"
)
// The regression this guards: a SIDtoday piece called "The Life of an Exec"
// used to list five FISC court filings as related, because their titles say
// "of" eight times and every occurrence scored like a shared codename. A
// shared word is worth only as much as it is rare, and it counts once.
func TestRelatedIgnoresFillerTitleWords(t *testing.T) {
filler := strings.Repeat("Motion of the Court for Disclosure of Records of the Court of Appeal ", 3)
docs := []Document{
{ID: "exec", Title: "The Life of an Exec (repost)", Collection: "sidtoday-archive"},
{ID: "sibling", Title: "A Day in the Life of an Exec", Collection: "sidtoday-archive"},
{ID: "filing", Title: filler, Collection: "foia"},
}
// Enough other titles saying "of" and "court" that those words are
// common - which is exactly what the real corpus looks like.
for i := 0; i < 30; i++ {
docs = append(docs, Document{
ID: fmt.Sprintf("noise-%02d", i),
Title: fmt.Sprintf("Order of the Court of Review number %d", i),
Collection: "foia",
})
}
got := corpus().buildRelatedDocs(docs[:1], docs)["exec"]
if len(got) == 0 || got[0] != "sibling" {
t.Fatalf("related = %v, want sibling first", got)
}
for _, id := range got {
if id == "filing" || strings.HasPrefix(id, "noise-") {
t.Errorf("related = %v, want no documents matched only on filler words", got)
break
}
}
}
// Shared metadata counts each distinct value once: a tag listed twice in one
// document is one tag, not two matches.
func TestRelatedScoreCountsEachValueOnce(t *testing.T) {
a := Document{ID: "a", Title: "Alpha", Tags: []string{"PRISM", "prism", "SSO"}}
b := Document{ID: "b", Title: "Beta", Tags: []string{"PRISM", "PRISM"}}
if got := corpus().relatedScore(&a, &b, titleWeights{}); got != 2.0 {
t.Errorf("score = %v, want 2 (one shared tag)", got)
}
// A shared program, tag, agency and country each still count.
a.DocMeta.Programs, a.DocMeta.Creator, a.DocMeta.Countries = []string{"XKEYSCORE"}, "NSA", []string{"Norway"}
b.DocMeta.Programs, b.DocMeta.Creator, b.DocMeta.Countries = []string{"xkeyscore"}, "nsa", []string{"NORWAY", "Norway"}
if got := corpus().relatedScore(&a, &b, titleWeights{}); got != 2.0+4.0+1.0+0.5 {
t.Errorf("score = %v, want 7.5", got)
}
}
// A word in most titles is worth almost nothing; one in a couple of titles is
// worth nearly full marks.
func TestTitleWeights(t *testing.T) {
tw := titleWeights{n: 1000, df: map[string]int{"of": 800, "exec": 2}}
if w := tw.weight("of"); w > 0.05 {
t.Errorf("weight(of) = %v, want near 0", w)
}
if w := tw.weight("exec"); w < 0.8 {
t.Errorf("weight(exec) = %v, want near 1", w)
}
// An unknown word is treated as unique, and a corpus too small to have
// frequencies weighs everything equally.
if w := tw.weight("never-seen"); w < 0.99 {
t.Errorf("weight(unseen) = %v, want ~1", w)
}
if w := (titleWeights{}).weight("anything"); w != 1 {
t.Errorf("empty weights = %v, want 1", w)
}
}

View file

@ -0,0 +1,49 @@
package main
import (
"encoding/json"
"fmt"
"strings"
"testing"
)
// A reporting-only partner keeps its entry, but is not a source of the file:
// publisher: must not match it and the source category must not list it.
func TestReportingOnlyIsNotASource(t *testing.T) {
var d Document
if err := json.Unmarshal([]byte(`{"publication": [
{"publisher": "The Intercept", "title": "Norway Used NSA Technology",
"date_published": "2018-03-01", "reporters": ["Gallagher, Ryan"]},
{"publisher": "NRK", "title": "Antennene som samler inn data",
"date_published": "2018-03-01", "reporters": ["Hopland, Standahl"],
"reporting_only": true}
]}`), &d); err != nil {
t.Fatal(err)
}
if got := fmt.Sprint(facetValues(&d, "PU")); got != "[The Intercept]" {
t.Errorf("publisher values %s, want only [The Intercept]", got)
}
// The /browse/publisher/{outlet} index is built from the same values, so
// the page and the facet agree: the publisher is a source, the partner
// that only reported the story is not. The index is package state, so
// put the corpus back for whatever runs next.
prev := corpus().archive.Docs
t.Cleanup(func() { corpus().archive.Docs = prev; corpus().buildCategoryDocs() })
corpus().archive.Docs = []Document{d}
corpus().buildCategoryDocs()
if got := len(corpus().categoryMatches("publisher", "The Intercept")); got != 1 {
t.Errorf("the publishing outlet's page lists %d documents, want 1", got)
}
if got := len(corpus().categoryMatches("publisher", "NRK")); got != 0 {
t.Errorf("the reporting-only partner's page lists %d documents, want 0", got)
}
// The entry itself, its byline and its article survive.
if len(d.Publication) != 2 || d.Publication[1].Publisher != "NRK" ||
len(d.Publication[1].Journalists) != 1 {
t.Errorf("the reporting-only entry lost data: %+v", d.Publication)
}
if !strings.Contains(fmt.Sprint(d.JournalistsDeduped()), "Hopland") {
t.Error("a partner's reporters should still be credited")
}
}

705
tests/routes_test.go Normal file
View file

@ -0,0 +1,705 @@
package main
import (
"fmt"
"html/template"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"regexp"
"strings"
"testing"
)
// Every route, through the real mux: the status it must answer, and a
// fragment of the body that proves the page is the one it should be. The
// handlers had no coverage at all before this - a missing template
// function, a nil map or a renamed data key reached production as a 500.
func TestRoutes(t *testing.T) {
cases := []struct {
path string
code int
want string // substring of the body
}{
{"/", 200, "Snowden"},
{"/browse", 200, "Reported"},
{"/browse?collection=foia", 200, "FOIA"},
{"/browse?collection=sidtoday-archive", 200, "SIDtoday Archive"},
{"/browse?collection=sidtoday-archive&view=card", 200, "SIDtoday Archive"},
{"/browse?collection=sidtoday-archive&view=table", 200, "SIDtoday Archive"},
{"/browse?collection=bogus", 200, "Reported"}, // unknown collection falls back
{"/browse/program", 200, "PRISM"},
{"/browse/program/PRISM", 200, "PRISM Collection Slides"},
{"/browse/program/NOSUCHTHING", 200, "0 document"},
{"/browse/tag/PRISM", 200, "PRISM Collection Slides"},
{"/browse/agency/NSA", 200, "PRISM Collection Slides"},
{"/browse/publisher/The%20Guardian", 200, "PRISM Collection Slides"},
{"/browse/publisher/NRK", 200, "0 document"}, // reporting-only: not a publisher
{"/browse/journalist/Greenwald%2C%20Glenn", 200, "PRISM"},
{"/browse/classification/TOP%20SECRET", 200, "PRISM"},
{"/browse/dissemination/NOFORN", 200, "PRISM"},
{"/browse/docdate/2013", 200, "PRISM"},
{"/browse/docdate/2013-04", 200, "PRISM"},
{"/browse/docdate/20", 200, "0 document"}, // not a valid date prefix
{"/browse/published/2013", 200, "PRISM"},
{"/browse/series", 200, "A Day in the Life"},
{"/browse/nosuchcategory", 404, ""},
{"/browse/nosuchcategory/value", 404, ""},
{"/doc/prism-slides", 200, "PRISM Collection Slides"},
{"/doc/prism-slides/pages", 200, "All pages"},
{"/doc/prism-slides/versions/lemonde", 200, "lemonde"},
{"/doc/prism-slides/versions/lemonde/pages", 200, "lemonde"},
{"/doc/prism-slides/versions/nope", 404, ""},
{"/doc/broken-no-pdf", 404, ""}, // skipped at load: no PDF
{"/doc/nope", 404, ""},
{"/search", 200, "Search"},
{"/search?q=prism&do_search=1", 200, "Collection Slides"}, // the term itself is <mark>ed
{"/search?q=zzzznomatch&do_search=1", 200, "No matches"},
{"/search?q=fairview&do_search=1", 200, "Copy:"}, // matched in a version
{"/articles", 200, "NSA Prism program"},
{"/explore", 200, "Explore"},
{"/explore/pages", 200, "Longest"},
{"/explore/ids", 200, "prism-slides"},
{"/explore/ids.txt", 200, "prism-slides.pdf"},
{"/explore/ids?missing=1", 200, "Technical IDs"},
{"/explore/outlets", 200, "Guardian"},
{"/explore/terms", 200, "PRISM"},
{"/explore/classification", 200, "Classification"},
{"/explore/nosuchmetric", 404, ""},
{"/glossary", 200, "XKEYSCORE"},
{"/changelog", 200, ""},
{"/about", 200, "About"},
{"/preferences", 200, "Preferences"},
{"/pinned", 200, "Pinned"},
{"/pinned/export", 200, ""},
{"/random", 302, ""},
{"/robots.txt", 200, "Sitemap"},
{"/sitemap.xml", 200, "/doc/prism-slides"},
{"/feed.xml", 404, ""},
{"/api/docs", 200, `"total"`},
{"/api/docs?per_page=all", 200, `"docs"`},
{"/api/doc/prism-slides", 200, `"full_text"`},
{"/api/doc/nope", 404, `"error"`},
{"/api/search?q=prism", 200, `"results"`},
{"/api/search", 400, `"error"`},
{"/api/facets", 200, `"programs"`},
{"/api/nosuch", 404, `"error"`},
{"/static/style.css", 200, "--"},
{"/static/nosuch.css", 404, ""},
{"/files/prism-slides/prism-slides.pdf", 200, "%PDF"},
{"/nosuchpage", 404, ""},
}
for _, c := range cases {
w := get(t, c.path)
if w.Code != c.code {
t.Errorf("%s: status %d, want %d", c.path, w.Code, c.code)
continue
}
if c.want != "" && !strings.Contains(w.Body.String(), c.want) {
t.Errorf("%s: body does not contain %q", c.path, c.want)
}
}
}
// The document file handler is the one place a URL reaches the filesystem.
func TestFileHandlerContainment(t *testing.T) {
for _, path := range []string{
"/files/prism-slides/images/page-1.jpg",
"/files/prism-slides/versions/lemonde/lemonde.pdf",
"/files/prism-slides/versions/lemonde/images/page-1.jpg",
} {
if w := get(t, path); w.Code != 200 {
t.Fatalf("%s: a file of the document should serve, got %d", path, w.Code)
}
}
// A file the loader never indexed is not served, whatever its type: a
// document directory gathers backups and second copies.
_, root := buildFixture(t)
dir := filepath.Join(root, "prism-slides")
for _, stray := range []string{"zz-stray.pdf", "images/notes/scan.jpg", "old-versions/draft.pdf"} {
writeFixtureFile(t, filepath.Join(dir, filepath.FromSlash(stray)), "%PDF-1.4\n")
}
t.Cleanup(func() {
for _, stray := range []string{"zz-stray.pdf", "images/notes", "old-versions"} {
os.RemoveAll(filepath.Join(dir, stray))
}
})
for _, path := range []string{
"/files/prism-slides/zz-stray.pdf", // a second PDF
"/files/prism-slides/images/notes/scan.jpg", // not a page image
"/files/prism-slides/old-versions/draft.pdf",
"/files/prism-slides/versions/nosuch/x.pdf",
"/files/prism-slides/thumb.jpg", // the fixture has no thumbnail
"/files/prism-slides/../../../etc/passwd",
"/files/prism-slides/..%2f..%2fetc%2fpasswd",
"/files/prism-slides/versions/lemonde/metadata.json", // a version has none
"/files/prism-slides/glossary.json",
"/files/prism-slides/", // no file
"/files/prism-slides/nope.pdf", // inside the doc, missing
"/files/nosuchdoc/x.pdf", // unknown id
"/files/prism-slides/versions/lemonde/", // directory
} {
if w := get(t, path); w.Code == 200 {
t.Errorf("%s: served 200, want a refusal", path)
}
}
// The curated metadata is a served file like the PDF it describes.
if m := get(t, "/files/prism-slides/metadata.json"); m.Code != 200 || !strings.HasPrefix(m.Header().Get("Content-Type"), "application/json") {
t.Errorf("metadata.json: status %d, Content-Type %q, want 200 application/json", m.Code, m.Header().Get("Content-Type"))
}
// A .md transcript is served as plain text, never as HTML.
w := get(t, "/files/prism-slides/prism-slides.md")
if w.Code != 200 {
t.Fatalf("transcript status %d", w.Code)
}
if ct := w.Header().Get("Content-Type"); !strings.HasPrefix(ct, "text/plain") {
t.Errorf("transcript Content-Type %q, want text/plain", ct)
}
}
// Cacheability is decided by one rule - a request carrying any cookie is
// private - and personal pages go further with no-store. The hosting story
// (a shared microcache in front) rests on this.
func TestCacheHeaders(t *testing.T) {
anon := get(t, "/browse")
if cc := anon.Header().Get("Cache-Control"); cc != "public, max-age=10" {
t.Errorf("cookieless page: Cache-Control %q", cc)
}
if v := anon.Header().Get("Vary"); v != "Cookie" {
t.Errorf("cookieless page: Vary %q, want Cookie", v)
}
cookied := get(t, "/browse", &http.Cookie{Name: "pref_theme", Value: "dark"})
if cc := cookied.Header().Get("Cache-Control"); cc != "private, no-cache" {
t.Errorf("cookied page: Cache-Control %q", cc)
}
for _, p := range []string{"/preferences", "/pinned", "/pinned/export"} {
if cc := get(t, p).Header().Get("Cache-Control"); !strings.Contains(cc, "no-store") {
t.Errorf("%s: Cache-Control %q, want no-store", p, cc)
}
}
if cc := get(t, "/files/prism-slides/prism-slides.pdf").Header().Get("Cache-Control"); cc != "public, max-age=86400" {
t.Errorf("document file: Cache-Control %q", cc)
}
// A 404 must never inherit the immutable static cache.
if cc := get(t, "/static/nosuch.css?v=abc123").Header().Get("Cache-Control"); strings.Contains(cc, "immutable") {
t.Errorf("missing static asset marked immutable: %q", cc)
}
}
// Every response carries the baseline security headers, whatever it is.
func TestSecurityHeaders(t *testing.T) {
for _, p := range []string{"/", "/doc/prism-slides", "/api/facets", "/nosuchpage", "/files/prism-slides/prism-slides.pdf"} {
h := get(t, p).Header()
for _, want := range []string{"Content-Security-Policy", "X-Frame-Options", "X-Content-Type-Options", "Referrer-Policy"} {
if h.Get(want) == "" {
t.Errorf("%s: missing %s", p, want)
}
}
if csp := h.Get("Content-Security-Policy"); !strings.Contains(csp, "default-src 'self'") {
t.Errorf("%s: CSP %q", p, csp)
}
}
// The method gate's 405 too: scanner traffic is what it answers.
mux, _ := buildFixture(t)
w := record(mux, httptest.NewRequest("DELETE", "/", nil))
if w.Code != http.StatusMethodNotAllowed || w.Header().Get("Content-Security-Policy") == "" {
t.Errorf("DELETE: status %d, CSP %q; want 405 with the headers", w.Code, w.Header().Get("Content-Security-Policy"))
}
}
// State-changing POSTs are cookie-backed with no login, so a cross-site
// submission must be refused rather than silently flipping preferences.
func TestCrossSitePOSTRejected(t *testing.T) {
mux, _ := buildFixture(t)
for _, p := range []string{"/preferences/save", "/pin/prism-slides", "/history/clear", "/pinsearch"} {
// same-site is a sibling subdomain: another origin, not this site.
for _, site := range []string{"cross-site", "same-site"} {
r := newPost(p)
r.Header.Set("Sec-Fetch-Site", site)
if w := record(mux, r); w.Code != http.StatusForbidden {
t.Errorf("%s from %s: status %d, want 403", p, site, w.Code)
}
}
r := newPost(p)
r.Header.Set("Sec-Fetch-Site", "same-origin")
if w := record(mux, r); w.Code == http.StatusForbidden {
t.Errorf("%s from this site: rejected", p)
}
}
}
// Loading is idempotent: every build* starts from empty, so a second load
// (a test with another corpus, a future reload) produces the same archive
// rather than a doubled one. Several builders used to append to package
// slices that nothing reset.
func TestLoadDataIsIdempotent(t *testing.T) {
buildFixture(t)
type snapshot struct {
docs, timeline, codewords, groups, titled, glossary, sitemap, articles int
}
take := func() snapshot {
years := 0
for _, y := range corpus().archiveTimeline {
years += len(y.Months)
}
return snapshot{
docs: len(corpus().archive.Docs), timeline: years, codewords: len(corpus().codewords),
groups: len(corpus().articleGroups), titled: corpus().articlesTitled,
glossary: len(corpus().glossaryIndex) + len(corpus().glossaryGroups), sitemap: len(corpus().sitemapPaths), articles: len(corpus().articles),
}
}
before := take()
loadData()
if after := take(); after != before {
t.Errorf("a second load changed the archive:\n before %+v\n after %+v", before, after)
}
}
// robots.txt keeps crawlers off the routes that cost a corpus scan, and the
// sitemap never advertises a path robots.txt disallows.
func TestRobotsAndSitemapAgree(t *testing.T) {
robots := get(t, "/robots.txt").Body.String()
for _, want := range []string{"Disallow: /search", "Disallow: /api/", "Sitemap: "} {
if !strings.Contains(robots, want) {
t.Errorf("robots.txt lacks %q:\n%s", want, robots)
}
}
var disallowed []string
for _, line := range strings.Split(robots, "\n") {
if v := strings.TrimPrefix(line, "Disallow:"); v != line {
disallowed = append(disallowed, strings.TrimSpace(v))
}
}
sitemap := get(t, "/sitemap.xml").Body.String()
for _, m := range regexp.MustCompile(`<loc>([^<]+)</loc>`).FindAllStringSubmatch(sitemap, -1) {
path := m[1]
if i := strings.Index(path[len("https://"):], "/"); i >= 0 {
path = path[len("https://")+i:]
}
for _, d := range disallowed {
if d != "" && strings.HasPrefix(path, d) {
t.Errorf("sitemap advertises %s, which robots.txt disallows (%s)", path, d)
}
}
}
}
// With a canonical base configured, /sitemap.xml is marshaled once at load
// rather than per request - and still says the same thing.
func TestCanonicalSitemapPrebuilt(t *testing.T) {
buildFixture(t)
prev := canonicalBase
t.Cleanup(func() {
canonicalBase = prev
corpus().canonicalSitemap = nil
corpus().buildCanonicalSitemap()
})
canonicalBase = "https://archive.example.org"
corpus().buildCanonicalSitemap()
if corpus().canonicalSitemap == nil {
t.Fatal("no sitemap was built for the canonical base")
}
body := get(t, "/sitemap.xml").Body.String()
if !strings.Contains(body, "<loc>https://archive.example.org/doc/prism-slides</loc>") {
t.Errorf("sitemap does not use the canonical base:\n%s", body[:200])
}
if body != string(corpus().canonicalSitemap) {
t.Error("the served sitemap is not the one built at load")
}
}
// Without a canonical base the sitemap is written from parts cut at load,
// with the request's base between them: the document marshalSitemap gives
// for that base, to the byte - a base that needs escaping included.
func TestSitemapFromParts(t *testing.T) {
h, _ := buildFixture(t)
if canonicalBase != "" || corpus().sitemapParts == nil {
t.Fatalf("the fixture should serve its sitemap from parts (base %q, %d parts)", canonicalBase, len(corpus().sitemapParts))
}
for _, host := range []string{"archive.example.org", "odd&host<x>.example"} {
r := httptest.NewRequest("GET", "/sitemap.xml", nil)
r.Host = host
want, err := corpus().marshalSitemap(requestBase(r))
if err != nil {
t.Fatal(err)
}
if got := record(h, r).Body.String(); got != string(want) {
t.Errorf("host %q: the sitemap written from parts differs from marshalSitemap", host)
}
}
}
// The big visitor-independent page bodies are executed once at startup
// (prerenderPages). The page must look exactly the same whether it used
// that copy or rendered its rows inline - which is what dev mode does, and
// what every one of these pages falls back to if a block ever fails to
// pre-render.
func TestPrerenderedMatchesLive(t *testing.T) {
buildFixture(t)
if len(corpus().prerendered) == 0 {
t.Fatal("nothing was pre-rendered")
}
pages := []string{
"/explore/ids",
"/explore/ids?missing=1",
"/glossary",
"/browse?collection=sidtoday-archive",
"/browse/program",
"/browse/tag",
// Category orderings are ASSEMBLED from pre-rendered rows rather
// than pre-rendered whole, so each one has to match too.
"/browse/tag?sort=count",
"/browse/program?sort=count&order=asc",
"/browse/agency?sort=name&order=desc",
"/browse/people?sort=foia",
// So are the doc tables, out of pre-rendered document rows.
"/browse?view=table",
"/browse?collection=sidtoday-archive&view=table&sort=title&order=desc",
"/browse/program/PRISM?view=table",
}
was, wasRows, wasAZ, wasDocRows := corpus().prerendered, corpus().categoryRowHTML, corpus().azRowHTML, corpus().docRowHTML
t.Cleanup(func() {
corpus().prerendered, corpus().categoryRowHTML, corpus().azRowHTML, corpus().docRowHTML = was, wasRows, wasAZ, wasDocRows
})
live := make([]string, len(pages))
corpus().prerendered = map[string]template.HTML{} // force the inline path
corpus().categoryRowHTML, corpus().azRowHTML, corpus().docRowHTML = nil, nil, nil
for i, p := range pages {
live[i] = get(t, p).Body.String()
}
corpus().prerendered, corpus().categoryRowHTML, corpus().azRowHTML, corpus().docRowHTML = was, wasRows, wasAZ, wasDocRows
for i, p := range pages {
if got := get(t, p).Body.String(); got != live[i] {
t.Errorf("%s: pre-rendered output differs from the live one", p)
}
}
}
// The table view's headings are its sorters, and the one marked active is
// the order the rows are actually in - the page's default when the URL
// names none (the SIDtoday tab sorts by date, not title).
func TestDocTableActiveHeading(t *testing.T) {
for path, want := range map[string]string{
"/browse?collection=sidtoday-archive&view=table": "Doc Date &#9650;",
"/browse?view=table": "Title &#9650;",
"/browse?view=table&sort=country&order=desc": "Countries &#9660;",
"/browse?view=table&sort=bogus": "Title &#9650;",
} {
body := get(t, path).Body.String()
if !strings.Contains(body, want) {
t.Errorf("%s: no active heading %q", path, want)
}
if n := strings.Count(body, "&#9650;") + strings.Count(body, "&#9660;"); n != 1 {
t.Errorf("%s: %d active headings, want 1", path, n)
}
}
}
// A reporter credited by two co-publishing outlets is one name in the
// table's Journalists column, as on the cards.
func TestDocTableJournalistsDeduped(t *testing.T) {
buildFixture(t)
d := testDoc(t, `{"id": "x", "publication": [
{"publisher": "A", "reporters": ["Greenwald, Glenn"]},
{"publisher": "B", "reporters": ["Greenwald, Glenn", "Poitras, Laura"]}]}`)
var b strings.Builder
err := tmplSet["list.html"].ExecuteTemplate(&b, "doc-table", listView{Docs: []*Document{&d}})
if err != nil {
t.Fatal(err)
}
if n := strings.Count(b.String(), "/browse/journalist/Greenwald"); n != 1 {
t.Errorf("Greenwald linked %d times, want once", n)
}
if !strings.Contains(b.String(), "/browse/journalist/Poitras") {
t.Error("Poitras missing")
}
}
// Without fetch metadata the Origin decides, and it omits a default port
// the Host header may carry.
func TestOriginIsHost(t *testing.T) {
for _, c := range []struct {
origin, host string
want bool
}{
{"https://example.org", "example.org", true},
{"https://example.org", "example.org:443", true},
{"https://EXAMPLE.org", "example.org", true},
{"http://127.0.0.1:8080", "127.0.0.1:8080", true},
{"http://[::1]:8080", "[::1]:8080", true},
{"https://example.org", "example.org:8443", false},
{"https://example.org:8443", "example.org", false},
{"http://example.org", "example.org:443", false},
{"https://evil.example", "example.org", false},
} {
u, err := url.Parse(c.origin)
if err != nil {
t.Fatal(err)
}
if got := originIsHost(u, c.host); got != c.want {
t.Errorf("originIsHost(%s, %s) = %v, want %v", c.origin, c.host, got, c.want)
}
}
}
func TestEnvBool(t *testing.T) {
for v, want := range map[string]bool{"": false, "1": true, "true": true, "TRUE": true, "0": false, "false": false} {
t.Setenv("SNOWCHIVE_TEST_BOOL", v)
if got := envBool("SNOWCHIVE_TEST_BOOL"); got != want {
t.Errorf("envBool(%q) = %v, want %v", v, got, want)
}
}
}
// TestBrowseValueSpelling: a value matches ignoring case, so another
// spelling of it redirects to the one the lists link instead of serving a
// second page for the same documents; the query string goes along.
func TestBrowseValueSpelling(t *testing.T) {
w := get(t, "/browse/program/prism?view=table")
if w.Code != http.StatusMovedPermanently || w.Header().Get("Location") != "/browse/program/PRISM?view=table" {
t.Errorf("/browse/program/prism: %d to %q, want 301 to /browse/program/PRISM?view=table", w.Code, w.Header().Get("Location"))
}
if w := get(t, "/browse/program/NOSUCHTHING"); w.Code != 200 {
t.Errorf("an unknown value: %d, want the empty page", w.Code)
}
}
// TestCardHighlightNeedsTheCategory: the cards on a value's page mark the
// links to that value - not every link that happens to read the same, such
// as the document date 2013 on the publication-year page for 2013.
func TestCardHighlightNeedsTheCategory(t *testing.T) {
if body := get(t, "/browse/published/2013").Body.String(); strings.Contains(body, "doc-created active") {
t.Error("/browse/published/2013 marks the document-date links")
}
if body := get(t, "/browse/docdate/2013").Body.String(); !strings.Contains(body, "doc-created active") {
t.Error("/browse/docdate/2013 does not mark the document-date links")
}
if body := get(t, "/browse/program/PRISM").Body.String(); !strings.Contains(body, `class="active">PRISM</a>`) {
t.Error("/browse/program/PRISM does not mark the program links")
}
}
// TestUnknownSortIsNotEchoed: a sort key no list has is the default sort,
// and stays out of the page's links and labels.
func TestUnknownSortIsNotEchoed(t *testing.T) {
for _, path := range []string{"/browse?sort=bogus", "/browse/program/PRISM?sort=bogus", "/search?q=prism&do_search=1&sort=bogus"} {
body := get(t, path).Body.String()
if strings.Contains(body, "sort=bogus") || strings.Contains(body, "Bogus") {
t.Errorf("%s: the unknown sort is echoed", path)
}
}
if body := get(t, "/search?q=prism&do_search=1").Body.String(); strings.Contains(body, "sort=relevance&amp;order=desc") {
t.Error("relevance offers a direction it does not have")
}
}
// TestOutletYearLinksKeepTheScope: on the reported-only outlets table the
// year headings open the reported-only year, like the rows and cells do.
func TestOutletYearLinksKeepTheScope(t *testing.T) {
body := get(t, "/explore/outlets?coll=news").Body.String()
if !strings.Contains(body, `href="/browse/published/2013?coll=news"`) {
t.Error("the 2013 heading drops coll=news")
}
}
// pinSearch posts qs to /pinsearch from this site, with the given pinned
// searches already in the cookie.
func pinSearch(t *testing.T, qs string, pinned []string) *httptest.ResponseRecorder {
t.Helper()
mux, _ := buildFixture(t)
r := httptest.NewRequest("POST", "/pinsearch", strings.NewReader(url.Values{"qs": {qs}, "back": {"/search"}}.Encode()))
r.Header.Set("Content-Type", "application/x-www-form-urlencoded")
r.Header.Set("Sec-Fetch-Site", "same-origin")
if len(pinned) > 0 {
r.AddCookie(&http.Cookie{Name: pinnedSearchesCookie, Value: strings.Join(pinned, "|")})
}
return record(mux, r)
}
// TestPinListFull: a pin the cookie has no room for is refused, and the
// reader lands on /pinned, which says so, rather than back on a page whose
// button still reads "Pin" as though nothing happened.
func TestPinListFull(t *testing.T) {
qs := url.Values{"do_search": {"1"}, "fqc": {"and"}, "fqf": {"TE"}, "fqv": {"prism"}}.Encode()
if w := pinSearch(t, qs, nil); w.Code != http.StatusSeeOther || w.Header().Get("Location") != "/search" {
t.Fatalf("pinning into an empty list: %d to %q", w.Code, w.Header().Get("Location"))
}
// Filler searches up to the byte budget: stored entries are not
// checked on read, so any escaped query string stands in for one.
var fill []string
for i := 0; ; i++ {
e := url.QueryEscape(fmt.Sprintf("fqv=filler%02d", i) + strings.Repeat("x", 180))
if len(strings.Join(append(fill, e), "|")) > cookieValueMax {
break
}
fill = append(fill, e)
}
// The last filler takes the cookie to exactly its budget.
fill[len(fill)-1] += strings.Repeat("x", cookieValueMax-len(strings.Join(fill, "|")))
w := pinSearch(t, qs, fill)
if w.Code != http.StatusSeeOther || w.Header().Get("Location") != "/pinned?full=1" {
t.Errorf("pinning into a full list: %d to %q, want 303 to /pinned?full=1", w.Code, w.Header().Get("Location"))
}
if body := get(t, "/pinned?full=1").Body.String(); !strings.Contains(body, "Your pinned list is full") {
t.Error("/pinned?full=1 does not say the list is full")
}
}
// TestSavableSearchMatchesPinSearch: the search page offers the pin button
// for exactly the searches /pinsearch accepts - one size limit for both.
func TestSavableSearchMatchesPinSearch(t *testing.T) {
long := strings.TrimSpace(strings.Repeat("abcdefgh ", 58)) // 58 words, under the word cap
q := url.Values{"do_search": {"1"}, "fqc": {"and"}, "fqf": {"TE"}, "fqv": {long}}
if len(q.Encode()) <= searchQSMax {
t.Fatalf("test query is %d bytes, not over the %d limit", len(q.Encode()), searchQSMax)
}
if body := get(t, "/search?"+q.Encode()).Body.String(); strings.Contains(body, `action="/pinsearch"`) {
t.Error("the search page offers to pin a search /pinsearch refuses")
}
if w := pinSearch(t, q.Encode(), nil); w.Code != http.StatusBadRequest {
t.Errorf("/pinsearch with an over-long search: %d, want 400", w.Code)
}
short := url.Values{"do_search": {"1"}, "fqc": {"and"}, "fqf": {"TE"}, "fqv": {"prism"}}
if body := get(t, "/search?"+short.Encode()).Body.String(); !strings.Contains(body, `action="/pinsearch"`) {
t.Error("the search page does not offer to pin a short search")
}
}
// TestAssetStamp: pages link each asset by its content hash; that URL is
// cached for a year, and any other stamp only revalidates.
func TestAssetStamp(t *testing.T) {
body := get(t, "/").Body.String()
want := "/static/style.css?v=" + staticHashes["style.css"]
if !strings.Contains(body, `href="`+want+`"`) {
t.Fatalf("home page does not link %s", want)
}
if cc := get(t, want).Header().Get("Cache-Control"); !strings.Contains(cc, "immutable") {
t.Errorf("%s: Cache-Control %q, want immutable", want, cc)
}
w := get(t, "/static/style.css?v=0000000000000000")
if cc := w.Header().Get("Cache-Control"); strings.Contains(cc, "immutable") || w.Header().Get("ETag") == "" {
t.Errorf("a stale stamp: Cache-Control %q, ETag %q; want revalidation", cc, w.Header().Get("ETag"))
}
}
// TestSitemapIgnoresFile: the sitemap is always the one built from the
// loaded corpus - a sitemap.xml lying in the data or project directory is
// never served, with -base or without it.
func TestSitemapIgnoresFile(t *testing.T) {
root := t.TempDir()
writeFixtureFile(t, filepath.Join(root, "sitemap.xml"), xmlHeaderForTest+"<urlset><url><loc>https://stale.example/</loc></url></urlset>\n")
withCorpus(t, root)
prev := canonicalBase
t.Cleanup(func() { canonicalBase = prev })
for _, base := range []string{"", "https://archive.example.org"} {
canonicalBase = base
loadData()
if body := get(t, "/sitemap.xml").Body.String(); strings.Contains(body, "stale.example") {
t.Errorf("base %q: a sitemap.xml file was served", base)
}
}
}
const xmlHeaderForTest = `<?xml version="1.0" encoding="UTF-8"?>` + "\n"
// TestPinnedSearchListed: /pinned lists a pinned search with its label,
// its link and the unpin form carrying the canonical query string.
func TestPinnedSearchListed(t *testing.T) {
qs := url.Values{"do_search": {"1"}, "fqc": {"and"}, "fqf": {"TE"}, "fqv": {"prism"}}.Encode()
body := get(t, "/pinned", &http.Cookie{Name: pinnedSearchesCookie, Value: url.QueryEscape(qs)}).Body.String()
for _, want := range []string{`href="/search?` + template.HTMLEscapeString(qs) + `"`, `name="qs" value="` + template.HTMLEscapeString(qs) + `"`, ">prism<"} {
if !strings.Contains(body, want) {
t.Errorf("/pinned lacks %s", want)
}
}
}
// TestPageTitles: every page names itself in the tab, not just the site.
func TestPageTitles(t *testing.T) {
buildFixture(t) // the article key comes from the loaded corpus
for path, want := range map[string]string{
"/browse/program/PRISM": "<title>Program: PRISM - Snowden Archive</title>",
"/browse/program": "<title>Browse by Programs - Snowden Archive</title>",
"/search?q=prism&do_search=1": "<title>prism - Search - Snowden Archive</title>",
"/search": "<title>Search - Snowden Archive</title>",
"/browse?collection=foia": "<title>FOIA documents - Snowden Archive</title>",
"/pinned": "<title>Pinned - Snowden Archive</title>",
"/nosuchpage": "<title>Page not found - Snowden Archive</title>",
"/explore/pages": "- Snowden Archive</title>",
"/article/" + corpus().articleByLink["https://example.org/prism"].Key: "<title>NSA Prism program - Snowden Archive</title>",
} {
if body := get(t, path).Body.String(); !strings.Contains(body, want) {
t.Errorf("%s: no %q", path, want)
}
}
}
// TestNoBlockContentInHeadings: a heading holds text and inline controls
// only - a form or a disclosure inside one is invalid HTML and lands in
// the heading's accessible name. Forms tie their buttons from outside.
func TestNoBlockContentInHeadings(t *testing.T) {
heading := regexp.MustCompile(`(?s)<h([1-6])\b[^>]*>(.*?)</h[1-6]>`)
block := regexp.MustCompile(`<(form|details|div|p|ul|ol|table)\b`)
cookie := &http.Cookie{Name: pinnedCookie, Value: "prism-slides"}
recent := []*http.Cookie{{Name: "pref_track_recent", Value: "1"}, {Name: recentDocsCookie, Value: "prism-slides"}}
pages := map[string][]*http.Cookie{
"/pinned": {cookie},
"/": recent,
"/search?q=prism&do_search=1": nil,
"/doc/prism-slides": nil,
}
for path, cookies := range pages {
body := get(t, path, cookies...).Body.String()
for _, m := range heading.FindAllStringSubmatch(body, -1) {
if b := block.FindString(m[2]); b != "" {
t.Errorf("%s: %s inside <h%s>", path, b, m[1])
}
}
}
if body := get(t, "/pinned", cookie).Body.String(); !strings.Contains(body, `form="unpin-prism-slides"`) || !strings.Contains(body, `id="unpin-prism-slides"`) {
t.Error("/pinned: the unpin button is not tied to its form")
}
}
// TestGlossarySeeAlsoResolves: a see-also name links to the entry it
// names - through an alias or another case too - and a name no entry has
// is plain text rather than a link to nowhere.
func TestGlossarySeeAlsoResolves(t *testing.T) {
root := t.TempDir()
writeFixtureFile(t, filepath.Join(root, "glossary.json"), `[
{"term": "XKEYSCORE", "definition": "A search system.", "aka": ["XKS"], "see": ["rt-rg", "TURMOIL", "NOSUCHTERM"]},
{"term": "RT-RG", "definition": "Real time regional gateway."},
{"term": "TURMOIL", "definition": "Passive collection.", "see": ["xks"]}
]`)
withCorpus(t, root)
body := get(t, "/glossary").Body.String()
for _, want := range []string{`<a href="#term-RT-RG">rt-rg</a>`, `<a href="#term-TURMOIL">TURMOIL</a>`, `<a href="#term-XKEYSCORE">xks</a>`} {
if !strings.Contains(body, want) {
t.Errorf("/glossary lacks %s", want)
}
}
if strings.Contains(body, `href="#term-NOSUCHTERM"`) {
t.Error("/glossary links a see-also name no entry has")
}
}
// A pager or sort link appends its parameter to queryBase, with nothing
// left over when the page has no query of its own.
func TestQueryBase(t *testing.T) {
for want, q := range map[string]url.Values{
"/browse?": {},
"/browse?view=table&": {"view": {"table"}},
"/browse?coll=foia&view=card&": {"view": {"card"}, "coll": {"foia"}},
} {
if got := queryBase("/browse", q); got != want {
t.Errorf("queryBase(%v) = %q, want %q", q, got, want)
}
}
}

22
tests/run.sh Executable file
View file

@ -0,0 +1,22 @@
#!/bin/sh
# Runs the tests in this directory. They are package main's own tests, and Go
# only compiles those from the package's directory, so an overlay presents
# them to the go tool as if they were there. Arguments go to "go test":
#
# tests/run.sh
# tests/run.sh -run TestCorpus -v
set -e
cd "$(dirname "$0")/.."
root=$(pwd)
overlay=$(mktemp)
trap 'rm -f "$overlay"' EXIT
{
printf '{"Replace":{'
sep=
for f in tests/*_test.go; do
printf '%s"%s/%s":"%s/%s"' "$sep" "$root" "${f#tests/}" "$root" "$f"
sep=,
done
printf '}}\n'
} >"$overlay"
go test -overlay "$overlay" "$@" .

1000
tests/search_test.go Normal file

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,58 @@
package main
import (
"path/filepath"
"strings"
"testing"
)
// The home figures describe the Reported collection; /?sidtoday=1 adds the
// SIDtoday archive, and the note above them links between the two. In the
// consistency corpus only s1 (SIDtoday archive) is dated 2011-05.
func TestHomeStatsWithSIDtoday(t *testing.T) {
withCorpus(t, writeConsistencyCorpus(t))
plain := get(t, "/").Body.String()
if !strings.Contains(plain, "exclude SIDtoday archive documents") || !strings.Contains(plain, `href="/?sidtoday=1"`) {
t.Error("/: no note linking to the figures with the SIDtoday archive")
}
if strings.Contains(plain, "coll=sidtoday-archive") {
t.Error("/: a figure links into the SIDtoday archive")
}
with := get(t, "/?sidtoday=1").Body.String()
if !strings.Contains(with, "include SIDtoday archive documents") {
t.Error("/?sidtoday=1: the note does not say the figures include the SIDtoday archive")
}
if !strings.Contains(with, `href="/browse/docdate/2011?coll=news&amp;coll=sidtoday-archive"`) {
t.Error("/?sidtoday=1: the document-date bars do not link with both collections")
}
}
// The "reported" badge tells an individually reported SIDtoday issue apart
// from the bulk release, so only the SIDtoday tab shows it: every view of
// that tab, and no other page.
func TestReportedBadgeOnlyOnSIDtodayTab(t *testing.T) {
root := writeConsistencyCorpus(t)
d := filepath.Join(root, "r1")
writeFixtureFile(t, filepath.Join(d, "r1.pdf"), "%PDF-1.4\n")
writeFixtureFile(t, filepath.Join(d, "metadata.json"), `{"title": "Reported issue", "type": "SIDTODAY",
"document": {"date_created": "2011-06-01"},
"publication": [{"publisher": "The Intercept", "date_published": "2014-03-01", "link": "https://example.org/r"}]}`)
withCorpus(t, root)
const badge = `class="issue-reported" title=`
for _, p := range []string{"/browse?collection=sidtoday-archive", "/browse?collection=sidtoday-archive&view=card",
"/browse?collection=sidtoday-archive&view=table"} {
if !strings.Contains(get(t, p).Body.String(), badge) {
t.Errorf("%s: no reported badge", p)
}
}
for _, p := range []string{"/browse", "/browse?view=table", "/search?q=reported&do_search=1", "/browse/publisher/The%20Intercept"} {
body := get(t, p).Body.String()
if !strings.Contains(body, `href="/doc/r1"`) {
t.Errorf("%s does not list r1", p)
}
if strings.Contains(body, badge) {
t.Errorf("%s shows the reported badge", p)
}
}
}

464
tests/tmplcheck_test.go Normal file
View file

@ -0,0 +1,464 @@
package main
import (
"fmt"
"io/fs"
"reflect"
"sort"
"strings"
"testing"
"text/template/parse"
)
// Every template is checked against the struct it is executed with, without
// executing it: each field or method it reads must exist on the type it is
// read from - in every branch, including the ones no test request takes.
// html/template only finds a wrong name when the action runs.
//
// The checker follows the data through range, with, variables and template
// calls. Where a type is not known statically it does not guess: a value
// behind an interface{} (a dict's missing key, the argument of a partial
// called with a literal) is accepted whatever is read from it.
func TestTemplatesMatchPages(t *testing.T) {
buildFixture(t)
for _, p := range pageTypes {
name := p.templateName()
set, err := parsePage(name) // a fresh parse: executing rewrites the trees
if err != nil {
t.Fatalf("%s: %v", name, err)
}
c := &tmplCheck{seen: map[string]bool{}, records: map[reflect.Type]bool{}}
c.lookup = func(n string) *parse.Tree {
if tt := set.Lookup(n); tt != nil {
return tt.Tree
}
return nil
}
c.call(name, reflect.TypeOf(p), 0)
sort.Strings(c.errs)
for _, e := range c.errs {
t.Errorf("%s (%T): %s", name, p, e)
}
}
}
// Each template file is some page's, and each page's template exists.
func TestEveryTemplateHasAPage(t *testing.T) {
buildFixture(t)
files, err := fs.Glob(templateFS, "*.html")
if err != nil {
t.Fatal(err)
}
byName := map[string]bool{}
for _, p := range pageTypes {
if byName[p.templateName()] {
t.Errorf("%s is the template of two page types", p.templateName())
}
byName[p.templateName()] = true
}
for _, f := range files {
if f == "base.html" || f == "partials.html" {
continue
}
if !byName[f] {
t.Errorf("%s has no page type in pageTypes", f)
}
delete(byName, f)
}
for n := range byName {
t.Errorf("page type for %s, which is not a template file", n)
}
}
// The checker itself: a misspelled field is found in a branch that would
// not run, behind a range, a variable and a partial; a dict's keys carry
// their types into the partial they are passed to.
func TestTemplateCheckerFindsWrongNames(t *testing.T) {
type row struct{ Name string }
type page struct {
Rows []row
Doc *Document
M map[string]int
}
for src, want := range map[string]string{
`{{ if .Nope }}x{{ end }}`: "Nope",
`{{ if false }}{{ range .Rows }}{{ .Nmae }}{{ end }}{{ end }}`: "Nmae",
`{{ range $i, $r := .Rows }}{{ $r.Name }}{{ $.Rowz }}{{ end }}`: "Rowz",
`{{ with .Doc }}{{ .Title }}{{ .DocMeta.Creatr }}{{ end }}`: "Creatr",
`{{ $d := .Doc }}{{ $d.Titel }}`: "Titel",
`{{ template "p" .Doc }}{{ define "p" }}{{ .ID }}{{ .Idd }}{{ end }}`: "Idd",
`{{ template "p" (dict "D" .Doc) }}{{ define "p" }}{{ .D.Idd }}{{ end }}`: "Idd",
`{{ index .M "a" }}{{ (index .Rows 0).Nmae }}`: "Nmae",
`{{ .Rows.Name }}`: "Name",
`{{ range .Rows }}{{ .Name }}{{ else }}{{ .Name }}{{ end }}`: "Name",
`{{ template "p" (dict "D" .Doc) }}{{ define "p" }}{{ .Missing.X }}{{ end }}`: "",
`{{ range .Rows }}{{ .Name }}{{ end }}{{ .Doc.Title }}{{ len .M }}`: "",
`{{ $x := "" }}{{ if .Doc }}{{ $x = .Doc }}{{ end }}{{ $x.Anything }}`: "",
} {
trees, err := parse.Parse("t", src, "", "", checkerFuncs())
if err != nil {
t.Fatalf("%s: %v", src, err)
}
c := &tmplCheck{seen: map[string]bool{}, records: map[reflect.Type]bool{}}
c.lookup = func(n string) *parse.Tree { return trees[n] }
c.call("t", reflect.TypeOf(page{}), 0)
got := strings.Join(c.errs, "; ")
if (want == "") != (got == "") || !strings.Contains(got, want) {
t.Errorf("%s\n errors: %q, want one naming %q", src, got, want)
}
}
}
// checkerFuncs is funcMap as the parser wants it, with the builtins the
// templates use.
func checkerFuncs() map[string]any {
m := map[string]any{}
for k, v := range funcMap {
m[k] = v
}
for _, b := range []string{"and", "or", "not", "eq", "ne", "lt", "le", "gt", "ge", "len", "index",
"slice", "print", "printf", "println", "html", "js", "urlquery", "call"} {
m[b] = true
}
return m
}
type tmplCheck struct {
lookup func(name string) *parse.Tree
seen map[string]bool // template name + argument type, already walked
records map[reflect.Type]bool // types made from a dict call: a missing key is nil, not an error
errs []string
where string // the template being walked
}
type tmplVar struct {
name string
typ reflect.Type // nil: not known
}
var (
stringType = reflect.TypeOf("")
intType = reflect.TypeOf(0)
boolType = reflect.TypeOf(false)
floatType = reflect.TypeOf(0.0)
anyType = reflect.TypeOf((*any)(nil)).Elem()
)
func (c *tmplCheck) errorf(n parse.Node, format string, a ...any) {
loc := c.where
if tree := c.lookup(c.where); tree != nil {
loc, _ = tree.ErrorContext(n)
}
c.errs = append(c.errs, fmt.Sprintf("%s (in %q): %s", loc, c.where, fmt.Sprintf(format, a...)))
}
// call walks the named template with dot of type arg.
func (c *tmplCheck) call(name string, arg reflect.Type, depth int) {
key := name + "|"
if arg != nil {
key += arg.String()
if c.records[arg] {
key += fmt.Sprintf("|%p", arg) // records print alike only when they are alike; keep them apart anyway
}
}
if c.seen[key] || depth > 20 {
return
}
c.seen[key] = true
tree := c.lookup(name)
if tree == nil || tree.Root == nil {
c.errs = append(c.errs, fmt.Sprintf("template %q is not defined", name))
return
}
prev := c.where
c.where = name
vars := []tmplVar{{"$", arg}}
c.walk(tree.Root, arg, &vars, depth)
c.where = prev
}
func (c *tmplCheck) walk(n parse.Node, dot reflect.Type, vars *[]tmplVar, depth int) {
switch n := n.(type) {
case *parse.ListNode:
if n == nil {
return
}
for _, x := range n.Nodes {
c.walk(x, dot, vars, depth)
}
case *parse.ActionNode:
c.pipe(n.Pipe, dot, vars)
case *parse.IfNode:
mark := len(*vars)
c.pipe(n.Pipe, dot, vars)
c.walk(n.List, dot, vars, depth)
c.walk(n.ElseList, dot, vars, depth)
*vars = (*vars)[:mark]
case *parse.WithNode:
mark := len(*vars)
t := c.pipe(n.Pipe, dot, vars)
c.walk(n.List, t, vars, depth)
c.walk(n.ElseList, dot, vars, depth)
*vars = (*vars)[:mark]
case *parse.RangeNode:
mark := len(*vars)
t := c.cmds(n.Pipe, dot, vars)
key, elem := rangeTypes(t)
switch len(n.Pipe.Decl) {
case 1:
*vars = append(*vars, tmplVar{n.Pipe.Decl[0].Ident[0], elem})
case 2:
*vars = append(*vars, tmplVar{n.Pipe.Decl[0].Ident[0], key}, tmplVar{n.Pipe.Decl[1].Ident[0], elem})
}
c.walk(n.List, elem, vars, depth)
*vars = (*vars)[:mark]
c.walk(n.ElseList, dot, vars, depth)
case *parse.TemplateNode:
var arg reflect.Type
if n.Pipe != nil {
arg = c.pipe(n.Pipe, dot, vars)
}
c.call(n.Name, arg, depth+1)
}
}
func rangeTypes(t reflect.Type) (key, elem reflect.Type) {
t = indirectType(t)
if t == nil {
return nil, nil
}
switch t.Kind() {
case reflect.Slice, reflect.Array:
return intType, known(t.Elem())
case reflect.Map:
return known(t.Key()), known(t.Elem())
case reflect.Int:
return intType, intType
}
return nil, nil
}
// known is t, or nil when t says nothing about the value (an interface).
func known(t reflect.Type) reflect.Type {
if t != nil && t.Kind() == reflect.Interface {
return nil
}
return t
}
func indirectType(t reflect.Type) reflect.Type {
for t != nil && t.Kind() == reflect.Pointer {
t = t.Elem()
}
return known(t)
}
// pipe is the type a pipeline yields, declaring or assigning its variable.
func (c *tmplCheck) pipe(p *parse.PipeNode, dot reflect.Type, vars *[]tmplVar) reflect.Type {
if p == nil {
return nil
}
t := c.cmds(p, dot, vars)
for _, d := range p.Decl {
name := d.Ident[0]
if !p.IsAssign {
*vars = append(*vars, tmplVar{name, t})
continue
}
for i := len(*vars) - 1; i >= 0; i-- {
if (*vars)[i].name == name {
if (*vars)[i].typ != t {
(*vars)[i].typ = nil // assigned two types: whichever branch ran decides
}
break
}
}
}
return t
}
func (c *tmplCheck) cmds(p *parse.PipeNode, dot reflect.Type, vars *[]tmplVar) reflect.Type {
var t reflect.Type
for i, cmd := range p.Cmds {
t = c.cmd(cmd, dot, vars, i > 0, t)
}
return t
}
func (c *tmplCheck) cmd(cmd *parse.CommandNode, dot reflect.Type, vars *[]tmplVar, piped bool, prev reflect.Type) reflect.Type {
first := cmd.Args[0]
if id, ok := first.(*parse.IdentifierNode); ok {
if id.Ident == "dict" {
return c.dict(cmd, dot, vars)
}
args := make([]reflect.Type, 0, len(cmd.Args))
for _, a := range cmd.Args[1:] {
args = append(args, c.arg(a, dot, vars))
}
if piped {
args = append(args, prev)
}
return c.fn(id, args)
}
for _, a := range cmd.Args[1:] { // a method's arguments
c.arg(a, dot, vars)
}
return c.arg(first, dot, vars)
}
func (c *tmplCheck) arg(n parse.Node, dot reflect.Type, vars *[]tmplVar) reflect.Type {
switch n := n.(type) {
case *parse.DotNode:
return dot
case *parse.FieldNode:
return c.fields(n, dot, n.Ident)
case *parse.VariableNode:
for i := len(*vars) - 1; i >= 0; i-- {
if (*vars)[i].name == n.Ident[0] {
return c.fields(n, (*vars)[i].typ, n.Ident[1:])
}
}
c.errorf(n, "undefined variable %s", n.Ident[0])
case *parse.ChainNode:
return c.fields(n, c.arg(n.Node, dot, vars), n.Field)
case *parse.PipeNode:
return c.pipe(n, dot, vars)
case *parse.IdentifierNode:
return c.fn(n, nil)
case *parse.StringNode:
return stringType
case *parse.BoolNode:
return boolType
case *parse.NumberNode:
if n.IsInt {
return intType
}
return floatType
}
return nil
}
// fields follows a chain of names from t.
func (c *tmplCheck) fields(n parse.Node, t reflect.Type, names []string) reflect.Type {
for _, name := range names {
if t == nil {
return nil
}
// A method, on the value or on its address.
recv := t
if recv.Kind() != reflect.Pointer && recv.Kind() != reflect.Interface {
recv = reflect.PointerTo(recv)
}
if m, ok := recv.MethodByName(name); ok {
t = nil
if m.Type.NumOut() > 0 {
t = known(m.Type.Out(0))
}
continue
}
s := indirectType(t)
if s == nil {
return nil
}
switch s.Kind() {
case reflect.Struct:
f, ok := s.FieldByName(name)
if !ok || !f.IsExported() {
if c.records[s] {
return nil // a key this dict call did not pass
}
c.errorf(n, "no field or method %s in %s", name, s)
return nil
}
t = known(f.Type)
case reflect.Map:
if s.Key().Kind() != reflect.String {
c.errorf(n, "%s read from %s, a map without string keys", name, s)
return nil
}
t = known(s.Elem())
default:
c.errorf(n, "%s read from %s, which has no fields", name, s)
return nil
}
}
return t
}
// dict is the record a dict call builds: each literal key with the type of
// the value passed for it.
func (c *tmplCheck) dict(cmd *parse.CommandNode, dot reflect.Type, vars *[]tmplVar) reflect.Type {
var fs []reflect.StructField
args := cmd.Args[1:]
for i := 0; i+1 < len(args); i += 2 {
k, ok := args[i].(*parse.StringNode)
t := c.arg(args[i+1], dot, vars)
if !ok {
return nil // a computed key: nothing is known about the record
}
if t == nil {
t = anyType
}
fs = append(fs, reflect.StructField{Name: k.Text, Type: t})
}
defer func() { recover() }() // a key that is not a Go identifier: no record
rt := reflect.StructOf(fs)
c.records[rt] = true
return rt
}
// fn is the type a function call yields.
func (c *tmplCheck) fn(id *parse.IdentifierNode, args []reflect.Type) reflect.Type {
switch id.Ident {
case "not", "eq", "ne", "lt", "le", "gt", "ge":
return boolType
case "len":
return intType
case "print", "printf", "println", "html", "js", "urlquery":
return stringType
case "and", "or":
for _, a := range args {
if a == nil || a != args[0] {
return nil
}
}
if len(args) > 0 {
return args[0]
}
return nil
case "slice":
if len(args) > 0 {
return args[0]
}
return nil
case "index":
if len(args) == 0 {
return nil
}
t := args[0]
for range args[1:] {
t = indirectType(t)
if t == nil {
return nil
}
switch t.Kind() {
case reflect.Map, reflect.Slice, reflect.Array:
t = known(t.Elem())
default:
return nil
}
}
return t
case "call":
return nil
}
f, ok := funcMap[id.Ident]
if !ok {
return nil // the parser has already refused an unknown function
}
ft := reflect.TypeOf(f)
if ft.NumOut() == 0 {
return nil
}
return known(ft.Out(0))
}

View file

@ -0,0 +1,59 @@
package main
import (
"encoding/json"
"fmt"
"path/filepath"
"testing"
)
func TestChangedPages(t *testing.T) {
dir := t.TempDir()
v := filepath.Join(dir, "versions", "entier")
writeTestFile(t, filepath.Join(v, "copy.pdf"), "%PDF-1.4")
for _, n := range []string{"1", "2", "7", "8"} {
writeTestFile(t, filepath.Join(v, "images", "image-"+n+".jpg"), "")
}
var d Document
if err := json.Unmarshal([]byte(`{
"publication": [{"publisher": "Le Monde", "version": "entier"}],
"versions": {"entier": {"changed_pages": [8, 7, 12, 7]}, "ghost": {"changed_pages": [1]}}
}`), &d); err != nil {
t.Fatal(err)
}
d.ID, d.Dir = "prism", dir
corpus().loadVersions(&d)
e := d.VersionFor("entier")
if e == nil {
t.Fatal("version entier not loaded")
}
// Numbers come from the filenames; positions are where those images sit
// in the page grid (image-7 is the 3rd image). 12 has no image: kept,
// without a position.
if got := fmt.Sprint(e.ChangedPages); got != "[{7 3} {8 4} {12 0}]" {
t.Errorf("changed pages %s, want [{7 3} {8 4} {12 0}]", got)
}
for pos, want := range map[int]bool{0: false, 1: false, 2: false, 3: true, 4: true} {
if e.PageChanged(pos) != want {
t.Errorf("PageChanged(%d) = %v, want %v", pos, !want, want)
}
}
if e.DocID != "prism" {
t.Errorf("DocID %q, want prism", e.DocID)
}
}
func TestPageNumber(t *testing.T) {
for name, want := range map[string]int{"page-12.jpg": 12, "image-07.png": 7} {
if n, err := pageNumber(name); err != nil || n != want {
t.Errorf("pageNumber(%q) = %d, %v; want %d", name, n, err, want)
}
}
for _, name := range []string{"cover.jpg", "page-a.jpg"} {
if _, err := pageNumber(name); err == nil {
t.Errorf("pageNumber(%q) should fail", name)
}
}
}

View file

@ -0,0 +1,101 @@
package main
import (
"encoding/json"
"strings"
"testing"
)
func versionedDoc(t *testing.T, main string, versions map[string]string, pubsJSON string) Document {
t.Helper()
var d Document
if pubsJSON != "" {
if err := json.Unmarshal([]byte(pubsJSON), &d); err != nil {
t.Fatal(err)
}
}
d.ID, d.FullText = "doc", main
for _, slug := range []string{"lemonde", "spiegel"} {
if text, ok := versions[slug]; ok {
d.Versions = append(d.Versions, DocVersion{Slug: slug, FullText: text})
}
}
return d
}
func TestVersionNovelText(t *testing.T) {
d := versionedDoc(t, "PRISM Overview\nShared line\n", map[string]string{
"lemonde": "shared LINE\r\nNew in Le Monde\n---\nIn both copies\nNew in Le Monde\n",
"spiegel": "In both copies\nOnly Der Spiegel\n",
}, "")
combined := versionNovelText(&d)
if got := d.Versions[0].NovelText; got != "New in Le Monde\nIn both copies\nNew in Le Monde" {
t.Errorf("lemonde new lines %q", got)
}
if got := d.Versions[1].NovelText; got != "In both copies\nOnly Der Spiegel" {
t.Errorf("spiegel new lines %q", got)
}
if combined != "New in Le Monde\nIn both copies\nOnly Der Spiegel" {
t.Errorf("combined new lines %q, want each line once", combined)
}
}
func TestVersionLabel(t *testing.T) {
d := versionedDoc(t, "", map[string]string{"lemonde": "", "spiegel": ""}, `{"publication": [
{"publisher": "Le Monde", "date_published": "2013-10-21"},
{"publisher": "Le Monde", "date_published": "2013-10-25"},
{"publisher": "The Guardian", "date_published": "2013-10-20"}
]}`)
d.Versions[0].PubIndex = []int{0, 1, 2}
if got := versionLabel(&d, &d.Versions[0]); got != "Le Monde / The Guardian, 2013-10-20" {
t.Errorf("label %q", got)
}
if got := versionLabel(&d, &d.Versions[1]); got != "spiegel" {
t.Errorf("label of a version no publication names %q, want its folder name", got)
}
}
func TestVersionTextIsSearchable(t *testing.T) {
d := versionedDoc(t, "The overview ends with alpha", map[string]string{
"lemonde": "The overview ends with alpha\nbeta starts the Wanadoo slide",
}, "")
setBodySearchText(&d)
d.StemmedText = d.StemmedFull
d.SearchText = d.FoldedFull
match := func(q, field string) bool {
return compileQuery([]string{q}, []string{field}, []string{"and"}).matches(&d)
}
if !match("wanadoo", "FT") || !match(`"wanadoo slide"`, "FT") || !match("=wanadoo", "FT") || !match("wanad*", "FT") {
t.Error("a term only in a version's new lines should match the document")
}
if match(`"alpha beta"`, "FT") {
t.Error("a quoted phrase matched across the join between the main text and a version's lines")
}
if tok := tokenize("overview")[0]; strings.Count(" "+d.StemmedFull+" ", " "+tok+" ") != 1 {
t.Errorf("a line shared with the main text was indexed twice: %q", d.StemmedFull)
}
// buildTFIDF skips the separator only as a token of its own; glued to a
// word it would mint terms and hide that word from phrase matching.
for _, tok := range strings.Fields(d.StemmedFull) {
if tok != valueSep && strings.Contains(tok, valueSep) {
t.Errorf("the join separator is glued to a term in StemmedFull: %q", tok)
}
}
idx := buildTFIDF([]Document{d})
for w := range idx.DocFreq {
if strings.Contains(w, valueSep) {
t.Errorf("the join separator was indexed in the TF-IDF term %q", w)
}
}
hits := versionHits(&d, "wanadoo", 3)
if len(hits) != 1 || hits[0].Version.Slug != "lemonde" || !strings.Contains(hits[0].Snippets[0], "Wanadoo") {
t.Errorf("versionHits = %+v, want one lemonde hit with a Wanadoo snippet", hits)
}
if hits := versionHits(&d, "overview", 3); len(hits) != 0 {
t.Errorf("versionHits found %+v for a word only in lines shared with the main text", hits)
}
}