first commit
This commit is contained in:
commit
8c4dcae475
89 changed files with 24623 additions and 0 deletions
661
LICENSE
Normal file
661
LICENSE
Normal file
|
|
@ -0,0 +1,661 @@
|
|||
GNU AFFERO GENERAL PUBLIC LICENSE
|
||||
Version 3, 19 November 2007
|
||||
|
||||
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
|
||||
Everyone is permitted to copy and distribute verbatim copies
|
||||
of this license document, but changing it is not allowed.
|
||||
|
||||
Preamble
|
||||
|
||||
The GNU Affero General Public License is a free, copyleft license for
|
||||
software and other kinds of works, specifically designed to ensure
|
||||
cooperation with the community in the case of network server software.
|
||||
|
||||
The licenses for most software and other practical works are designed
|
||||
to take away your freedom to share and change the works. By contrast,
|
||||
our General Public Licenses are intended to guarantee your freedom to
|
||||
share and change all versions of a program--to make sure it remains free
|
||||
software for all its users.
|
||||
|
||||
When we speak of free software, we are referring to freedom, not
|
||||
price. Our General Public Licenses are designed to make sure that you
|
||||
have the freedom to distribute copies of free software (and charge for
|
||||
them if you wish), that you receive source code or can get it if you
|
||||
want it, that you can change the software or use pieces of it in new
|
||||
free programs, and that you know you can do these things.
|
||||
|
||||
Developers that use our General Public Licenses protect your rights
|
||||
with two steps: (1) assert copyright on the software, and (2) offer
|
||||
you this License which gives you legal permission to copy, distribute
|
||||
and/or modify the software.
|
||||
|
||||
A secondary benefit of defending all users' freedom is that
|
||||
improvements made in alternate versions of the program, if they
|
||||
receive widespread use, become available for other developers to
|
||||
incorporate. Many developers of free software are heartened and
|
||||
encouraged by the resulting cooperation. However, in the case of
|
||||
software used on network servers, this result may fail to come about.
|
||||
The GNU General Public License permits making a modified version and
|
||||
letting the public access it on a server without ever releasing its
|
||||
source code to the public.
|
||||
|
||||
The GNU Affero General Public License is designed specifically to
|
||||
ensure that, in such cases, the modified source code becomes available
|
||||
to the community. It requires the operator of a network server to
|
||||
provide the source code of the modified version running there to the
|
||||
users of that server. Therefore, public use of a modified version, on
|
||||
a publicly accessible server, gives the public access to the source
|
||||
code of the modified version.
|
||||
|
||||
An older license, called the Affero General Public License and
|
||||
published by Affero, was designed to accomplish similar goals. This is
|
||||
a different license, not a version of the Affero GPL, but Affero has
|
||||
released a new version of the Affero GPL which permits relicensing under
|
||||
this license.
|
||||
|
||||
The precise terms and conditions for copying, distribution and
|
||||
modification follow.
|
||||
|
||||
TERMS AND CONDITIONS
|
||||
|
||||
0. Definitions.
|
||||
|
||||
"This License" refers to version 3 of the GNU Affero General Public License.
|
||||
|
||||
"Copyright" also means copyright-like laws that apply to other kinds of
|
||||
works, such as semiconductor masks.
|
||||
|
||||
"The Program" refers to any copyrightable work licensed under this
|
||||
License. Each licensee is addressed as "you". "Licensees" and
|
||||
"recipients" may be individuals or organizations.
|
||||
|
||||
To "modify" a work means to copy from or adapt all or part of the work
|
||||
in a fashion requiring copyright permission, other than the making of an
|
||||
exact copy. The resulting work is called a "modified version" of the
|
||||
earlier work or a work "based on" the earlier work.
|
||||
|
||||
A "covered work" means either the unmodified Program or a work based
|
||||
on the Program.
|
||||
|
||||
To "propagate" a work means to do anything with it that, without
|
||||
permission, would make you directly or secondarily liable for
|
||||
infringement under applicable copyright law, except executing it on a
|
||||
computer or modifying a private copy. Propagation includes copying,
|
||||
distribution (with or without modification), making available to the
|
||||
public, and in some countries other activities as well.
|
||||
|
||||
To "convey" a work means any kind of propagation that enables other
|
||||
parties to make or receive copies. Mere interaction with a user through
|
||||
a computer network, with no transfer of a copy, is not conveying.
|
||||
|
||||
An interactive user interface displays "Appropriate Legal Notices"
|
||||
to the extent that it includes a convenient and prominently visible
|
||||
feature that (1) displays an appropriate copyright notice, and (2)
|
||||
tells the user that there is no warranty for the work (except to the
|
||||
extent that warranties are provided), that licensees may convey the
|
||||
work under this License, and how to view a copy of this License. If
|
||||
the interface presents a list of user commands or options, such as a
|
||||
menu, a prominent item in the list meets this criterion.
|
||||
|
||||
1. Source Code.
|
||||
|
||||
The "source code" for a work means the preferred form of the work
|
||||
for making modifications to it. "Object code" means any non-source
|
||||
form of a work.
|
||||
|
||||
A "Standard Interface" means an interface that either is an official
|
||||
standard defined by a recognized standards body, or, in the case of
|
||||
interfaces specified for a particular programming language, one that
|
||||
is widely used among developers working in that language.
|
||||
|
||||
The "System Libraries" of an executable work include anything, other
|
||||
than the work as a whole, that (a) is included in the normal form of
|
||||
packaging a Major Component, but which is not part of that Major
|
||||
Component, and (b) serves only to enable use of the work with that
|
||||
Major Component, or to implement a Standard Interface for which an
|
||||
implementation is available to the public in source code form. A
|
||||
"Major Component", in this context, means a major essential component
|
||||
(kernel, window system, and so on) of the specific operating system
|
||||
(if any) on which the executable work runs, or a compiler used to
|
||||
produce the work, or an object code interpreter used to run it.
|
||||
|
||||
The "Corresponding Source" for a work in object code form means all
|
||||
the source code needed to generate, install, and (for an executable
|
||||
work) run the object code and to modify the work, including scripts to
|
||||
control those activities. However, it does not include the work's
|
||||
System Libraries, or general-purpose tools or generally available free
|
||||
programs which are used unmodified in performing those activities but
|
||||
which are not part of the work. For example, Corresponding Source
|
||||
includes interface definition files associated with source files for
|
||||
the work, and the source code for shared libraries and dynamically
|
||||
linked subprograms that the work is specifically designed to require,
|
||||
such as by intimate data communication or control flow between those
|
||||
subprograms and other parts of the work.
|
||||
|
||||
The Corresponding Source need not include anything that users
|
||||
can regenerate automatically from other parts of the Corresponding
|
||||
Source.
|
||||
|
||||
The Corresponding Source for a work in source code form is that
|
||||
same work.
|
||||
|
||||
2. Basic Permissions.
|
||||
|
||||
All rights granted under this License are granted for the term of
|
||||
copyright on the Program, and are irrevocable provided the stated
|
||||
conditions are met. This License explicitly affirms your unlimited
|
||||
permission to run the unmodified Program. The output from running a
|
||||
covered work is covered by this License only if the output, given its
|
||||
content, constitutes a covered work. This License acknowledges your
|
||||
rights of fair use or other equivalent, as provided by copyright law.
|
||||
|
||||
You may make, run and propagate covered works that you do not
|
||||
convey, without conditions so long as your license otherwise remains
|
||||
in force. You may convey covered works to others for the sole purpose
|
||||
of having them make modifications exclusively for you, or provide you
|
||||
with facilities for running those works, provided that you comply with
|
||||
the terms of this License in conveying all material for which you do
|
||||
not control copyright. Those thus making or running the covered works
|
||||
for you must do so exclusively on your behalf, under your direction
|
||||
and control, on terms that prohibit them from making any copies of
|
||||
your copyrighted material outside their relationship with you.
|
||||
|
||||
Conveying under any other circumstances is permitted solely under
|
||||
the conditions stated below. Sublicensing is not allowed; section 10
|
||||
makes it unnecessary.
|
||||
|
||||
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
|
||||
|
||||
No covered work shall be deemed part of an effective technological
|
||||
measure under any applicable law fulfilling obligations under article
|
||||
11 of the WIPO copyright treaty adopted on 20 December 1996, or
|
||||
similar laws prohibiting or restricting circumvention of such
|
||||
measures.
|
||||
|
||||
When you convey a covered work, you waive any legal power to forbid
|
||||
circumvention of technological measures to the extent such circumvention
|
||||
is effected by exercising rights under this License with respect to
|
||||
the covered work, and you disclaim any intention to limit operation or
|
||||
modification of the work as a means of enforcing, against the work's
|
||||
users, your or third parties' legal rights to forbid circumvention of
|
||||
technological measures.
|
||||
|
||||
4. Conveying Verbatim Copies.
|
||||
|
||||
You may convey verbatim copies of the Program's source code as you
|
||||
receive it, in any medium, provided that you conspicuously and
|
||||
appropriately publish on each copy an appropriate copyright notice;
|
||||
keep intact all notices stating that this License and any
|
||||
non-permissive terms added in accord with section 7 apply to the code;
|
||||
keep intact all notices of the absence of any warranty; and give all
|
||||
recipients a copy of this License along with the Program.
|
||||
|
||||
You may charge any price or no price for each copy that you convey,
|
||||
and you may offer support or warranty protection for a fee.
|
||||
|
||||
5. Conveying Modified Source Versions.
|
||||
|
||||
You may convey a work based on the Program, or the modifications to
|
||||
produce it from the Program, in the form of source code under the
|
||||
terms of section 4, provided that you also meet all of these conditions:
|
||||
|
||||
a) The work must carry prominent notices stating that you modified
|
||||
it, and giving a relevant date.
|
||||
|
||||
b) The work must carry prominent notices stating that it is
|
||||
released under this License and any conditions added under section
|
||||
7. This requirement modifies the requirement in section 4 to
|
||||
"keep intact all notices".
|
||||
|
||||
c) You must license the entire work, as a whole, under this
|
||||
License to anyone who comes into possession of a copy. This
|
||||
License will therefore apply, along with any applicable section 7
|
||||
additional terms, to the whole of the work, and all its parts,
|
||||
regardless of how they are packaged. This License gives no
|
||||
permission to license the work in any other way, but it does not
|
||||
invalidate such permission if you have separately received it.
|
||||
|
||||
d) If the work has interactive user interfaces, each must display
|
||||
Appropriate Legal Notices; however, if the Program has interactive
|
||||
interfaces that do not display Appropriate Legal Notices, your
|
||||
work need not make them do so.
|
||||
|
||||
A compilation of a covered work with other separate and independent
|
||||
works, which are not by their nature extensions of the covered work,
|
||||
and which are not combined with it such as to form a larger program,
|
||||
in or on a volume of a storage or distribution medium, is called an
|
||||
"aggregate" if the compilation and its resulting copyright are not
|
||||
used to limit the access or legal rights of the compilation's users
|
||||
beyond what the individual works permit. Inclusion of a covered work
|
||||
in an aggregate does not cause this License to apply to the other
|
||||
parts of the aggregate.
|
||||
|
||||
6. Conveying Non-Source Forms.
|
||||
|
||||
You may convey a covered work in object code form under the terms
|
||||
of sections 4 and 5, provided that you also convey the
|
||||
machine-readable Corresponding Source under the terms of this License,
|
||||
in one of these ways:
|
||||
|
||||
a) Convey the object code in, or embodied in, a physical product
|
||||
(including a physical distribution medium), accompanied by the
|
||||
Corresponding Source fixed on a durable physical medium
|
||||
customarily used for software interchange.
|
||||
|
||||
b) Convey the object code in, or embodied in, a physical product
|
||||
(including a physical distribution medium), accompanied by a
|
||||
written offer, valid for at least three years and valid for as
|
||||
long as you offer spare parts or customer support for that product
|
||||
model, to give anyone who possesses the object code either (1) a
|
||||
copy of the Corresponding Source for all the software in the
|
||||
product that is covered by this License, on a durable physical
|
||||
medium customarily used for software interchange, for a price no
|
||||
more than your reasonable cost of physically performing this
|
||||
conveying of source, or (2) access to copy the
|
||||
Corresponding Source from a network server at no charge.
|
||||
|
||||
c) Convey individual copies of the object code with a copy of the
|
||||
written offer to provide the Corresponding Source. This
|
||||
alternative is allowed only occasionally and noncommercially, and
|
||||
only if you received the object code with such an offer, in accord
|
||||
with subsection 6b.
|
||||
|
||||
d) Convey the object code by offering access from a designated
|
||||
place (gratis or for a charge), and offer equivalent access to the
|
||||
Corresponding Source in the same way through the same place at no
|
||||
further charge. You need not require recipients to copy the
|
||||
Corresponding Source along with the object code. If the place to
|
||||
copy the object code is a network server, the Corresponding Source
|
||||
may be on a different server (operated by you or a third party)
|
||||
that supports equivalent copying facilities, provided you maintain
|
||||
clear directions next to the object code saying where to find the
|
||||
Corresponding Source. Regardless of what server hosts the
|
||||
Corresponding Source, you remain obligated to ensure that it is
|
||||
available for as long as needed to satisfy these requirements.
|
||||
|
||||
e) Convey the object code using peer-to-peer transmission, provided
|
||||
you inform other peers where the object code and Corresponding
|
||||
Source of the work are being offered to the general public at no
|
||||
charge under subsection 6d.
|
||||
|
||||
A separable portion of the object code, whose source code is excluded
|
||||
from the Corresponding Source as a System Library, need not be
|
||||
included in conveying the object code work.
|
||||
|
||||
A "User Product" is either (1) a "consumer product", which means any
|
||||
tangible personal property which is normally used for personal, family,
|
||||
or household purposes, or (2) anything designed or sold for incorporation
|
||||
into a dwelling. In determining whether a product is a consumer product,
|
||||
doubtful cases shall be resolved in favor of coverage. For a particular
|
||||
product received by a particular user, "normally used" refers to a
|
||||
typical or common use of that class of product, regardless of the status
|
||||
of the particular user or of the way in which the particular user
|
||||
actually uses, or expects or is expected to use, the product. A product
|
||||
is a consumer product regardless of whether the product has substantial
|
||||
commercial, industrial or non-consumer uses, unless such uses represent
|
||||
the only significant mode of use of the product.
|
||||
|
||||
"Installation Information" for a User Product means any methods,
|
||||
procedures, authorization keys, or other information required to install
|
||||
and execute modified versions of a covered work in that User Product from
|
||||
a modified version of its Corresponding Source. The information must
|
||||
suffice to ensure that the continued functioning of the modified object
|
||||
code is in no case prevented or interfered with solely because
|
||||
modification has been made.
|
||||
|
||||
If you convey an object code work under this section in, or with, or
|
||||
specifically for use in, a User Product, and the conveying occurs as
|
||||
part of a transaction in which the right of possession and use of the
|
||||
User Product is transferred to the recipient in perpetuity or for a
|
||||
fixed term (regardless of how the transaction is characterized), the
|
||||
Corresponding Source conveyed under this section must be accompanied
|
||||
by the Installation Information. But this requirement does not apply
|
||||
if neither you nor any third party retains the ability to install
|
||||
modified object code on the User Product (for example, the work has
|
||||
been installed in ROM).
|
||||
|
||||
The requirement to provide Installation Information does not include a
|
||||
requirement to continue to provide support service, warranty, or updates
|
||||
for a work that has been modified or installed by the recipient, or for
|
||||
the User Product in which it has been modified or installed. Access to a
|
||||
network may be denied when the modification itself materially and
|
||||
adversely affects the operation of the network or violates the rules and
|
||||
protocols for communication across the network.
|
||||
|
||||
Corresponding Source conveyed, and Installation Information provided,
|
||||
in accord with this section must be in a format that is publicly
|
||||
documented (and with an implementation available to the public in
|
||||
source code form), and must require no special password or key for
|
||||
unpacking, reading or copying.
|
||||
|
||||
7. Additional Terms.
|
||||
|
||||
"Additional permissions" are terms that supplement the terms of this
|
||||
License by making exceptions from one or more of its conditions.
|
||||
Additional permissions that are applicable to the entire Program shall
|
||||
be treated as though they were included in this License, to the extent
|
||||
that they are valid under applicable law. If additional permissions
|
||||
apply only to part of the Program, that part may be used separately
|
||||
under those permissions, but the entire Program remains governed by
|
||||
this License without regard to the additional permissions.
|
||||
|
||||
When you convey a copy of a covered work, you may at your option
|
||||
remove any additional permissions from that copy, or from any part of
|
||||
it. (Additional permissions may be written to require their own
|
||||
removal in certain cases when you modify the work.) You may place
|
||||
additional permissions on material, added by you to a covered work,
|
||||
for which you have or can give appropriate copyright permission.
|
||||
|
||||
Notwithstanding any other provision of this License, for material you
|
||||
add to a covered work, you may (if authorized by the copyright holders of
|
||||
that material) supplement the terms of this License with terms:
|
||||
|
||||
a) Disclaiming warranty or limiting liability differently from the
|
||||
terms of sections 15 and 16 of this License; or
|
||||
|
||||
b) Requiring preservation of specified reasonable legal notices or
|
||||
author attributions in that material or in the Appropriate Legal
|
||||
Notices displayed by works containing it; or
|
||||
|
||||
c) Prohibiting misrepresentation of the origin of that material, or
|
||||
requiring that modified versions of such material be marked in
|
||||
reasonable ways as different from the original version; or
|
||||
|
||||
d) Limiting the use for publicity purposes of names of licensors or
|
||||
authors of the material; or
|
||||
|
||||
e) Declining to grant rights under trademark law for use of some
|
||||
trade names, trademarks, or service marks; or
|
||||
|
||||
f) Requiring indemnification of licensors and authors of that
|
||||
material by anyone who conveys the material (or modified versions of
|
||||
it) with contractual assumptions of liability to the recipient, for
|
||||
any liability that these contractual assumptions directly impose on
|
||||
those licensors and authors.
|
||||
|
||||
All other non-permissive additional terms are considered "further
|
||||
restrictions" within the meaning of section 10. If the Program as you
|
||||
received it, or any part of it, contains a notice stating that it is
|
||||
governed by this License along with a term that is a further
|
||||
restriction, you may remove that term. If a license document contains
|
||||
a further restriction but permits relicensing or conveying under this
|
||||
License, you may add to a covered work material governed by the terms
|
||||
of that license document, provided that the further restriction does
|
||||
not survive such relicensing or conveying.
|
||||
|
||||
If you add terms to a covered work in accord with this section, you
|
||||
must place, in the relevant source files, a statement of the
|
||||
additional terms that apply to those files, or a notice indicating
|
||||
where to find the applicable terms.
|
||||
|
||||
Additional terms, permissive or non-permissive, may be stated in the
|
||||
form of a separately written license, or stated as exceptions;
|
||||
the above requirements apply either way.
|
||||
|
||||
8. Termination.
|
||||
|
||||
You may not propagate or modify a covered work except as expressly
|
||||
provided under this License. Any attempt otherwise to propagate or
|
||||
modify it is void, and will automatically terminate your rights under
|
||||
this License (including any patent licenses granted under the third
|
||||
paragraph of section 11).
|
||||
|
||||
However, if you cease all violation of this License, then your
|
||||
license from a particular copyright holder is reinstated (a)
|
||||
provisionally, unless and until the copyright holder explicitly and
|
||||
finally terminates your license, and (b) permanently, if the copyright
|
||||
holder fails to notify you of the violation by some reasonable means
|
||||
prior to 60 days after the cessation.
|
||||
|
||||
Moreover, your license from a particular copyright holder is
|
||||
reinstated permanently if the copyright holder notifies you of the
|
||||
violation by some reasonable means, this is the first time you have
|
||||
received notice of violation of this License (for any work) from that
|
||||
copyright holder, and you cure the violation prior to 30 days after
|
||||
your receipt of the notice.
|
||||
|
||||
Termination of your rights under this section does not terminate the
|
||||
licenses of parties who have received copies or rights from you under
|
||||
this License. If your rights have been terminated and not permanently
|
||||
reinstated, you do not qualify to receive new licenses for the same
|
||||
material under section 10.
|
||||
|
||||
9. Acceptance Not Required for Having Copies.
|
||||
|
||||
You are not required to accept this License in order to receive or
|
||||
run a copy of the Program. Ancillary propagation of a covered work
|
||||
occurring solely as a consequence of using peer-to-peer transmission
|
||||
to receive a copy likewise does not require acceptance. However,
|
||||
nothing other than this License grants you permission to propagate or
|
||||
modify any covered work. These actions infringe copyright if you do
|
||||
not accept this License. Therefore, by modifying or propagating a
|
||||
covered work, you indicate your acceptance of this License to do so.
|
||||
|
||||
10. Automatic Licensing of Downstream Recipients.
|
||||
|
||||
Each time you convey a covered work, the recipient automatically
|
||||
receives a license from the original licensors, to run, modify and
|
||||
propagate that work, subject to this License. You are not responsible
|
||||
for enforcing compliance by third parties with this License.
|
||||
|
||||
An "entity transaction" is a transaction transferring control of an
|
||||
organization, or substantially all assets of one, or subdividing an
|
||||
organization, or merging organizations. If propagation of a covered
|
||||
work results from an entity transaction, each party to that
|
||||
transaction who receives a copy of the work also receives whatever
|
||||
licenses to the work the party's predecessor in interest had or could
|
||||
give under the previous paragraph, plus a right to possession of the
|
||||
Corresponding Source of the work from the predecessor in interest, if
|
||||
the predecessor has it or can get it with reasonable efforts.
|
||||
|
||||
You may not impose any further restrictions on the exercise of the
|
||||
rights granted or affirmed under this License. For example, you may
|
||||
not impose a license fee, royalty, or other charge for exercise of
|
||||
rights granted under this License, and you may not initiate litigation
|
||||
(including a cross-claim or counterclaim in a lawsuit) alleging that
|
||||
any patent claim is infringed by making, using, selling, offering for
|
||||
sale, or importing the Program or any portion of it.
|
||||
|
||||
11. Patents.
|
||||
|
||||
A "contributor" is a copyright holder who authorizes use under this
|
||||
License of the Program or a work on which the Program is based. The
|
||||
work thus licensed is called the contributor's "contributor version".
|
||||
|
||||
A contributor's "essential patent claims" are all patent claims
|
||||
owned or controlled by the contributor, whether already acquired or
|
||||
hereafter acquired, that would be infringed by some manner, permitted
|
||||
by this License, of making, using, or selling its contributor version,
|
||||
but do not include claims that would be infringed only as a
|
||||
consequence of further modification of the contributor version. For
|
||||
purposes of this definition, "control" includes the right to grant
|
||||
patent sublicenses in a manner consistent with the requirements of
|
||||
this License.
|
||||
|
||||
Each contributor grants you a non-exclusive, worldwide, royalty-free
|
||||
patent license under the contributor's essential patent claims, to
|
||||
make, use, sell, offer for sale, import and otherwise run, modify and
|
||||
propagate the contents of its contributor version.
|
||||
|
||||
In the following three paragraphs, a "patent license" is any express
|
||||
agreement or commitment, however denominated, not to enforce a patent
|
||||
(such as an express permission to practice a patent or covenant not to
|
||||
sue for patent infringement). To "grant" such a patent license to a
|
||||
party means to make such an agreement or commitment not to enforce a
|
||||
patent against the party.
|
||||
|
||||
If you convey a covered work, knowingly relying on a patent license,
|
||||
and the Corresponding Source of the work is not available for anyone
|
||||
to copy, free of charge and under the terms of this License, through a
|
||||
publicly available network server or other readily accessible means,
|
||||
then you must either (1) cause the Corresponding Source to be so
|
||||
available, or (2) arrange to deprive yourself of the benefit of the
|
||||
patent license for this particular work, or (3) arrange, in a manner
|
||||
consistent with the requirements of this License, to extend the patent
|
||||
license to downstream recipients. "Knowingly relying" means you have
|
||||
actual knowledge that, but for the patent license, your conveying the
|
||||
covered work in a country, or your recipient's use of the covered work
|
||||
in a country, would infringe one or more identifiable patents in that
|
||||
country that you have reason to believe are valid.
|
||||
|
||||
If, pursuant to or in connection with a single transaction or
|
||||
arrangement, you convey, or propagate by procuring conveyance of, a
|
||||
covered work, and grant a patent license to some of the parties
|
||||
receiving the covered work authorizing them to use, propagate, modify
|
||||
or convey a specific copy of the covered work, then the patent license
|
||||
you grant is automatically extended to all recipients of the covered
|
||||
work and works based on it.
|
||||
|
||||
A patent license is "discriminatory" if it does not include within
|
||||
the scope of its coverage, prohibits the exercise of, or is
|
||||
conditioned on the non-exercise of one or more of the rights that are
|
||||
specifically granted under this License. You may not convey a covered
|
||||
work if you are a party to an arrangement with a third party that is
|
||||
in the business of distributing software, under which you make payment
|
||||
to the third party based on the extent of your activity of conveying
|
||||
the work, and under which the third party grants, to any of the
|
||||
parties who would receive the covered work from you, a discriminatory
|
||||
patent license (a) in connection with copies of the covered work
|
||||
conveyed by you (or copies made from those copies), or (b) primarily
|
||||
for and in connection with specific products or compilations that
|
||||
contain the covered work, unless you entered into that arrangement,
|
||||
or that patent license was granted, prior to 28 March 2007.
|
||||
|
||||
Nothing in this License shall be construed as excluding or limiting
|
||||
any implied license or other defenses to infringement that may
|
||||
otherwise be available to you under applicable patent law.
|
||||
|
||||
12. No Surrender of Others' Freedom.
|
||||
|
||||
If conditions are imposed on you (whether by court order, agreement or
|
||||
otherwise) that contradict the conditions of this License, they do not
|
||||
excuse you from the conditions of this License. If you cannot convey a
|
||||
covered work so as to satisfy simultaneously your obligations under this
|
||||
License and any other pertinent obligations, then as a consequence you may
|
||||
not convey it at all. For example, if you agree to terms that obligate you
|
||||
to collect a royalty for further conveying from those to whom you convey
|
||||
the Program, the only way you could satisfy both those terms and this
|
||||
License would be to refrain entirely from conveying the Program.
|
||||
|
||||
13. Remote Network Interaction; Use with the GNU General Public License.
|
||||
|
||||
Notwithstanding any other provision of this License, if you modify the
|
||||
Program, your modified version must prominently offer all users
|
||||
interacting with it remotely through a computer network (if your version
|
||||
supports such interaction) an opportunity to receive the Corresponding
|
||||
Source of your version by providing access to the Corresponding Source
|
||||
from a network server at no charge, through some standard or customary
|
||||
means of facilitating copying of software. This Corresponding Source
|
||||
shall include the Corresponding Source for any work covered by version 3
|
||||
of the GNU General Public License that is incorporated pursuant to the
|
||||
following paragraph.
|
||||
|
||||
Notwithstanding any other provision of this License, you have
|
||||
permission to link or combine any covered work with a work licensed
|
||||
under version 3 of the GNU General Public License into a single
|
||||
combined work, and to convey the resulting work. The terms of this
|
||||
License will continue to apply to the part which is the covered work,
|
||||
but the work with which it is combined will remain governed by version
|
||||
3 of the GNU General Public License.
|
||||
|
||||
14. Revised Versions of this License.
|
||||
|
||||
The Free Software Foundation may publish revised and/or new versions of
|
||||
the GNU Affero General Public License from time to time. Such new versions
|
||||
will be similar in spirit to the present version, but may differ in detail to
|
||||
address new problems or concerns.
|
||||
|
||||
Each version is given a distinguishing version number. If the
|
||||
Program specifies that a certain numbered version of the GNU Affero General
|
||||
Public License "or any later version" applies to it, you have the
|
||||
option of following the terms and conditions either of that numbered
|
||||
version or of any later version published by the Free Software
|
||||
Foundation. If the Program does not specify a version number of the
|
||||
GNU Affero General Public License, you may choose any version ever published
|
||||
by the Free Software Foundation.
|
||||
|
||||
If the Program specifies that a proxy can decide which future
|
||||
versions of the GNU Affero General Public License can be used, that proxy's
|
||||
public statement of acceptance of a version permanently authorizes you
|
||||
to choose that version for the Program.
|
||||
|
||||
Later license versions may give you additional or different
|
||||
permissions. However, no additional obligations are imposed on any
|
||||
author or copyright holder as a result of your choosing to follow a
|
||||
later version.
|
||||
|
||||
15. Disclaimer of Warranty.
|
||||
|
||||
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
|
||||
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
|
||||
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
|
||||
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
|
||||
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
||||
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
|
||||
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
|
||||
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
|
||||
|
||||
16. Limitation of Liability.
|
||||
|
||||
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
|
||||
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
|
||||
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
|
||||
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
|
||||
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
|
||||
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
|
||||
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
|
||||
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
|
||||
SUCH DAMAGES.
|
||||
|
||||
17. Interpretation of Sections 15 and 16.
|
||||
|
||||
If the disclaimer of warranty and limitation of liability provided
|
||||
above cannot be given local legal effect according to their terms,
|
||||
reviewing courts shall apply local law that most closely approximates
|
||||
an absolute waiver of all civil liability in connection with the
|
||||
Program, unless a warranty or assumption of liability accompanies a
|
||||
copy of the Program in return for a fee.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
How to Apply These Terms to Your New Programs
|
||||
|
||||
If you develop a new program, and you want it to be of the greatest
|
||||
possible use to the public, the best way to achieve this is to make it
|
||||
free software which everyone can redistribute and change under these terms.
|
||||
|
||||
To do so, attach the following notices to the program. It is safest
|
||||
to attach them to the start of each source file to most effectively
|
||||
state the exclusion of warranty; and each file should have at least
|
||||
the "copyright" line and a pointer to where the full notice is found.
|
||||
|
||||
<one line to give the program's name and a brief idea of what it does.>
|
||||
Copyright (C) <year> <name of author>
|
||||
|
||||
This program is free software: you can redistribute it and/or modify
|
||||
it under the terms of the GNU Affero General Public License as published by
|
||||
the Free Software Foundation, either version 3 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU Affero General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU Affero General Public License
|
||||
along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
Also add information on how to contact you by electronic and paper mail.
|
||||
|
||||
If your software can interact with users remotely through a computer
|
||||
network, you should also make sure that it provides a way for users to
|
||||
get its source. For example, if your program is a web application, its
|
||||
interface could display a "Source" link that leads users to an archive
|
||||
of the code. There are many ways you could offer source, and different
|
||||
solutions will be better for different programs; see section 13 for the
|
||||
specific requirements.
|
||||
|
||||
You should also get your employer (if you work as a programmer) or school,
|
||||
if any, to sign a "copyright disclaimer" for the program, if necessary.
|
||||
For more information on this, and how to apply and follow the GNU AGPL, see
|
||||
<https://www.gnu.org/licenses/>.
|
||||
177
README.md
Normal file
177
README.md
Normal file
|
|
@ -0,0 +1,177 @@
|
|||
# Snowchive server
|
||||
|
||||
A self-contained web server for browsing and searching an archive of published
|
||||
Snowden documents in Snowchive website. The server includes faceted browsing, a
|
||||
boolean/fielded search engine with stemming and accent folding, a SIDtoday
|
||||
serial reader, a read-only JSON API, sitemaps, and per-visitor preferences, pins
|
||||
and history that live only in the visitor's cookies. The server uses just Go's
|
||||
standard library.
|
||||
|
||||
No JavaScript is required to use the website. A file in `static/main.js` adds
|
||||
progressive enhancements only. No database and no external services.
|
||||
|
||||
See it in action: https://snowchive.org
|
||||
|
||||
If you are looking for Snowchive data or metadata:
|
||||
|
||||
Data (PDFs, page images):
|
||||
|
||||
https://snowchive.org/about#downloading-all-data
|
||||
|
||||
Metadata (with extracted texts):
|
||||
|
||||
https://git.disroot.org/libroot/snowchive-meta
|
||||
|
||||
## Run
|
||||
|
||||
Full Snowchive data is required (containing all the PDFs and metadata) to run
|
||||
the server. If you don't yet have the data, download it first:
|
||||
https://snowchive.org/about#downloading-all-data
|
||||
|
||||
```sh
|
||||
go run . -root=/path/to/snowchive-data/
|
||||
```
|
||||
|
||||
## Building
|
||||
|
||||
If you want to build a binary to deploy one executable for production use:
|
||||
|
||||
```sh
|
||||
go build -ldflags "-X main.buildCommit=$(git rev-parse --short HEAD)" -o snowchive .
|
||||
./snowchive -root=/path/to/snowchive-data/ -base=https://snowchive.example.org
|
||||
```
|
||||
|
||||
## Flags
|
||||
|
||||
| flag | purpose |
|
||||
|---|---|
|
||||
| `-root` | Snowchive data directory holding `data/` (the corpus, one directory per document) and `project/` (`glossary.json`, `assemblies.json`, `changelog.json`, `related.json`) (default: the working directory, `$SNOWCHIVE_ROOT`) |
|
||||
| `-listen` | TCP address (default `127.0.0.1:8080`, `$SNOWCHIVE_LISTEN`). |
|
||||
| `-base` | canonical `scheme://host` for absolute URLs in robots.txt and the sitemap; set it on anything internet-facing. Without it those URLs follow each request's host. |
|
||||
| `-no-api` | disable the JSON API (`$SNOWCHIVE_NO_API=1`) |
|
||||
| `-accel-files PREFIX` | behind nginx only: answer document files with an `X-Accel-Redirect` into this internal location (e.g. `/_files/`, an `alias` of the data directory), so nginx sends the bytes and this server only decides whether the file is served (`$SNOWCHIVE_ACCEL_FILES`) |
|
||||
| `-accel-expires DURATION` | let the proxy cache a page served to a reader without cookies for this long (e.g. `10m`, sent as `X-Accel-Expires`); browsers keep the 10-second `max-age`. Clear the proxy cache after a restart with new data (`$SNOWCHIVE_ACCEL_EXPIRES`) |
|
||||
| `-dev` | serve `templates/` and `static/` from the working directory, re-parse per request, verbose errors. Never in production. |
|
||||
| `-check` | load the corpus, print the warnings and the tally, then exit non-zero if anything was reported - run it after a curation pass or in CI. |
|
||||
| `-rebuild-related` | compute `related.json` entries for documents that lack one, or whose list links to a document no longer in the corpus, drop the entries of documents that are gone, then exit. It only fills gaps, so after a scoring change delete `related.json` first to rebuild every list. |
|
||||
|
||||
## What it costs to run
|
||||
|
||||
Measured on a 2-core VM against the 3,106-document corpus: ~7 seconds to load
|
||||
(warm page cache), ~260 MB of live heap, and ~750 MB of address space under
|
||||
sustained concurrent load.
|
||||
|
||||
## Directory layout
|
||||
|
||||
```
|
||||
data/
|
||||
<document-id>/ one directory per document
|
||||
metadata.json document metadata
|
||||
<anything>.pdf document PDF file
|
||||
<anything>.md extracted text transcript from PDF
|
||||
thumb.jpg, thumb.gif document thumbnail images
|
||||
images/page-N.jpg PDF page scans
|
||||
versions/<slug>/ another published copy that differs (e.g. less
|
||||
redacted, extra pages): its own PDF,
|
||||
.md and images/ (optional)
|
||||
FOIA/<document-id>/ the FOIA document collection
|
||||
SIDtoday-archive/<id>/ the bulk SIDtoday document collection
|
||||
```
|
||||
|
||||
The files about the archive as a whole are in the project directory
|
||||
(`-project`):
|
||||
|
||||
```
|
||||
project/
|
||||
glossary.json format documented in glossary.go
|
||||
assemblies.json see "Assemblies" below
|
||||
changelog.json snowchive changelog, format in changelog.go
|
||||
related.json documents relating to each other, generated by
|
||||
-rebuild-related
|
||||
```
|
||||
|
||||
The corpus is read exactly once at startup and everything derived from it is
|
||||
computed there. All of it lives in one `Corpus` value (`corpus.go`), which is
|
||||
built whole and pre-rendered. The server writes nothing while it runs. Only
|
||||
`-rebuild-related` writes a file (`related.json`, in the project directory).
|
||||
|
||||
## About metadata
|
||||
|
||||
`metadata.json` carries the title, description, tags, the `document`
|
||||
block (creator, date_created, programs, classification, dissemination,
|
||||
banner, genre, series, countries, people) and the `publication` list
|
||||
(publisher, title, link, reporters, contributors, date_published, plus
|
||||
`doc_source` and `doc_url` for the copy that entry published, and `note`
|
||||
and `version` when that copy differs from the primary one). Every field
|
||||
except `title` is optional. There is no document-level `doc_source`/`doc_url`,
|
||||
because where a copy came from is a property of the publication that published
|
||||
it.
|
||||
|
||||
A publication entry is an outlet that published the document. A partner in a
|
||||
joint investigation that published the story but no documents is the one
|
||||
exception (they have entry `"reporting_only": true`).
|
||||
|
||||
A publication that released a different copy of the same document is in
|
||||
`version` directory. The directory `versions/<slug>/` is in the document
|
||||
directory, holding that copy's PDF, `.md` and `images/`. The document page shows
|
||||
the publication's `note` and links the copy, which has its own page at
|
||||
`/doc/<id>/versions/<slug>`. Versions never count as documents. Search covers
|
||||
the lines of a version's transcript that the main transcript lacks; results
|
||||
label those matches with the copy they came from. A version no publication names
|
||||
is listed as another copy.
|
||||
|
||||
The `versions` object records, per version directory, the SHA-256 of that copy's
|
||||
PDF and the pages that differ from the main document, by the number in the page
|
||||
image's filename:
|
||||
`"versions": {"<slug>": {"pdf_sha256": "...", "changed_pages": [7, 12]}}`.
|
||||
|
||||
`aka_id` lists other names the document was published under, as an array of
|
||||
objects: `{"id": "...", "doc_source": "...", "doc_url": "..."}`. An aka is a
|
||||
pointer and stores no file, which is what separates it from a `versions/` entry.
|
||||
|
||||
More info about the metadata: https://git.disroot.org/libroot/snowchive-meta
|
||||
|
||||
## Assemblies
|
||||
|
||||
`assemblies.json` is a hand-maintained list of sets of pages drawn from several
|
||||
documents and versions, for material that was published in pieces, such as a
|
||||
slide deck. Each gets a page at `/explore/assemblies/<id>` showing all its
|
||||
slides in one grid and lightbox, listed at `/explore/assemblies`
|
||||
|
||||
```json
|
||||
[{
|
||||
"id": "prism-overview",
|
||||
"title": "PRISM/US-984XN briefing",
|
||||
"note": "Shown at the top: how the set was put together and what is unproven.",
|
||||
"slides": [
|
||||
{"doc": "<document-id>", "pages": "1-7"},
|
||||
{"doc": "<document-id>", "version": "<slug>", "pages": "8", "note": "Shown under the slide."},
|
||||
{"doc": "<document-id>", "pages": "9,11",
|
||||
"alt": [{"doc": "<document-id>", "version": "<slug>", "pages": "1-2"}]}
|
||||
]
|
||||
}]
|
||||
```
|
||||
An assembly is the archive's own arrangement, it's not an original document and
|
||||
stores no files, and is neither counted nor searched.
|
||||
|
||||
## Tests
|
||||
|
||||
./tests/run.sh
|
||||
|
||||
TODO. The tests are currently written largely by Claude.
|
||||
|
||||
## Other/contact
|
||||
|
||||
The maintainers of this project do not endorse any public Snowchive instance
|
||||
other than the official snowchive.org instance (unless you are a reputable
|
||||
organization that wants to host a public instance).
|
||||
|
||||
Please submit issues/feedback on the Snowchive server's repository:
|
||||
|
||||
https://git.disroot.org/libroot/snowchive-server
|
||||
|
||||
or to Codeberg (mirror): https://codeberg.org/libroot/snowchive-server
|
||||
|
||||
Contact the maintainers via email:
|
||||
|
||||
https://libroot.org/about
|
||||
60
agencies.go
Normal file
60
agencies.go
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
// agencies.go - display names for the acronyms used as document creators
|
||||
|
||||
package main
|
||||
|
||||
import "strings"
|
||||
|
||||
var agencyNames = map[string]string{
|
||||
"CSE": "Communications Security Establishment (Canada)",
|
||||
"DFS": "Directorate for Signals Intelligence (Japan)",
|
||||
"DNI": "Director of National Intelligence (US)",
|
||||
"DOD": "Department of Defense (US)",
|
||||
"DOJ": "Department of Justice (US)",
|
||||
"DOS": "Department of State (US)",
|
||||
"DSD": "Defence Signals Directorate (Australia; now ASD)",
|
||||
"FISC": "Foreign Intelligence Surveillance Court (US)",
|
||||
"FVEY": "Five Eyes (US, UK, Canada, Australia, New Zealand)",
|
||||
"GCHQ": "Government Communications Headquarters (UK)",
|
||||
"GCSB": "Government Communications Security Bureau (New Zealand)",
|
||||
"MI5": "Security Service (UK)",
|
||||
"NATO": "North Atlantic Treaty Organization",
|
||||
"NCTC": "National Counterterrorism Center (US)",
|
||||
"NSA": "National Security Agency (US)",
|
||||
"ODNI": "Office of the Director of National Intelligence (US)",
|
||||
"SIS": "Secret Intelligence Service (UK)",
|
||||
"SWAC": "Southwest Analysis Center",
|
||||
"USG": "United States Government",
|
||||
}
|
||||
|
||||
func agencyName(creator string) string {
|
||||
var out strings.Builder
|
||||
known := false
|
||||
start := 0
|
||||
flush := func(end int) {
|
||||
part := creator[start:end]
|
||||
key := strings.ToUpper(strings.TrimSpace(part))
|
||||
if name, ok := agencyNames[key]; ok {
|
||||
known = true
|
||||
out.WriteString(name)
|
||||
} else {
|
||||
out.WriteString(strings.TrimSpace(part))
|
||||
}
|
||||
}
|
||||
for i := 0; i < len(creator); i++ {
|
||||
switch creator[i] {
|
||||
case '/':
|
||||
flush(i)
|
||||
out.WriteString(" / ")
|
||||
start = i + 1
|
||||
case ',':
|
||||
flush(i)
|
||||
out.WriteString(", ")
|
||||
start = i + 1
|
||||
}
|
||||
}
|
||||
flush(len(creator))
|
||||
if !known {
|
||||
return ""
|
||||
}
|
||||
return out.String()
|
||||
}
|
||||
335
api.go
Normal file
335
api.go
Normal file
|
|
@ -0,0 +1,335 @@
|
|||
// api.go - read-only JSON API mirroring the UI's features.
|
||||
//
|
||||
// Endpoints (all GET):
|
||||
//
|
||||
// /api/docs list documents; params: coll (repeatable:
|
||||
// news, sidtoday-archive, foia), type
|
||||
// (repeatable), sort (any list-page sort: title,
|
||||
// agency, date, pubdate, classification, ...; see
|
||||
// docSorts), order (asc|desc), p, per_page (max
|
||||
// 500, or "all" for exports)
|
||||
// /api/doc/{id} full document: metadata, full text, images,
|
||||
// related IDs, published-together IDs,
|
||||
// version_copies (other published copies: PDF,
|
||||
// page images, transcript, publication indices)
|
||||
// /api/search params: q (simple) OR fqv/fqf/fqc clause
|
||||
// triples (same field codes and value syntax as
|
||||
// the UI: word, "text", =exact, *wildcard*),
|
||||
// plus coll and type (both repeatable, as on
|
||||
// /api/docs; no coll means every collection),
|
||||
// date_doc_start/_end, date_pub_start/_end
|
||||
// (partial END dates are inclusive: 2013 means
|
||||
// through 2013-12-31), sort, order, and the same
|
||||
// p / per_page pagination as /api/docs (default
|
||||
// 50, max 500, "all" for every match without
|
||||
// snippets see apiPageSize). results come
|
||||
// as one flat list, mirroring the search page:
|
||||
// all collections rank equally. a result whose match
|
||||
// is in another published copy of the document
|
||||
// lists that copy in matched_versions (slug,
|
||||
// label, url, snippet)
|
||||
// /api/facets facet maps (whole corpus, like the UI; the
|
||||
// one news-scoped map is news_types) plus
|
||||
// collection counts
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
apiPerPageDefault = 50
|
||||
apiMaxPerPage = 500
|
||||
)
|
||||
|
||||
func writeJSON(w http.ResponseWriter, status int, v interface{}) {
|
||||
b, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
log.Printf("API: marshal error: %v", err)
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
w.Header().Set("Content-Type", "application/json; charset=utf-8")
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
w.Write([]byte(`{"error":"internal error"}`))
|
||||
return
|
||||
}
|
||||
writeJSONBytes(w, status, b)
|
||||
}
|
||||
|
||||
func writeJSONBytes(w http.ResponseWriter, status int, b []byte) {
|
||||
setJSONHeaders(w)
|
||||
w.WriteHeader(status)
|
||||
w.Write(b)
|
||||
}
|
||||
|
||||
func setJSONHeaders(w http.ResponseWriter) {
|
||||
cc := w.Header().Get("Cache-Control")
|
||||
if cc == "" {
|
||||
cc = "public, max-age=60"
|
||||
w.Header().Set("Cache-Control", cc)
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json; charset=utf-8")
|
||||
|
||||
if !strings.Contains(cc, "private") {
|
||||
w.Header().Set("Access-Control-Allow-Origin", "*")
|
||||
}
|
||||
}
|
||||
|
||||
func apiError(w http.ResponseWriter, status int, msg string) {
|
||||
writeJSON(w, status, map[string]string{"error": msg})
|
||||
}
|
||||
|
||||
type apiDocSummary struct {
|
||||
Document
|
||||
URL string `json:"url"`
|
||||
PDF string `json:"pdf"`
|
||||
}
|
||||
|
||||
func docSummary(d Document) apiDocSummary {
|
||||
return apiDocSummary{
|
||||
Document: d,
|
||||
URL: docPath(d.ID),
|
||||
PDF: filePath(d.ID, d.PDFFile),
|
||||
}
|
||||
}
|
||||
|
||||
type apiDocDetail struct {
|
||||
apiDocSummary
|
||||
FullText string `json:"full_text,omitempty"`
|
||||
Images []string `json:"images,omitempty"`
|
||||
Related []string `json:"related,omitempty"`
|
||||
PublishedTogether []string `json:"published_together,omitempty"`
|
||||
VersionCopies []apiVersion `json:"version_copies,omitempty"`
|
||||
}
|
||||
|
||||
type apiSearchResult struct {
|
||||
apiDocSummary
|
||||
Score float64 `json:"score"`
|
||||
Snippet string `json:"snippet,omitempty"`
|
||||
MatchedVersions []apiMatchedVersion `json:"matched_versions,omitempty"`
|
||||
}
|
||||
|
||||
func (co *Corpus) handleAPIDocs(w http.ResponseWriter, r *http.Request) {
|
||||
q := r.URL.Query()
|
||||
|
||||
typeFilter, _ := selectedTypes(q)
|
||||
|
||||
// coll=, repeatable, as /api/search and the pages take it.
|
||||
pool := filterByCollection(co.everyDoc, collectionFilter(q))
|
||||
docs := filterByType(append([]*Document(nil), pool...), typeFilter)
|
||||
|
||||
sortDocs(docs, q.Get("sort"), q.Get("order"))
|
||||
|
||||
perPage := apiPageSize(q)
|
||||
page, _ := strconv.Atoi(q.Get("p"))
|
||||
start, end, page, totalPages := paginate(len(docs), page, perPage)
|
||||
|
||||
out := make([]apiDocSummary, 0, end-start)
|
||||
for _, d := range docs[start:end] {
|
||||
out = append(out, docSummary(*d))
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, map[string]interface{}{
|
||||
"total": len(docs),
|
||||
"page": page,
|
||||
"per_page": apiRespPerPage(perPage, len(docs)),
|
||||
"total_pages": totalPages,
|
||||
"docs": out,
|
||||
})
|
||||
}
|
||||
|
||||
func apiPageSize(q url.Values) int {
|
||||
if q.Get("per_page") == "all" {
|
||||
return 0
|
||||
}
|
||||
if v, err := strconv.Atoi(q.Get("per_page")); err == nil && v > 0 {
|
||||
if v > apiMaxPerPage {
|
||||
return apiMaxPerPage
|
||||
}
|
||||
return v
|
||||
}
|
||||
return apiPerPageDefault
|
||||
}
|
||||
|
||||
func apiRespPerPage(perPage, total int) int {
|
||||
if perPage > 0 {
|
||||
return perPage
|
||||
}
|
||||
if total == 0 {
|
||||
return 1
|
||||
}
|
||||
return total
|
||||
}
|
||||
|
||||
func (co *Corpus) handleAPIDoc(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
doc := co.docIndex[id]
|
||||
if doc == nil {
|
||||
apiError(w, http.StatusNotFound, "document not found")
|
||||
return
|
||||
}
|
||||
|
||||
var pubIDs []string
|
||||
for _, d := range co.publishedTogether(doc) {
|
||||
pubIDs = append(pubIDs, d.ID)
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, apiDocDetail{
|
||||
apiDocSummary: docSummary(*doc),
|
||||
FullText: doc.FullText,
|
||||
Images: doc.Images,
|
||||
Related: co.archive.RelatedDocs[id],
|
||||
PublishedTogether: pubIDs,
|
||||
VersionCopies: apiVersions(doc),
|
||||
})
|
||||
}
|
||||
|
||||
func (co *Corpus) handleAPISearch(w http.ResponseWriter, r *http.Request) {
|
||||
q := r.URL.Query()
|
||||
|
||||
sp := readSearchParams(q)
|
||||
if len(sp.Queries) == 0 {
|
||||
apiError(w, http.StatusBadRequest,
|
||||
"missing query: pass ?q=term or fqv/fqf/fqc clause parameters")
|
||||
return
|
||||
}
|
||||
if sp.OverClauses > 0 {
|
||||
apiError(w, http.StatusBadRequest, fmt.Sprintf(
|
||||
"too many clauses: %d (the limit is %d)", sp.OverClauses, maxSearchClauses))
|
||||
return
|
||||
}
|
||||
if sp.OverWords > 0 {
|
||||
apiError(w, http.StatusBadRequest, fmt.Sprintf(
|
||||
"too many words: %d (the limit is %d)", sp.OverWords, maxSearchWords))
|
||||
return
|
||||
}
|
||||
scored, _ := co.executeSearch(sp)
|
||||
snippetTerm, _ := snippetAnchor(sp.Queries, sp.Fields, sp.Conjs)
|
||||
|
||||
perPage := apiPageSize(q)
|
||||
page, _ := strconv.Atoi(q.Get("p"))
|
||||
start, end, page, totalPages := paginate(len(scored), page, perPage)
|
||||
withSnippets := snippetTerm != "" && perPage > 0
|
||||
|
||||
results := make([]apiSearchResult, 0, end-start)
|
||||
for _, s := range scored[start:end] {
|
||||
res := apiSearchResult{apiDocSummary: docSummary(*s.Doc), Score: s.Score}
|
||||
if withSnippets {
|
||||
if body := s.Doc.bodySnippets(snippetTerm, 1); len(body) > 0 {
|
||||
res.Snippet = body[0]
|
||||
}
|
||||
for _, h := range versionHits(s.Doc, snippetTerm, 1) {
|
||||
res.MatchedVersions = append(res.MatchedVersions, apiMatchedVersion{
|
||||
Slug: h.Version.Slug,
|
||||
Label: h.Version.Label,
|
||||
URL: versionPath(s.Doc.ID, h.Version.Slug),
|
||||
Snippet: h.Snippets[0],
|
||||
})
|
||||
}
|
||||
}
|
||||
results = append(results, res)
|
||||
}
|
||||
|
||||
body := map[string]interface{}{
|
||||
"total": len(scored),
|
||||
"page": page,
|
||||
"per_page": apiRespPerPage(perPage, len(scored)),
|
||||
"total_pages": totalPages,
|
||||
"results": results,
|
||||
}
|
||||
if snippetTerm != "" && !withSnippets {
|
||||
body["snippets_omitted"] = true
|
||||
}
|
||||
writeJSON(w, http.StatusOK, body)
|
||||
}
|
||||
|
||||
func (co *Corpus) buildFacetsJSON() {
|
||||
b, err := json.Marshal(map[string]interface{}{
|
||||
"note": "facet counts cover the whole corpus; news_types covers the news collection only",
|
||||
"collections": co.archive.CollectionCounts,
|
||||
"sidtoday_total": co.archive.SIDTodayCount,
|
||||
"types": co.archive.Types,
|
||||
"news_types": co.archive.NewsTypes,
|
||||
"agencies": co.archive.Facets["agency"],
|
||||
"programs": co.archive.Facets["program"],
|
||||
"journalists": co.archive.Facets["journalist"],
|
||||
"publishers": co.archive.Facets["publisher"],
|
||||
"countries": co.archive.Facets["country"],
|
||||
"people": co.archive.Facets["people"],
|
||||
"classification": co.archive.Facets["classification"],
|
||||
"dissemination": co.archive.Facets["dissemination"],
|
||||
"banners": co.archive.Facets["banner"],
|
||||
"genres": co.archive.Facets["genre"],
|
||||
"series": co.archive.Facets["series"],
|
||||
"tags": co.archive.Facets["tag"],
|
||||
"pub_years": co.archive.Facets["published"],
|
||||
"doc_years": co.archive.Facets["docdate"],
|
||||
"by_collection": co.archive.FacetsByColl,
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatalf("facets: marshal failed: %v", err)
|
||||
}
|
||||
co.facetsJSON = b
|
||||
sum := sha256.Sum256(b)
|
||||
co.facetsETag = `"` + hex.EncodeToString(sum[:8]) + `"`
|
||||
log.Printf("Facets JSON pre-built: %d bytes", len(co.facetsJSON))
|
||||
}
|
||||
|
||||
func (co *Corpus) handleAPIFacets(w http.ResponseWriter, r *http.Request) {
|
||||
setJSONHeaders(w)
|
||||
w.Header().Set("ETag", co.facetsETag)
|
||||
http.ServeContent(w, r, "", time.Time{}, bytes.NewReader(co.facetsJSON))
|
||||
}
|
||||
|
||||
// "version" as in document versions (metadata.json can have "versions" object)
|
||||
type apiVersion struct {
|
||||
Slug string `json:"slug"`
|
||||
URL string `json:"url"`
|
||||
PDF string `json:"pdf"`
|
||||
PDFSha256 string `json:"pdf_sha256,omitempty"`
|
||||
Images []string `json:"images,omitempty"`
|
||||
FullText string `json:"full_text,omitempty"`
|
||||
Publications []int `json:"publications,omitempty"`
|
||||
ChangedPages []int `json:"changed_pages,omitempty"`
|
||||
}
|
||||
|
||||
func apiVersions(doc *Document) []apiVersion {
|
||||
var out []apiVersion
|
||||
for _, v := range doc.Versions {
|
||||
out = append(out, apiVersion{
|
||||
Slug: v.Slug,
|
||||
URL: versionPath(doc.ID, v.Slug),
|
||||
PDF: versionFilePath(doc.ID, v.Slug, v.PDFFile),
|
||||
PDFSha256: v.SHA,
|
||||
Images: v.Images,
|
||||
FullText: v.FullText,
|
||||
Publications: v.PubIndex,
|
||||
ChangedPages: changedPageNumbers(v.ChangedPages),
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
type apiMatchedVersion struct {
|
||||
Slug string `json:"slug"`
|
||||
Label string `json:"label"`
|
||||
URL string `json:"url"`
|
||||
Snippet string `json:"snippet"`
|
||||
}
|
||||
|
||||
func changedPageNumbers(pages []ChangedPage) []int {
|
||||
var out []int
|
||||
for _, p := range pages {
|
||||
out = append(out, p.Page)
|
||||
}
|
||||
return out
|
||||
}
|
||||
211
articles.go
Normal file
211
articles.go
Normal file
|
|
@ -0,0 +1,211 @@
|
|||
// articles.go - news articles as first-class objects.
|
||||
//
|
||||
// An article's identity is its publication LINK. Giving a publication entry
|
||||
// a title is what promotes its article to a story.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"slices"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type Article struct {
|
||||
Key string // short stable hash of the link (the /article/{key} path)
|
||||
Title string
|
||||
Link string
|
||||
Host string // display fallback when Title is empty
|
||||
Publisher string
|
||||
Date string // earliest date_published seen
|
||||
Journalists []string // union across docs, first-seen order
|
||||
DocIDs []string
|
||||
Book bool // published by a book publisher, not a news outlet
|
||||
}
|
||||
|
||||
// bookPublishers are the publication entries that are books, not news
|
||||
// articles (Greenwald's No Place to Hide, Gellman's Dark Mirror, Poitras's
|
||||
// Astro Noise).
|
||||
var bookPublishers = map[string]bool{
|
||||
"Metropolitan Books": true,
|
||||
"Penguin Press": true,
|
||||
"Yale University Press": true,
|
||||
}
|
||||
|
||||
// articleKeyFor derives the stable URL key for a link
|
||||
func articleKeyFor(link string) string {
|
||||
h := sha256.Sum256([]byte(link))
|
||||
return hex.EncodeToString(h[:])[:12]
|
||||
}
|
||||
|
||||
func hostOf(link string) string {
|
||||
u, err := url.Parse(link)
|
||||
if err != nil || u.Host == "" {
|
||||
return link
|
||||
}
|
||||
return strings.TrimPrefix(u.Host, "www.")
|
||||
}
|
||||
|
||||
// buildArticles indexes every publication entry that has a link
|
||||
func (co *Corpus) buildArticles() {
|
||||
co.articleByLink = make(map[string]*Article)
|
||||
co.articleIndex = make(map[string]*Article)
|
||||
co.articles = nil
|
||||
|
||||
for _, d := range co.archive.Docs {
|
||||
for _, p := range d.Publication {
|
||||
if p.Link == "" {
|
||||
continue
|
||||
}
|
||||
a := co.articleByLink[p.Link]
|
||||
if a == nil {
|
||||
a = &Article{
|
||||
Key: articleKeyFor(p.Link),
|
||||
Link: p.Link,
|
||||
Host: hostOf(p.Link),
|
||||
}
|
||||
co.articleByLink[p.Link] = a
|
||||
co.articleIndex[a.Key] = a
|
||||
}
|
||||
if a.Title == "" && p.Title != "" {
|
||||
a.Title = p.Title
|
||||
}
|
||||
if a.Publisher == "" && p.Publisher != "" {
|
||||
a.Publisher = p.Publisher
|
||||
a.Book = bookPublishers[p.Publisher]
|
||||
}
|
||||
if p.DatePublished != "" && (a.Date == "" || p.DatePublished < a.Date) {
|
||||
a.Date = p.DatePublished
|
||||
}
|
||||
for _, j := range p.Journalists {
|
||||
if j == "" {
|
||||
continue
|
||||
}
|
||||
if !slices.ContainsFunc(a.Journalists, func(x string) bool { return sameValue(x, j) }) {
|
||||
a.Journalists = append(a.Journalists, j)
|
||||
}
|
||||
}
|
||||
dup := false
|
||||
for _, id := range a.DocIDs {
|
||||
if id == d.ID {
|
||||
dup = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !dup {
|
||||
a.DocIDs = append(a.DocIDs, d.ID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for _, a := range co.articleByLink {
|
||||
co.articles = append(co.articles, *a)
|
||||
}
|
||||
// Newest first; undated articles sort last; ties by title then link so
|
||||
// the order is deterministic.
|
||||
sort.Slice(co.articles, func(i, j int) bool {
|
||||
di, dj := co.articles[i].Date, co.articles[j].Date
|
||||
if di == "" && dj != "" {
|
||||
return false
|
||||
}
|
||||
if di != "" && dj == "" {
|
||||
return true
|
||||
}
|
||||
if di != dj {
|
||||
return di > dj
|
||||
}
|
||||
ti := strings.ToLower(displayTitle(co.articles[i]))
|
||||
tj := strings.ToLower(displayTitle(co.articles[j]))
|
||||
if ti != tj {
|
||||
return ti < tj
|
||||
}
|
||||
return co.articles[i].Link < co.articles[j].Link
|
||||
})
|
||||
|
||||
co.buildArticleGroups()
|
||||
}
|
||||
|
||||
// inCollection reports whether one of the article's documents is in coll.
|
||||
func (a *Article) inCollection(co *Corpus, coll string) bool {
|
||||
for _, id := range a.DocIDs {
|
||||
if d := co.docIndex[id]; d != nil && d.Collection == coll {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func displayTitle(a Article) string {
|
||||
if a.Title != "" {
|
||||
return a.Title
|
||||
}
|
||||
return a.Host
|
||||
}
|
||||
|
||||
type articleYearGroup struct {
|
||||
Year string
|
||||
Articles []Article
|
||||
}
|
||||
|
||||
func (co *Corpus) buildArticleGroups() {
|
||||
co.articleGroups, co.articlesTitled, co.articlesBooks = nil, 0, 0
|
||||
for _, a := range co.articles {
|
||||
if a.Title == "" {
|
||||
continue
|
||||
}
|
||||
co.articlesTitled++
|
||||
if a.Book {
|
||||
co.articlesBooks++
|
||||
}
|
||||
year := "Undated"
|
||||
if len(a.Date) >= 4 {
|
||||
year = a.Date[:4]
|
||||
}
|
||||
if len(co.articleGroups) == 0 || co.articleGroups[len(co.articleGroups)-1].Year != year {
|
||||
co.articleGroups = append(co.articleGroups, articleYearGroup{Year: year})
|
||||
}
|
||||
co.articleGroups[len(co.articleGroups)-1].Articles = append(co.articleGroups[len(co.articleGroups)-1].Articles, a)
|
||||
}
|
||||
}
|
||||
|
||||
func (co *Corpus) handleArticles(w http.ResponseWriter, r *http.Request) {
|
||||
render(w, r, &articlesPage{
|
||||
Groups: co.articleGroups,
|
||||
GroupsHTML: co.preHTML("articles"),
|
||||
Total: co.articlesTitled - co.articlesBooks,
|
||||
Books: co.articlesBooks,
|
||||
})
|
||||
}
|
||||
|
||||
func (co *Corpus) handleArticleView(w http.ResponseWriter, r *http.Request) {
|
||||
a := co.articleIndex[r.PathValue("key")]
|
||||
if a == nil {
|
||||
notFound(w, r)
|
||||
return
|
||||
}
|
||||
prefs := readPrefs(r)
|
||||
scope := pageScope(nil, prefs, false)
|
||||
docs := make([]*Document, 0, len(a.DocIDs))
|
||||
for _, id := range a.DocIDs {
|
||||
if d := co.docIndex[id]; d != nil {
|
||||
docs = append(docs, d)
|
||||
scope = scope.including(d.Collection)
|
||||
}
|
||||
}
|
||||
data := &articlePage{
|
||||
Base: pageWith(prefs),
|
||||
Article: *a,
|
||||
Total: len(docs),
|
||||
BrowseQS: scope.suffix(),
|
||||
Cards: newCardView(prefs, docs),
|
||||
}
|
||||
// A partner that only reported the story has no publisher page.
|
||||
if scope.hasValue(co, "publisher", a.Publisher) {
|
||||
data.PublisherURL = scope.link(categoryValueURL("publisher", a.Publisher))
|
||||
}
|
||||
render(w, r, data)
|
||||
}
|
||||
273
assemblies.go
Normal file
273
assemblies.go
Normal file
|
|
@ -0,0 +1,273 @@
|
|||
// assemblies.go - hand-built sets of pages drawn from several documents,
|
||||
// loaded from data/assemblies.json.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type assemblyRef struct {
|
||||
Doc string `json:"doc"`
|
||||
Version string `json:"version,omitempty"`
|
||||
Pages string `json:"pages"`
|
||||
}
|
||||
|
||||
type assemblySlideSpec struct {
|
||||
assemblyRef
|
||||
Note string `json:"note,omitempty"`
|
||||
Alt []assemblyRef `json:"alt,omitempty"`
|
||||
}
|
||||
|
||||
type assemblySpec struct {
|
||||
ID string `json:"id"`
|
||||
Title string `json:"title"`
|
||||
Note string `json:"note,omitempty"`
|
||||
Slides []assemblySlideSpec `json:"slides"`
|
||||
}
|
||||
|
||||
type AssemblyCopy struct {
|
||||
Image string
|
||||
Size imageSize
|
||||
DocID string
|
||||
DocTitle string
|
||||
Version string
|
||||
Page int
|
||||
Source string // the outlets that published this copy
|
||||
Path string // the document or version page it came from
|
||||
}
|
||||
|
||||
type AssemblySlide struct {
|
||||
AssemblyCopy
|
||||
Pos int
|
||||
Note string
|
||||
Alts []AssemblyCopy
|
||||
}
|
||||
|
||||
type Assembly struct {
|
||||
ID string
|
||||
Title string
|
||||
Note string
|
||||
Slides []AssemblySlide
|
||||
DocList []string // documents the slides are shown from, in slide order
|
||||
Copies int // copies the slides are shown from: a document or one of its versions
|
||||
}
|
||||
|
||||
// parsePageList expands "1-3,7" into page numbers, keeping the order given.
|
||||
func parsePageList(s string) ([]int, error) {
|
||||
var out []int
|
||||
for _, part := range strings.Split(s, ",") {
|
||||
part = strings.TrimSpace(part)
|
||||
first, last, isRange := strings.Cut(part, "-")
|
||||
if !isRange {
|
||||
last = first
|
||||
}
|
||||
a, err := strconv.Atoi(strings.TrimSpace(first))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%q is not a page or a page range", part)
|
||||
}
|
||||
b, err := strconv.Atoi(strings.TrimSpace(last))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%q is not a page or a page range", part)
|
||||
}
|
||||
if a < 1 || b < a {
|
||||
return nil, fmt.Errorf("%q is not a page or a page range", part)
|
||||
}
|
||||
for n := a; n <= b; n++ {
|
||||
out = append(out, n)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// copySource names the outlets that published a copy, oldest entry first as
|
||||
// the metadata lists them. Partners that published no documents are left out.
|
||||
func copySource(d *Document, slug string) string {
|
||||
var names []string
|
||||
seen := map[string]bool{}
|
||||
for _, p := range d.Publication {
|
||||
key := valueKey(p.Publisher)
|
||||
if p.ReportingOnly || p.Version != slug || key == "" || seen[key] {
|
||||
continue
|
||||
}
|
||||
seen[key] = true
|
||||
names = append(names, p.Publisher)
|
||||
}
|
||||
return strings.Join(names, ", ")
|
||||
}
|
||||
|
||||
func (co *Corpus) resolveAssemblyRef(label string, r assemblyRef) ([]*AssemblyCopy, bool) {
|
||||
pages, err := parsePageList(r.Pages)
|
||||
if err != nil {
|
||||
warnf("%s: %s: pages: %v - skipped", label, r.Doc, err)
|
||||
return nil, false
|
||||
}
|
||||
out := make([]*AssemblyCopy, len(pages))
|
||||
d := co.docIndex[r.Doc]
|
||||
if d == nil {
|
||||
warnf("%s: no document %q - skipped", label, r.Doc)
|
||||
return out, true
|
||||
}
|
||||
images, sizes, base, path := d.Images, d.ImageSizes, imageBase(d.ID, ""), docPath(d.ID)
|
||||
if r.Version != "" {
|
||||
v := d.VersionFor(r.Version)
|
||||
if v == nil {
|
||||
warnf("%s: %s has no version %q - skipped", label, r.Doc, r.Version)
|
||||
return out, true
|
||||
}
|
||||
images, sizes, base, path = v.Images, v.ImageSizes, imageBase(d.ID, v.Slug), versionPath(d.ID, v.Slug)
|
||||
}
|
||||
byNumber := make(map[int]int, len(images))
|
||||
for i, name := range images {
|
||||
if n, err := pageNumber(name); err == nil {
|
||||
byNumber[n] = i
|
||||
}
|
||||
}
|
||||
source := copySource(d, r.Version)
|
||||
for i, n := range pages {
|
||||
k, ok := byNumber[n]
|
||||
if !ok {
|
||||
warnf("%s: %s %s has no page image %d - skipped", label, r.Doc, r.Version, n)
|
||||
continue
|
||||
}
|
||||
out[i] = &AssemblyCopy{
|
||||
Image: base + url.PathEscape(images[k]), Size: sizes[k], DocID: d.ID, DocTitle: d.Title,
|
||||
Version: r.Version, Page: n, Source: source, Path: path,
|
||||
}
|
||||
}
|
||||
return out, true
|
||||
}
|
||||
|
||||
func (co *Corpus) buildAssembly(spec assemblySpec) *Assembly {
|
||||
label := "assemblies.json: " + spec.ID
|
||||
a := &Assembly{ID: spec.ID, Title: strings.TrimSpace(spec.Title), Note: strings.TrimSpace(spec.Note)}
|
||||
seenImage := map[string]bool{}
|
||||
seenDoc := map[string]bool{}
|
||||
seenCopy := map[[2]string]bool{}
|
||||
for _, s := range spec.Slides {
|
||||
copies, ok := co.resolveAssemblyRef(label, s.assemblyRef)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
alts := make([][]*AssemblyCopy, 0, len(s.Alt))
|
||||
for _, ref := range s.Alt {
|
||||
c, ok := co.resolveAssemblyRef(label, ref)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if len(c) != len(copies) {
|
||||
warnf("%s: alt %s lists %d pages for a run of %d - skipped", label, ref.Doc, len(c), len(copies))
|
||||
continue
|
||||
}
|
||||
alts = append(alts, c)
|
||||
}
|
||||
for i, c := range copies {
|
||||
if c == nil {
|
||||
continue
|
||||
}
|
||||
if seenImage[c.Image] {
|
||||
warnf("%s: %s page %d is listed twice", label, c.DocID, c.Page)
|
||||
}
|
||||
seenImage[c.Image] = true
|
||||
if !seenDoc[c.DocID] {
|
||||
seenDoc[c.DocID] = true
|
||||
a.DocList = append(a.DocList, c.DocID)
|
||||
}
|
||||
slide := AssemblySlide{AssemblyCopy: *c, Pos: len(a.Slides) + 1, Note: s.Note}
|
||||
seenCopy[[2]string{c.DocID, c.Version}] = true
|
||||
for _, alt := range alts {
|
||||
if alt[i] != nil {
|
||||
slide.Alts = append(slide.Alts, *alt[i])
|
||||
}
|
||||
}
|
||||
a.Slides = append(a.Slides, slide)
|
||||
}
|
||||
}
|
||||
a.Copies = len(seenCopy)
|
||||
return a
|
||||
}
|
||||
|
||||
func (co *Corpus) loadAssemblies() {
|
||||
co.assemblies = nil
|
||||
co.assemblyIndex = map[string]*Assembly{}
|
||||
co.assembliesByDoc = map[string][]*Assembly{}
|
||||
|
||||
p := projectPath("assemblies.json")
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
log.Printf("Note: %s not found - no assemblies.", p)
|
||||
return
|
||||
}
|
||||
var specs []assemblySpec
|
||||
if err := json.Unmarshal(b, &specs); err != nil {
|
||||
warnf("%s is malformed: %v - assemblies disabled.", p, err)
|
||||
return
|
||||
}
|
||||
for _, spec := range specs {
|
||||
spec.ID = strings.TrimSpace(spec.ID)
|
||||
if spec.ID == "" || !docIDSafe(spec.ID) {
|
||||
warnf("assemblies.json: id %q is empty or unsafe in a URL - skipped", spec.ID)
|
||||
continue
|
||||
}
|
||||
if co.assemblyIndex[spec.ID] != nil {
|
||||
warnf("assemblies.json: id %q appears twice - keeping the first", spec.ID)
|
||||
continue
|
||||
}
|
||||
a := co.buildAssembly(spec)
|
||||
if len(a.Slides) == 0 {
|
||||
warnf("assemblies.json: %s has no slides that can be shown - skipped", spec.ID)
|
||||
continue
|
||||
}
|
||||
co.assemblies = append(co.assemblies, a)
|
||||
co.assemblyIndex[a.ID] = a
|
||||
linked := map[string]bool{}
|
||||
link := func(id string) {
|
||||
if !linked[id] {
|
||||
linked[id] = true
|
||||
co.assembliesByDoc[id] = append(co.assembliesByDoc[id], a)
|
||||
}
|
||||
}
|
||||
for _, s := range a.Slides {
|
||||
link(s.DocID)
|
||||
for _, alt := range s.Alts {
|
||||
link(alt.DocID)
|
||||
}
|
||||
}
|
||||
}
|
||||
log.Printf("Assemblies loaded: %d", len(co.assemblies))
|
||||
}
|
||||
|
||||
// assemblyPath is an assembly's page; assemblies live under /explore.
|
||||
const assembliesPath = "/explore/assemblies"
|
||||
|
||||
func assemblyPath(id string) string { return assembliesPath + "/" + url.PathEscape(id) }
|
||||
|
||||
func (co *Corpus) handleAssemblies(w http.ResponseWriter, r *http.Request) {
|
||||
prefs := readPrefs(r)
|
||||
render(w, r, &assembliesPage{
|
||||
Base: pageWith(prefs),
|
||||
Assemblies: co.assemblies,
|
||||
TextOnly: prefs.TextOnly,
|
||||
})
|
||||
}
|
||||
|
||||
func (co *Corpus) handleAssemblyView(w http.ResponseWriter, r *http.Request) {
|
||||
a := co.assemblyIndex[r.PathValue("id")]
|
||||
if a == nil {
|
||||
notFound(w, r)
|
||||
return
|
||||
}
|
||||
prefs := readPrefs(r)
|
||||
render(w, r, &assemblyPage{
|
||||
Base: pageWith(prefs),
|
||||
Assembly: a,
|
||||
TextOnly: prefs.TextOnly,
|
||||
})
|
||||
}
|
||||
102
changelog.go
Normal file
102
changelog.go
Normal file
|
|
@ -0,0 +1,102 @@
|
|||
// changelog.go - the archive changelog (/changelog).
|
||||
//
|
||||
// The changelog is a HAND-MAINTAINED file: data/changelog.json. Shape:
|
||||
//
|
||||
// [
|
||||
// {
|
||||
// "date": "2026-06-11", // YYYY-MM-DD (or full RFC3339 if you care)
|
||||
// "type": "added", // added | removed | updated | note
|
||||
// "id": "some-doc-id", // which document (omit for type=note)
|
||||
// "title": "Document title", // shown in the headline (omit for note)
|
||||
// "components": ["pdf"], // optional, for updated: what changed
|
||||
// "note": "free text" // the body for type=note, optional otherwise
|
||||
// }
|
||||
// ]
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
type ChangeEntry struct {
|
||||
Date string `json:"date"`
|
||||
Type string `json:"type"`
|
||||
ID string `json:"id,omitempty"`
|
||||
Title string `json:"title,omitempty"`
|
||||
Components []string `json:"components,omitempty"`
|
||||
Note string `json:"note,omitempty"`
|
||||
}
|
||||
|
||||
func (co *Corpus) loadChangelog() {
|
||||
p := projectPath("changelog.json")
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
log.Printf("Note: %s not found - the changelog will be empty (see the format at the top of changelog.go).", p)
|
||||
return
|
||||
}
|
||||
if err := json.Unmarshal(b, &co.changelog); err != nil {
|
||||
warnf("%s is malformed: %v - changelog disabled.", p, err)
|
||||
co.changelog = nil
|
||||
return
|
||||
}
|
||||
sort.SliceStable(co.changelog, func(i, j int) bool {
|
||||
return co.changelog[i].Date > co.changelog[j].Date
|
||||
})
|
||||
for _, e := range co.changelog {
|
||||
if !validChangelogDate(e.Date) {
|
||||
warnf("changelog entry %q has date %q - must be YYYY-MM-DD or RFC3339", e.Headline(), e.Date)
|
||||
}
|
||||
if e.Type != "removed" && e.ID != "" && co.docIndex[e.ID] == nil {
|
||||
warnf("changelog entry %q names document %q, which is not in the corpus", e.Headline(), e.ID)
|
||||
}
|
||||
}
|
||||
log.Printf("Changelog loaded: %d entries", len(co.changelog))
|
||||
}
|
||||
|
||||
func validChangelogDate(d string) bool {
|
||||
if _, err := time.Parse("2006-01-02", d); err == nil {
|
||||
return true
|
||||
}
|
||||
_, err := time.Parse(time.RFC3339, d)
|
||||
return err == nil
|
||||
}
|
||||
|
||||
func (e ChangeEntry) Headline() string {
|
||||
switch e.Type {
|
||||
case "added":
|
||||
return "Added: " + e.Title
|
||||
case "removed":
|
||||
return "Removed: " + e.Title
|
||||
case "updated":
|
||||
s := "Updated: " + e.Title
|
||||
if len(e.Components) > 0 {
|
||||
s += " (" + strings.Join(e.Components, ", ") + ")"
|
||||
}
|
||||
return s
|
||||
default: // note
|
||||
if e.Note != "" {
|
||||
return e.Note
|
||||
}
|
||||
return "Archive note"
|
||||
}
|
||||
}
|
||||
|
||||
func (e ChangeEntry) DocURL() string {
|
||||
if e.ID != "" && e.Type != "removed" {
|
||||
return docPath(e.ID)
|
||||
}
|
||||
return "/changelog"
|
||||
}
|
||||
|
||||
func (co *Corpus) handleChangelog(w http.ResponseWriter, r *http.Request) {
|
||||
render(w, r, &changelogPage{
|
||||
Entries: co.changelog,
|
||||
Total: len(co.changelog),
|
||||
})
|
||||
}
|
||||
298
codewords.go
Normal file
298
codewords.go
Normal file
|
|
@ -0,0 +1,298 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"html/template"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// The codeword concordance: an automatically extracted index of ALL-CAPS
|
||||
// tokens across the corpus - program names, compartments, org designators.
|
||||
|
||||
var codewordStoplist = map[string]bool{
|
||||
"REDACTED": true,
|
||||
"DYNAMIC": true,
|
||||
"PAGE": true,
|
||||
"HIGHEST": true,
|
||||
"POSSIBLE": true,
|
||||
"DERIVED": true,
|
||||
"DECLASSIFY": true,
|
||||
"JAN": true,
|
||||
"ADDED": true,
|
||||
"CLASSIFICATION": true,
|
||||
}
|
||||
|
||||
type codewordEntry struct {
|
||||
Token string
|
||||
Count int // total occurrences across the corpus
|
||||
Docs int // number of documents containing it
|
||||
First string // earliest document year containing it ("" if only undated docs)
|
||||
Last string // latest such year
|
||||
}
|
||||
|
||||
func isCodewordToken(w string) bool {
|
||||
if len(w) < 3 || w[0] < 'A' || w[0] > 'Z' {
|
||||
return false
|
||||
}
|
||||
letters := 0
|
||||
var seen [26]bool
|
||||
distinct := 0
|
||||
for i := 0; i < len(w); i++ {
|
||||
c := w[i]
|
||||
switch {
|
||||
case c >= 'A' && c <= 'Z':
|
||||
letters++
|
||||
if !seen[c-'A'] {
|
||||
seen[c-'A'] = true
|
||||
distinct++
|
||||
}
|
||||
case c >= '0' && c <= '9':
|
||||
// digits allowed
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
return letters >= 3 && distinct >= 2
|
||||
}
|
||||
|
||||
func scanRuns(text string, mention func(w string)) {
|
||||
start := -1
|
||||
for j := 0; j <= len(text); j++ {
|
||||
alnum := false
|
||||
if j < len(text) {
|
||||
ch := text[j]
|
||||
alnum = (ch >= 'A' && ch <= 'Z') || (ch >= 'a' && ch <= 'z') || (ch >= '0' && ch <= '9')
|
||||
}
|
||||
if alnum {
|
||||
if start < 0 {
|
||||
start = j
|
||||
}
|
||||
continue
|
||||
}
|
||||
if start >= 0 {
|
||||
mention(text[start:j])
|
||||
start = -1
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (co *Corpus) buildCodewords() {
|
||||
co.codewords, co.codewordTable = nil, nil
|
||||
caps := map[string]int{}
|
||||
other := map[string]int{}
|
||||
docCaps := make([][]string, len(co.archive.Docs))
|
||||
for i := range co.archive.Docs {
|
||||
d := &co.archive.Docs[i]
|
||||
here := map[string]bool{}
|
||||
count := func(w string) {
|
||||
switch {
|
||||
case codewordStoplist[w]:
|
||||
// boilerplate or archive markup (codewordStoplist)
|
||||
case isCodewordToken(w):
|
||||
caps[w]++
|
||||
here[w] = true
|
||||
default:
|
||||
other[strings.ToLower(w)]++
|
||||
}
|
||||
}
|
||||
scanRuns(d.Title, count)
|
||||
scanRuns(d.FullText, count)
|
||||
for w := range here {
|
||||
docCaps[i] = append(docCaps[i], w)
|
||||
}
|
||||
}
|
||||
for w, cnt := range caps {
|
||||
if cnt < 2 || cnt <= other[strings.ToLower(w)] {
|
||||
continue
|
||||
}
|
||||
co.codewords = append(co.codewords, codewordEntry{Token: w, Count: cnt})
|
||||
}
|
||||
kept := make(map[string]*codewordEntry, len(co.codewords))
|
||||
for i := range co.codewords {
|
||||
kept[co.codewords[i].Token] = &co.codewords[i]
|
||||
}
|
||||
|
||||
for i := range co.archive.Docs {
|
||||
d := &co.archive.Docs[i]
|
||||
year := docYear(d)
|
||||
inDoc := map[string]bool{}
|
||||
exact := func(w string) {
|
||||
if e := kept[w]; e != nil {
|
||||
inDoc[e.Token] = true
|
||||
}
|
||||
}
|
||||
folded := func(w string) {
|
||||
if e := kept[strings.ToUpper(w)]; e != nil {
|
||||
inDoc[e.Token] = true
|
||||
}
|
||||
}
|
||||
for _, w := range docCaps[i] {
|
||||
exact(w)
|
||||
}
|
||||
scanRuns(d.Description, exact)
|
||||
scanRuns(d.LongDescription, exact)
|
||||
for _, p := range d.DocMeta.Programs {
|
||||
scanRuns(p, folded)
|
||||
}
|
||||
for _, t := range d.Tags {
|
||||
scanRuns(t, folded)
|
||||
}
|
||||
for w := range inDoc {
|
||||
e := kept[w]
|
||||
e.Docs++
|
||||
if year == "" {
|
||||
continue
|
||||
}
|
||||
if e.First == "" || year < e.First {
|
||||
e.First = year
|
||||
}
|
||||
if year > e.Last {
|
||||
e.Last = year
|
||||
}
|
||||
}
|
||||
}
|
||||
sort.Slice(co.codewords, func(i, j int) bool {
|
||||
if co.codewords[i].Count != co.codewords[j].Count {
|
||||
return co.codewords[i].Count > co.codewords[j].Count
|
||||
}
|
||||
return co.codewords[i].Token < co.codewords[j].Token
|
||||
})
|
||||
|
||||
lowerProg := make(map[string]string, len(co.archive.Facets["program"]))
|
||||
for p := range co.archive.Facets["program"] {
|
||||
lowerProg[valueKey(p)] = p
|
||||
}
|
||||
shown := co.codewords
|
||||
if len(shown) > codewordRows {
|
||||
shown = shown[:codewordRows]
|
||||
}
|
||||
co.codewordTable = make([]codewordRow, 0, len(shown))
|
||||
for _, e := range shown {
|
||||
row := codewordRow{codewordEntry: e, Gloss: co.glossaryLookup(e.Token), SearchURL: tokenSearchURL(e.Token)}
|
||||
if p, ok := lowerProg[valueKey(e.Token)]; ok {
|
||||
row.ProgURL = wholeArchive.link(categoryValueURL("program", p))
|
||||
}
|
||||
if e.First != "" {
|
||||
row.FirstURL = yearTokenURL(e.Token, e.First)
|
||||
}
|
||||
if e.Last != "" {
|
||||
row.LastURL = yearTokenURL(e.Token, e.Last)
|
||||
}
|
||||
if row.Gloss != nil {
|
||||
row.GlossURL = template.URL("/glossary#term-" + url.PathEscape(row.Gloss.Term))
|
||||
}
|
||||
co.codewordTable = append(co.codewordTable, row)
|
||||
}
|
||||
co.buildCodewordViews()
|
||||
}
|
||||
|
||||
func yearTokenURL(token, year string) template.URL {
|
||||
return template.URL(wholeArchive.link(dateRangeSearchURL("TE", exactClause(token), [2]string{"date_doc", year})))
|
||||
}
|
||||
|
||||
func tokenSearchURL(token string) template.URL {
|
||||
return template.URL(wholeArchive.link("/search?q=" + url.QueryEscape(exactClause(token))))
|
||||
}
|
||||
|
||||
type codewordRow struct {
|
||||
codewordEntry
|
||||
Gloss *GlossaryEntry
|
||||
ProgURL string
|
||||
SearchURL template.URL
|
||||
FirstURL, LastURL template.URL
|
||||
GlossURL template.URL
|
||||
}
|
||||
|
||||
// codewordRows caps the page
|
||||
const codewordRows = 500
|
||||
|
||||
var codewordCols = []sortCol{
|
||||
{Key: "token", Label: "Token"},
|
||||
{Key: "count", Label: "Occurrences", DescFirst: true},
|
||||
{Key: "docs", Label: "Documents", DescFirst: true},
|
||||
{Key: "first", Label: "First seen"},
|
||||
{Key: "last", Label: "Last seen", DescFirst: true},
|
||||
{Key: "gloss", Label: "Glossary"},
|
||||
}
|
||||
|
||||
func (co *Corpus) handleCodewords(w http.ResponseWriter, r *http.Request) {
|
||||
q := r.URL.Query()
|
||||
sortBy := sortToken(q.Get("sort"))
|
||||
order := sortToken(q.Get("order"))
|
||||
view, ok := co.codewordViews[sortBy+"/"+order]
|
||||
if !ok {
|
||||
view, ok = co.codewordViews[sortBy+"/"+codewordDefaultOrder(sortBy)]
|
||||
}
|
||||
if !ok {
|
||||
view = co.codewordViews["count/desc"]
|
||||
}
|
||||
render(w, r, &codewordsPage{
|
||||
Rows: view.Rows,
|
||||
RowsHTML: co.preHTML("codewords:" + view.Key),
|
||||
Total: len(co.codewords),
|
||||
Shown: len(view.Rows),
|
||||
Heads: view.Heads,
|
||||
})
|
||||
}
|
||||
|
||||
type codewordView struct {
|
||||
Key string
|
||||
Rows []codewordRow
|
||||
Heads []sortHead
|
||||
}
|
||||
|
||||
func codewordDefaultOrder(sortBy string) string {
|
||||
for _, col := range codewordCols {
|
||||
if col.Key == sortBy {
|
||||
return orderParam(col.DescFirst)
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (co *Corpus) buildCodewordViews() {
|
||||
co.codewordViews = make(map[string]codewordView, 2*len(codewordCols))
|
||||
for _, sc := range codewordCols {
|
||||
for _, order := range []string{"asc", "desc"} {
|
||||
rows := append([]codewordRow(nil), co.codewordTable...)
|
||||
sortCodewordRows(rows, sc.Key, order == "desc")
|
||||
heads := sortHeads("?", codewordCols, sc.Key, order == "desc")
|
||||
co.codewordViews[sc.Key+"/"+order] = codewordView{sc.Key + "/" + order, rows, heads}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func sortCodewordRows(rows []codewordRow, sortBy string, desc bool) {
|
||||
sort.SliceStable(rows, func(i, j int) bool {
|
||||
var less, eq bool
|
||||
switch sortBy {
|
||||
case "token":
|
||||
less, eq = rows[i].Token < rows[j].Token, rows[i].Token == rows[j].Token
|
||||
case "docs":
|
||||
less, eq = rows[i].Docs < rows[j].Docs, rows[i].Docs == rows[j].Docs
|
||||
case "first", "last":
|
||||
a, b := rows[i].First, rows[j].First
|
||||
if sortBy == "last" {
|
||||
a, b = rows[i].Last, rows[j].Last
|
||||
}
|
||||
if (a == "") != (b == "") {
|
||||
return b == ""
|
||||
}
|
||||
less, eq = a < b, a == b
|
||||
case "gloss":
|
||||
a, b := rows[i].Gloss != nil, rows[j].Gloss != nil
|
||||
less, eq = !a && b, a == b // missing before defined, ascending
|
||||
default:
|
||||
less, eq = rows[i].Count < rows[j].Count, rows[i].Count == rows[j].Count
|
||||
}
|
||||
if eq {
|
||||
return rows[i].Token < rows[j].Token
|
||||
}
|
||||
if desc {
|
||||
return !less
|
||||
}
|
||||
return less
|
||||
})
|
||||
}
|
||||
162
corpus.go
Normal file
162
corpus.go
Normal file
|
|
@ -0,0 +1,162 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"html/template"
|
||||
"net/http"
|
||||
"sync/atomic"
|
||||
)
|
||||
|
||||
// Corpus is everything a load derives from the data and project directories: the
|
||||
// documents, every index and table built from them, and the page bodies
|
||||
// rendered ahead of time. One is built per load and never changed after it
|
||||
// goes live, so handlers read it without locks.
|
||||
type Corpus struct {
|
||||
// api.go
|
||||
facetsJSON []byte
|
||||
facetsETag string
|
||||
|
||||
// articles.go
|
||||
articles []Article
|
||||
articleIndex map[string]*Article // key -> article
|
||||
articleByLink map[string]*Article // link -> article
|
||||
articleGroups []articleYearGroup
|
||||
articlesTitled int
|
||||
articlesBooks int // of articlesTitled
|
||||
|
||||
// assemblies.go
|
||||
assemblies []*Assembly
|
||||
assemblyIndex map[string]*Assembly
|
||||
assembliesByDoc map[string][]*Assembly
|
||||
|
||||
// browse.go
|
||||
// homeStats holds the corpus-derived widgets on the home page - the year
|
||||
// histograms and top-N facet lists.
|
||||
//
|
||||
// By default every figure on the home page describes the REPORTED
|
||||
// collection; homeStatsSIDtoday (/?sidtoday=1) adds the SIDtoday archive. FOIA
|
||||
// material can postdate the June 2013 disclosures (a 2014 "year written"
|
||||
// bar on a Snowden archive reads as impossible at a glance), and the
|
||||
// SIDtoday archive is +2,100 issues of an internal newsletter that swamps
|
||||
// the lists and the leak-era shape the charts exist to show.
|
||||
// Every bar and list entry links to its /browse/{category}/{value} page
|
||||
// scoped to that collection, so the number shown is the number of
|
||||
// documents the page lists.
|
||||
homeStats, homeStatsSIDtoday homeFigures
|
||||
|
||||
randomPools map[string][]int
|
||||
collectionDocs map[string][]*Document
|
||||
sidtodayDocs []*Document
|
||||
|
||||
// everyDoc is every document as pointers, built once.
|
||||
everyDoc []*Document
|
||||
|
||||
archiveTimeline []archiveYear
|
||||
categoryRowHTML map[string][]template.HTML // table key -> each entry's row, as Entries has them
|
||||
azRowHTML map[string]template.HTML
|
||||
categoryRowSize map[string]int
|
||||
docRowHTML map[string]template.HTML
|
||||
docRowMarkedHTML map[string]template.HTML // rows with the "reported" badge, for ReportedIssue documents
|
||||
|
||||
// categoryTables holds every category's table under its slug.
|
||||
categoryTables map[string]*categoryTable
|
||||
|
||||
// seriesBySize is the series index of the SIDtoday tab.
|
||||
seriesBySize map[bool][]categoryEntry
|
||||
|
||||
// categoryValueCounts is how many values each category's list holds.
|
||||
categoryValueCounts map[bool]map[string]int
|
||||
|
||||
// categoryDocs is the posting list behind every /browse/{category}/{value}
|
||||
// page: slug -> folded value -> the documents holding it, in corpus order.
|
||||
categoryDocs map[string]map[string][]int32
|
||||
|
||||
// categoryNames maps each category's index keys to the spelling the lists
|
||||
// and counts show: the first one in corpus order.
|
||||
categoryNames map[string]map[string]string
|
||||
|
||||
// categoryTypeCounts splits each category value's count by document type
|
||||
// (REPORTED / SIDTODAY / FOIA): the list pages' breakdown columns. They
|
||||
// count type, not collection.
|
||||
categoryTypeCounts map[string]map[string]*collCounts
|
||||
|
||||
// changelog.go
|
||||
changelog []ChangeEntry
|
||||
|
||||
// codewords.go
|
||||
codewords []codewordEntry
|
||||
codewordTable []codewordRow
|
||||
codewordViews map[string]codewordView
|
||||
|
||||
// data.go
|
||||
archive Archive
|
||||
serialPrev map[string]string
|
||||
serialNext map[string]string
|
||||
seriesIndex map[string]*seriesGroup
|
||||
|
||||
// explore.go
|
||||
exploreTables map[string][]exploreTable
|
||||
classPatterns struct {
|
||||
Strips []clsYearRow
|
||||
Sparse []clsYearRow
|
||||
Cols []facetEntry // crossCols, each linked to its page
|
||||
HeatRows []heatRow
|
||||
}
|
||||
|
||||
// explore_ids.go
|
||||
idRows []idRow
|
||||
idChecksums string
|
||||
|
||||
// glossary.go
|
||||
glossary []GlossaryEntry
|
||||
glossaryIndex map[string]*GlossaryEntry
|
||||
glossaryGroups []glossaryGroup
|
||||
|
||||
// main.go
|
||||
prerendered map[string]template.HTML
|
||||
|
||||
// outlets.go
|
||||
outletTables map[string]outletTable
|
||||
|
||||
// portions.go
|
||||
portionStats portionSummary
|
||||
|
||||
// search.go
|
||||
tfidf TFIDFIndex
|
||||
|
||||
// docIndex is populated by loadData() in data.go after archive.Docs is fully
|
||||
// built. Provides O(1) doc lookup by ID.
|
||||
docIndex map[string]*Document
|
||||
|
||||
// seo.go
|
||||
sitemapPaths []string
|
||||
canonicalSitemap []byte
|
||||
|
||||
// sitemap.go
|
||||
sitemapParts [][]byte
|
||||
}
|
||||
|
||||
// live is the corpus requests are served from. It is replaced whole, never
|
||||
// modified: a load builds a new Corpus and stores it here when it is done.
|
||||
var live atomic.Pointer[Corpus]
|
||||
|
||||
func init() { live.Store(&Corpus{}) }
|
||||
|
||||
// corpus returns the live corpus. Handlers do not call it. bind hands each
|
||||
// request the corpus that was live when it arrived, and everything the
|
||||
// request touches is reached through that one value.
|
||||
func corpus() *Corpus { return live.Load() }
|
||||
|
||||
// loadData reads the corpus under dataDir into a new Corpus, pre-renders its
|
||||
// big page bodies (from tmplSet, so parse the templates first).
|
||||
func loadData() {
|
||||
co := &Corpus{}
|
||||
co.load()
|
||||
co.prerenderPages()
|
||||
live.Store(co)
|
||||
}
|
||||
|
||||
// bind adapts a handler to the mux. the request is served from the corpus
|
||||
// that is live when it arrives, and from that one throughout.
|
||||
func bind(h func(*Corpus, http.ResponseWriter, *http.Request)) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) { h(live.Load(), w, r) }
|
||||
}
|
||||
377
explore.go
Normal file
377
explore.go
Normal file
|
|
@ -0,0 +1,377 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"log"
|
||||
"net/http"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// The Explore section (/explore).
|
||||
|
||||
type exploreCol struct {
|
||||
Header string
|
||||
Key string
|
||||
Ratio bool
|
||||
}
|
||||
|
||||
type exploreMetric struct {
|
||||
Slug string
|
||||
Title string
|
||||
Blurb string
|
||||
Cols []exploreCol
|
||||
Vals func(d *Document) ([]float64, bool)
|
||||
}
|
||||
|
||||
const exploreRows = 200 // rows shown per page ("top 200 of N")
|
||||
|
||||
var exploreMetrics = []exploreMetric{
|
||||
{
|
||||
Slug: "programs",
|
||||
Title: "Most programs mentioned",
|
||||
Blurb: "Documents referencing the most surveillance programs.",
|
||||
Cols: []exploreCol{
|
||||
{Header: "Programs", Key: "programs"},
|
||||
{Header: "Pages", Key: "pages"},
|
||||
{Header: "Programs / page", Key: "ratio", Ratio: true},
|
||||
},
|
||||
Vals: func(d *Document) ([]float64, bool) {
|
||||
n := len(d.DocMeta.Programs)
|
||||
if n == 0 {
|
||||
return nil, false
|
||||
}
|
||||
pages, ratio := float64(d.Pages), -1.0
|
||||
if d.Pages > 0 {
|
||||
ratio = float64(n) / float64(d.Pages)
|
||||
}
|
||||
return []float64{float64(n), pages, ratio}, true
|
||||
},
|
||||
},
|
||||
{
|
||||
Slug: "people",
|
||||
Title: "Most people mentioned",
|
||||
Blurb: "Documents whose metadata names the most individuals.",
|
||||
Cols: []exploreCol{{Header: "People", Key: "people"}},
|
||||
Vals: func(d *Document) ([]float64, bool) {
|
||||
n := len(d.DocMeta.People)
|
||||
return []float64{float64(n)}, n > 0
|
||||
},
|
||||
},
|
||||
{
|
||||
Slug: "countries",
|
||||
Title: "Most countries mentioned",
|
||||
Blurb: "Documents spanning the most countries.",
|
||||
Cols: []exploreCol{{Header: "Countries", Key: "countries"}},
|
||||
Vals: func(d *Document) ([]float64, bool) {
|
||||
n := len(d.DocMeta.Countries)
|
||||
return []float64{float64(n)}, n > 0
|
||||
},
|
||||
},
|
||||
{
|
||||
Slug: "pages",
|
||||
Title: "Longest documents",
|
||||
Blurb: "Documents with the most pages.",
|
||||
Cols: []exploreCol{{Header: "Pages", Key: "pages"}},
|
||||
Vals: func(d *Document) ([]float64, bool) {
|
||||
return []float64{float64(d.Pages)}, d.Pages > 0
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
func exploreMetricBySlug(slug string) *exploreMetric {
|
||||
for i := range exploreMetrics {
|
||||
if exploreMetrics[i].Slug == slug {
|
||||
return &exploreMetrics[i]
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (co *Corpus) handleExploreIndex(w http.ResponseWriter, r *http.Request) {
|
||||
render(w, r, &explorePage{
|
||||
Metrics: exploreMetrics,
|
||||
HasAssemblies: len(co.assemblies) > 0,
|
||||
})
|
||||
}
|
||||
|
||||
type exploreRow struct {
|
||||
Rank int
|
||||
Doc Document
|
||||
Vals []string
|
||||
}
|
||||
|
||||
type exploreTable struct {
|
||||
Rows []exploreRow
|
||||
Total int
|
||||
}
|
||||
|
||||
func exploreSet(slug string, foia bool) string {
|
||||
if foia {
|
||||
return slug
|
||||
}
|
||||
return slug + ":nofoia"
|
||||
}
|
||||
|
||||
func (co *Corpus) buildExploreTables() {
|
||||
co.exploreTables = make(map[string][]exploreTable, 2*len(exploreMetrics))
|
||||
for mi := range exploreMetrics {
|
||||
m := &exploreMetrics[mi]
|
||||
if _, dup := co.exploreTables[m.Slug]; dup {
|
||||
log.Fatalf("explore: duplicate metric slug %q", m.Slug)
|
||||
}
|
||||
for _, foia := range []bool{true, false} {
|
||||
co.exploreTables[exploreSet(m.Slug, foia)] = co.buildExploreTable(m, foia)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (co *Corpus) buildExploreTable(m *exploreMetric, foia bool) []exploreTable {
|
||||
type scored struct {
|
||||
doc *Document
|
||||
vals []float64
|
||||
}
|
||||
var rows []scored
|
||||
for i := range co.archive.Docs {
|
||||
d := &co.archive.Docs[i]
|
||||
if !foia && d.Collection == collectionFOIA {
|
||||
continue
|
||||
}
|
||||
if vals, ok := m.Vals(d); ok {
|
||||
rows = append(rows, scored{d, vals})
|
||||
}
|
||||
}
|
||||
|
||||
tables := make([]exploreTable, len(m.Cols))
|
||||
for sortIdx := range m.Cols {
|
||||
sorted := append([]scored(nil), rows...)
|
||||
sort.SliceStable(sorted, func(i, j int) bool {
|
||||
if a, b := sorted[i].vals[sortIdx], sorted[j].vals[sortIdx]; a != b {
|
||||
return a > b
|
||||
}
|
||||
if a, b := sorted[i].vals[0], sorted[j].vals[0]; a != b {
|
||||
return a > b
|
||||
}
|
||||
return sorted[i].doc.Title < sorted[j].doc.Title
|
||||
})
|
||||
if len(sorted) > exploreRows {
|
||||
sorted = sorted[:exploreRows]
|
||||
}
|
||||
out := make([]exploreRow, len(sorted))
|
||||
for i, rw := range sorted {
|
||||
vals := make([]string, len(rw.vals))
|
||||
for k, v := range rw.vals {
|
||||
switch {
|
||||
case v < 0:
|
||||
vals[k] = "-"
|
||||
case m.Cols[k].Ratio:
|
||||
vals[k] = trimRatio(v)
|
||||
default:
|
||||
vals[k] = strconv.Itoa(int(v))
|
||||
}
|
||||
}
|
||||
out[i] = exploreRow{Rank: i + 1, Doc: *rw.doc, Vals: vals}
|
||||
}
|
||||
tables[sortIdx] = exploreTable{Rows: out, Total: len(rows)}
|
||||
}
|
||||
return tables
|
||||
}
|
||||
|
||||
func exploreKey(set string, sortIdx int) string {
|
||||
return "explore:" + set + ":" + strconv.Itoa(sortIdx)
|
||||
}
|
||||
|
||||
func (co *Corpus) handleExploreMetric(w http.ResponseWriter, r *http.Request) {
|
||||
m := exploreMetricBySlug(r.PathValue("metric"))
|
||||
if m == nil {
|
||||
notFound(w, r)
|
||||
return
|
||||
}
|
||||
|
||||
sortIdx := 0
|
||||
sortBy := r.URL.Query().Get("sort")
|
||||
for i, c := range m.Cols {
|
||||
if c.Key == sortBy {
|
||||
sortIdx = i
|
||||
break
|
||||
}
|
||||
}
|
||||
set := exploreSet(m.Slug, readPrefs(r).IncludeFOIA)
|
||||
t := co.exploreTables[set][sortIdx]
|
||||
|
||||
cols := []sortCol{{Label: "#", Class: "num-col"}, {Label: "Document"}, {Label: "Date"}}
|
||||
for _, c := range m.Cols {
|
||||
cols = append(cols, sortCol{Key: c.Key, Label: c.Header, Class: "num-col", DescFirst: true, Fixed: true})
|
||||
}
|
||||
|
||||
render(w, r, &exploreTablePage{
|
||||
Metric: m,
|
||||
Rows: t.Rows,
|
||||
RowsHTML: co.preHTML(exploreKey(set, sortIdx)),
|
||||
Total: t.Total,
|
||||
Shown: len(t.Rows),
|
||||
Heads: sortHeads("?", cols, m.Cols[sortIdx].Key, true),
|
||||
})
|
||||
}
|
||||
|
||||
func trimRatio(v float64) string {
|
||||
s := strconv.FormatFloat(v, 'f', 2, 64)
|
||||
s = strings.TrimRight(s, "0")
|
||||
return strings.TrimRight(s, ".")
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// /explore/classification
|
||||
|
||||
type clsSeg struct {
|
||||
Value string
|
||||
Count int
|
||||
Pct int
|
||||
URL string
|
||||
Class string // seg-0..seg-3, one per stripLevels entry
|
||||
}
|
||||
|
||||
type clsYearRow struct {
|
||||
Year string
|
||||
URL string // the year's page, over every collection as the counts are
|
||||
Total int
|
||||
Segs []clsSeg
|
||||
}
|
||||
|
||||
type heatCell struct {
|
||||
Count int
|
||||
URL string
|
||||
}
|
||||
|
||||
type heatRow struct {
|
||||
Name string
|
||||
URL string
|
||||
Cells []heatCell
|
||||
}
|
||||
|
||||
var crossCols = []string{"TOP SECRET", "SECRET", "CONFIDENTIAL", "UNCLASSIFIED", "SI", "COMINT", "STRAP 1", "STRAP 2", "TK"}
|
||||
var stripLevels = []string{"TOP SECRET", "SECRET", "CONFIDENTIAL", "UNCLASSIFIED"}
|
||||
|
||||
const stripMinMarks = 5
|
||||
|
||||
func (co *Corpus) buildClassPatterns() {
|
||||
top := crossCols
|
||||
rank := make(map[string]int, len(top))
|
||||
for i, v := range top {
|
||||
rank[valueKey(v)] = i
|
||||
}
|
||||
levelRank := make(map[string]int, len(stripLevels))
|
||||
for i, v := range stripLevels {
|
||||
levelRank[valueKey(v)] = i
|
||||
}
|
||||
|
||||
yearCounts := map[string][]int{}
|
||||
agCounts := map[string][]int{}
|
||||
agName := map[string]string{}
|
||||
for i := range co.archive.Docs {
|
||||
d := &co.archive.Docs[i]
|
||||
year := docYear(d)
|
||||
agency := d.DocMeta.Creator
|
||||
if n, ok := agName[valueKey(agency)]; ok {
|
||||
agency = n
|
||||
} else {
|
||||
agName[valueKey(agency)] = agency
|
||||
}
|
||||
for _, cv := range d.DocMeta.Classification {
|
||||
if lvl, isLevel := levelRank[valueKey(cv)]; isLevel && year != "" {
|
||||
yc := yearCounts[year]
|
||||
if yc == nil {
|
||||
yc = make([]int, len(stripLevels))
|
||||
yearCounts[year] = yc
|
||||
}
|
||||
yc[lvl]++
|
||||
}
|
||||
idx, known := rank[valueKey(cv)]
|
||||
if known && agency != "" {
|
||||
ac := agCounts[agency]
|
||||
if ac == nil {
|
||||
ac = make([]int, len(top))
|
||||
agCounts[agency] = ac
|
||||
}
|
||||
ac[idx]++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
years := make([]string, 0, len(yearCounts))
|
||||
for y := range yearCounts {
|
||||
years = append(years, y)
|
||||
}
|
||||
sort.Strings(years)
|
||||
strips := make([]clsYearRow, 0, len(years))
|
||||
var sparse []clsYearRow
|
||||
for _, y := range years {
|
||||
yc := yearCounts[y]
|
||||
total := 0
|
||||
for _, n := range yc {
|
||||
total += n
|
||||
}
|
||||
row := clsYearRow{Year: y, URL: wholeArchive.link(categoryValueURL("docdate", y)), Total: total}
|
||||
if total < stripMinMarks {
|
||||
sparse = append(sparse, row)
|
||||
continue
|
||||
}
|
||||
for i, n := range yc {
|
||||
if n == 0 {
|
||||
continue
|
||||
}
|
||||
row.Segs = append(row.Segs, clsSeg{
|
||||
Value: stripLevels[i],
|
||||
Count: n,
|
||||
Pct: barPct(n, total, 1),
|
||||
URL: wholeArchive.link(dateRangeSearchURL("CL", exactClause(stripLevels[i]), [2]string{"date_doc", y})),
|
||||
Class: "seg-" + strconv.Itoa(i),
|
||||
})
|
||||
}
|
||||
strips = append(strips, row)
|
||||
}
|
||||
|
||||
mass := make(map[string]int, len(agCounts))
|
||||
for a, cs := range agCounts {
|
||||
for _, n := range cs {
|
||||
mass[a] += n
|
||||
}
|
||||
}
|
||||
ags := rankCounts(mass, -1)
|
||||
rows := make([]heatRow, 0, len(ags))
|
||||
for _, a := range ags {
|
||||
hr := heatRow{Name: a, URL: wholeArchive.link(categoryValueURL("agency", a))}
|
||||
for i, n := range agCounts[a] {
|
||||
cell := heatCell{Count: n}
|
||||
if n > 0 {
|
||||
cell.URL = wholeArchive.link(searchURL([]searchClause{
|
||||
{Field: "CR", Value: exactClause(a)},
|
||||
{Field: "CL", Value: exactClause(top[i])},
|
||||
}))
|
||||
}
|
||||
hr.Cells = append(hr.Cells, cell)
|
||||
}
|
||||
rows = append(rows, hr)
|
||||
}
|
||||
|
||||
cols := make([]facetEntry, len(top))
|
||||
for i, v := range top {
|
||||
cols[i] = facetEntry{Name: v, URL: wholeArchive.link(categoryValueURL("classification", v))}
|
||||
}
|
||||
|
||||
co.classPatterns.Strips = strips
|
||||
co.classPatterns.Sparse = sparse
|
||||
co.classPatterns.Cols = cols
|
||||
co.classPatterns.HeatRows = rows
|
||||
}
|
||||
|
||||
func (co *Corpus) handleClassPatterns(w http.ResponseWriter, r *http.Request) {
|
||||
render(w, r, &classPatternsPage{
|
||||
Cols: co.classPatterns.Cols,
|
||||
Levels: stripLevels,
|
||||
Strips: co.classPatterns.Strips,
|
||||
Sparse: co.classPatterns.Sparse,
|
||||
MinMarks: stripMinMarks,
|
||||
HeatRows: co.classPatterns.HeatRows,
|
||||
})
|
||||
}
|
||||
114
explore_ids.go
Normal file
114
explore_ids.go
Normal file
|
|
@ -0,0 +1,114 @@
|
|||
// explore_ids.go - /explore/ids: every document's identifiers in one table.
|
||||
//
|
||||
// /explore/ids.txt is the same checksums in sha256sum(1) format, so a mirror
|
||||
// or a download verifies with "sha256sum -c" in the corpus root.
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type idVersionRow struct {
|
||||
Slug string
|
||||
PDFFile string
|
||||
PDFURL string
|
||||
Size string
|
||||
Pages int
|
||||
SHA string
|
||||
}
|
||||
|
||||
type idRow struct {
|
||||
ID string
|
||||
Collection string
|
||||
Pages int
|
||||
PDFFile string
|
||||
PDFURL string
|
||||
Size string
|
||||
SHA string
|
||||
Aka []AkaEntry
|
||||
Versions []idVersionRow
|
||||
}
|
||||
|
||||
func humanSize(n int64) string {
|
||||
switch {
|
||||
case n <= 0:
|
||||
return ""
|
||||
case n < 1<<10:
|
||||
return fmt.Sprintf("%d B", n)
|
||||
case n < 1<<20:
|
||||
return fmt.Sprintf("%.0f KB", float64(n)/(1<<10))
|
||||
case n < 1<<30:
|
||||
return fmt.Sprintf("%.1f MB", float64(n)/(1<<20))
|
||||
}
|
||||
return fmt.Sprintf("%.2f GB", float64(n)/(1<<30))
|
||||
}
|
||||
|
||||
func (co *Corpus) buildIDRows() {
|
||||
co.idRows = make([]idRow, 0, len(co.archive.Docs))
|
||||
var sums strings.Builder
|
||||
|
||||
for i := range co.archive.Docs {
|
||||
d := &co.archive.Docs[i]
|
||||
row := idRow{
|
||||
ID: d.ID,
|
||||
Collection: d.Collection,
|
||||
Pages: d.Pages,
|
||||
PDFFile: d.PDFFile,
|
||||
PDFURL: filePath(d.ID, d.PDFFile),
|
||||
Size: humanSize(d.PDFSize),
|
||||
SHA: strings.TrimSpace(d.PDFSha256),
|
||||
Aka: d.AkaID,
|
||||
}
|
||||
for _, v := range d.Versions {
|
||||
row.Versions = append(row.Versions, idVersionRow{
|
||||
Slug: v.Slug,
|
||||
PDFFile: v.PDFFile,
|
||||
PDFURL: versionFilePath(d.ID, v.Slug, v.PDFFile),
|
||||
Size: humanSize(v.PDFSize),
|
||||
Pages: len(v.Images),
|
||||
SHA: v.SHA,
|
||||
})
|
||||
}
|
||||
co.idRows = append(co.idRows, row)
|
||||
|
||||
rel, err := filepath.Rel(dataDir, d.Dir)
|
||||
if err != nil {
|
||||
rel = d.ID
|
||||
}
|
||||
if row.SHA != "" {
|
||||
sums.WriteString(row.SHA + " " + filepath.ToSlash(filepath.Join(rel, d.PDFFile)) + "\n")
|
||||
}
|
||||
for _, v := range d.Versions {
|
||||
if v.SHA != "" {
|
||||
sums.WriteString(v.SHA + " " + filepath.ToSlash(filepath.Join(rel, "versions", v.Slug, v.PDFFile)) + "\n")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
sort.Slice(co.idRows, func(i, j int) bool {
|
||||
a, b := strings.ToLower(co.idRows[i].ID), strings.ToLower(co.idRows[j].ID)
|
||||
if a != b {
|
||||
return a < b
|
||||
}
|
||||
return co.idRows[i].ID < co.idRows[j].ID
|
||||
})
|
||||
co.idChecksums = sums.String()
|
||||
}
|
||||
|
||||
func (co *Corpus) handleExploreIDs(w http.ResponseWriter, r *http.Request) {
|
||||
render(w, r, &idsPage{
|
||||
Rows: co.idRows,
|
||||
RowsHTML: co.preHTML("ids"),
|
||||
Shown: len(co.idRows),
|
||||
})
|
||||
}
|
||||
|
||||
func (co *Corpus) handleExploreIDsText(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
|
||||
w.Header().Set("Cache-Control", "public, max-age=3600")
|
||||
w.Write([]byte(co.idChecksums))
|
||||
}
|
||||
135
glossary.go
Normal file
135
glossary.go
Normal file
|
|
@ -0,0 +1,135 @@
|
|||
// glossary.go - codename/term glossary loaded from data/glossary.json.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"html/template"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type GlossaryEntry struct {
|
||||
Term string `json:"term"`
|
||||
Aka []string `json:"aka,omitempty"`
|
||||
Definition string `json:"definition"`
|
||||
See []string `json:"see,omitempty"`
|
||||
SeeLinks []seeLink `json:"-"`
|
||||
|
||||
searchLink string
|
||||
}
|
||||
|
||||
type seeLink struct{ Name, Term string }
|
||||
|
||||
func (e GlossaryEntry) DefinitionHTML() template.HTML {
|
||||
return template.HTML(e.Definition)
|
||||
}
|
||||
|
||||
func (e GlossaryEntry) SearchURL() string {
|
||||
if e.searchLink != "" {
|
||||
return e.searchLink
|
||||
}
|
||||
return e.buildSearchURL()
|
||||
}
|
||||
|
||||
func (e GlossaryEntry) buildSearchURL() string {
|
||||
clauses := []searchClause{{Conj: "and", Field: "TE", Value: searchClauseValue(e.Term)}}
|
||||
for _, a := range e.Aka {
|
||||
if strings.TrimSpace(a) != "" {
|
||||
clauses = append(clauses, searchClause{Conj: "or", Field: "TE", Value: searchClauseValue(a)})
|
||||
}
|
||||
}
|
||||
return searchURL(clauses)
|
||||
}
|
||||
|
||||
func searchClauseValue(v string) string {
|
||||
v = strings.TrimSpace(v)
|
||||
if len(splitWords(v)) == 1 && len(splitWords(v)[0]) == len(v) {
|
||||
return v
|
||||
}
|
||||
return `"` + v + `"`
|
||||
}
|
||||
|
||||
func (co *Corpus) glossaryLookup(term string) *GlossaryEntry {
|
||||
return co.glossaryIndex[strings.ToLower(term)]
|
||||
}
|
||||
|
||||
func (co *Corpus) loadGlossary() {
|
||||
p := projectPath("glossary.json")
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
log.Printf("Note: %s not found - glossary page will be empty.", p)
|
||||
return
|
||||
}
|
||||
if err := json.Unmarshal(b, &co.glossary); err != nil {
|
||||
warnf("%s is malformed: %v - glossary disabled.", p, err)
|
||||
co.glossary = nil
|
||||
return
|
||||
}
|
||||
co.glossaryIndex = map[string]*GlossaryEntry{}
|
||||
sort.SliceStable(co.glossary, func(i, j int) bool {
|
||||
return normFoldString(co.glossary[i].Term) < normFoldString(co.glossary[j].Term)
|
||||
})
|
||||
for i := range co.glossary {
|
||||
e := &co.glossary[i]
|
||||
e.searchLink = e.buildSearchURL()
|
||||
addIndex := func(key string) {
|
||||
low := strings.ToLower(key)
|
||||
if prev := co.glossaryIndex[low]; prev != nil && prev != e {
|
||||
warnf("glossary term/alias %q appears under both %q and %q - the later entry wins", key, prev.Term, e.Term)
|
||||
}
|
||||
co.glossaryIndex[low] = e
|
||||
}
|
||||
addIndex(e.Term)
|
||||
for _, a := range e.Aka {
|
||||
addIndex(a)
|
||||
}
|
||||
}
|
||||
var unknown []string
|
||||
for i := range co.glossary {
|
||||
e := &co.glossary[i]
|
||||
e.SeeLinks = nil
|
||||
for _, name := range e.See {
|
||||
l := seeLink{Name: name}
|
||||
if t := co.glossaryLookup(name); t != nil {
|
||||
l.Term = t.Term
|
||||
} else {
|
||||
unknown = append(unknown, name)
|
||||
}
|
||||
e.SeeLinks = append(e.SeeLinks, l)
|
||||
}
|
||||
}
|
||||
if len(unknown) > 0 {
|
||||
warnf("glossary: %d see-also names match no term or alias, shown unlinked: %s",
|
||||
len(unknown), strings.Join(firstN(unknown, 8), ", "))
|
||||
}
|
||||
co.buildGlossaryGroups()
|
||||
log.Printf("Glossary loaded: %d terms", len(co.glossary))
|
||||
}
|
||||
|
||||
type glossaryGroup struct {
|
||||
Letter string
|
||||
Entries []GlossaryEntry
|
||||
}
|
||||
|
||||
func (co *Corpus) buildGlossaryGroups() {
|
||||
co.glossaryGroups = nil
|
||||
for _, e := range co.glossary {
|
||||
letter := firstLetter(e.Term)
|
||||
if len(co.glossaryGroups) == 0 || co.glossaryGroups[len(co.glossaryGroups)-1].Letter != letter {
|
||||
co.glossaryGroups = append(co.glossaryGroups, glossaryGroup{Letter: letter})
|
||||
}
|
||||
co.glossaryGroups[len(co.glossaryGroups)-1].Entries = append(co.glossaryGroups[len(co.glossaryGroups)-1].Entries, e)
|
||||
}
|
||||
}
|
||||
|
||||
func (co *Corpus) handleGlossary(w http.ResponseWriter, r *http.Request) {
|
||||
render(w, r, &glossaryPage{
|
||||
Groups: co.glossaryGroups,
|
||||
GroupsHTML: co.preHTML("glossary"),
|
||||
Total: len(co.glossary),
|
||||
})
|
||||
}
|
||||
3
go.mod
Normal file
3
go.mod
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
module snowchive
|
||||
|
||||
go 1.22
|
||||
194
history.go
Normal file
194
history.go
Normal file
|
|
@ -0,0 +1,194 @@
|
|||
// history.go - opt-in, cookie-local browsing history: recently viewed
|
||||
// documents and recent searches. Both are OFF by default (see Prefs).
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
)
|
||||
|
||||
const (
|
||||
recentDocsCookie = "recent_docs"
|
||||
recentSearchesCookie = "recent_searches"
|
||||
recentDocsMax = 20
|
||||
recentSearchesMax = 15
|
||||
)
|
||||
|
||||
func readIDListCookie(r *http.Request, name string, max int) []string {
|
||||
c, err := r.Cookie(name)
|
||||
if err != nil || c.Value == "" {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
for _, id := range strings.Split(c.Value, "|") {
|
||||
if id != "" {
|
||||
out = append(out, id)
|
||||
if len(out) == max {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func pushFront(list []string, id string, max int) []string {
|
||||
out := make([]string, 0, max)
|
||||
out = append(out, id)
|
||||
for _, x := range list {
|
||||
if len(out) == max {
|
||||
break
|
||||
}
|
||||
if x != id {
|
||||
out = append(out, x)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func pushRecentDoc(w http.ResponseWriter, r *http.Request, id string) {
|
||||
ids := pushFront(readIDListCookie(r, recentDocsCookie, recentDocsMax), id, recentDocsMax)
|
||||
writeListCookie(w, recentDocsCookie, ids)
|
||||
}
|
||||
|
||||
type recentSearch struct {
|
||||
Label string
|
||||
URL string
|
||||
QS string
|
||||
}
|
||||
|
||||
func recordRecentSearch(w http.ResponseWriter, r *http.Request, qs string) []string {
|
||||
if len(qs) > searchQSMax {
|
||||
return nil
|
||||
}
|
||||
entry := url.QueryEscape(qs)
|
||||
|
||||
entries := readIDListCookie(r, recentSearchesCookie, recentSearchesMax)
|
||||
entries = pushFront(entries, entry, recentSearchesMax)
|
||||
return writeListCookie(w, recentSearchesCookie, entries)
|
||||
}
|
||||
|
||||
func (co *Corpus) readRecentSearches(r *http.Request) []recentSearch {
|
||||
return co.parseSearchEntries(readIDListCookie(r, recentSearchesCookie, recentSearchesMax))
|
||||
}
|
||||
|
||||
func (co *Corpus) parseSearchEntries(entries []string) []recentSearch {
|
||||
var out []recentSearch
|
||||
for _, e := range entries {
|
||||
qs, err := url.QueryUnescape(e)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
v, err := url.ParseQuery(qs)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if label, ok := co.searchLabel(v); ok {
|
||||
out = append(out, recentSearch{Label: label, URL: "/search?" + qs, QS: qs})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
const searchQSMax = 500
|
||||
|
||||
func (co *Corpus) savableSearch(v url.Values) bool {
|
||||
if len(v.Encode()) > searchQSMax {
|
||||
return false
|
||||
}
|
||||
_, ok := co.searchLabel(v)
|
||||
return ok
|
||||
}
|
||||
|
||||
func (co *Corpus) searchLabel(v url.Values) (label string, ok bool) {
|
||||
var parts []string
|
||||
fields := v["fqf"]
|
||||
conjs := v["fqc"]
|
||||
for i, term := range v["fqv"] {
|
||||
if term == "" {
|
||||
continue
|
||||
}
|
||||
label := term
|
||||
if i < len(fields) && fields[i] != "" && fields[i] != "TE" {
|
||||
code := fields[i]
|
||||
if f, ok := fieldByCode[code]; ok {
|
||||
label = term + " [" + f.Label + "]"
|
||||
} else {
|
||||
label = term + " [" + code + "]"
|
||||
}
|
||||
}
|
||||
if i < len(conjs) && conjs[i] == "not" {
|
||||
label = "-" + label
|
||||
}
|
||||
parts = append(parts, label)
|
||||
}
|
||||
var notes, filters []string
|
||||
if colls := v["coll"]; len(colls) > 0 && len(colls) < len(validCollections) {
|
||||
filters = append(filters, colls...)
|
||||
}
|
||||
if types := v["type"]; len(types) > 0 && (len(co.archive.Types) == 0 || len(types) < len(co.archive.Types)) {
|
||||
filters = append(filters, types...)
|
||||
}
|
||||
if len(filters) > 0 {
|
||||
notes = append(notes, "("+strings.Join(filters, ", ")+")")
|
||||
}
|
||||
if n := dateRangeNote("doc", v.Get("date_doc_start"), v.Get("date_doc_end")); n != "" {
|
||||
notes = append(notes, n)
|
||||
}
|
||||
if n := dateRangeNote("pub", v.Get("date_pub_start"), v.Get("date_pub_end")); n != "" {
|
||||
notes = append(notes, n)
|
||||
}
|
||||
switch {
|
||||
case len(parts) == 0 && len(notes) == 0:
|
||||
return "", false
|
||||
case len(parts) == 0:
|
||||
parts = []string{"all documents"}
|
||||
}
|
||||
return strings.Join(append(parts, notes...), " · "), true
|
||||
}
|
||||
|
||||
func dateRangeNote(kind, start, end string) string {
|
||||
switch {
|
||||
case start == "" && end == "":
|
||||
return ""
|
||||
case start == end || (start != "" && end == padDateEnd(start)):
|
||||
// date:2013 arrives as start "2013", end "2013-12-31": one year
|
||||
return "(" + kind + " " + start + ")"
|
||||
case start == "":
|
||||
return "(" + kind + " to " + end + ")"
|
||||
case end == "":
|
||||
return "(" + kind + " from " + start + ")"
|
||||
default:
|
||||
return "(" + kind + " " + start + " to " + end + ")"
|
||||
}
|
||||
}
|
||||
|
||||
func handleHistoryClear(w http.ResponseWriter, r *http.Request) {
|
||||
if !readPOSTForm(w, r) {
|
||||
return
|
||||
}
|
||||
switch r.FormValue("which") {
|
||||
case "docs":
|
||||
delCookie(w, recentDocsCookie)
|
||||
case "searches":
|
||||
delCookie(w, recentSearchesCookie)
|
||||
default:
|
||||
delCookie(w, recentDocsCookie)
|
||||
delCookie(w, recentSearchesCookie)
|
||||
}
|
||||
http.Redirect(w, r, safeBackPath(r.FormValue("back")), http.StatusSeeOther)
|
||||
}
|
||||
|
||||
// safeBackPath only allows local absolute paths as redirect targets.
|
||||
func safeBackPath(p string) string {
|
||||
if !strings.HasPrefix(p, "/") || strings.HasPrefix(p, "//") {
|
||||
return "/"
|
||||
}
|
||||
for i := 0; i < len(p); i++ {
|
||||
if p[i] == '\\' || p[i] < 0x20 {
|
||||
return "/"
|
||||
}
|
||||
}
|
||||
return p
|
||||
}
|
||||
773
main.go
Normal file
773
main.go
Normal file
|
|
@ -0,0 +1,773 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"embed"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"html/template"
|
||||
"io/fs"
|
||||
"log"
|
||||
"maps"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/signal"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
// devMode is set by the -dev flag. In dev mode templates and static files
|
||||
// are read from ./templates and ./static on every request (so edits show
|
||||
// without a rebuild), caching is disabled and error details are shown in
|
||||
// responses. Never run production with -dev: the error detail leaks
|
||||
// filesystem paths and template internals.
|
||||
var devMode bool
|
||||
|
||||
// The templates and static assets are compiled into the binary, so a
|
||||
// deployment is one executable plus the data and project directories - nothing depends
|
||||
// on the working directory the process happens to start in. -dev swaps in
|
||||
// the on-disk copies (see initAssets).
|
||||
//
|
||||
//go:embed templates static
|
||||
var embeddedAssets embed.FS
|
||||
|
||||
var (
|
||||
templateFS fs.FS // templates/*.html
|
||||
staticFS fs.FS // static/*
|
||||
)
|
||||
|
||||
// staticHashes holds a content hash per embedded asset, computed once at
|
||||
// startup. It is both the assets' ETag (embed.FS files have a zero modtime,
|
||||
// so net/http sends no Last-Modified and an unvalidated asset would be
|
||||
// re-downloaded on every view) and their version stamp: base.html links
|
||||
// them as ?v=<hash> (the asset func), and handleStatic caches exactly that
|
||||
// URL forever. A changed file gets a new hash and so a new URL, whatever
|
||||
// the build was stamped with.
|
||||
var staticHashes map[string]string
|
||||
|
||||
func buildStaticHashes() {
|
||||
staticHashes = map[string]string{}
|
||||
err := fs.WalkDir(staticFS, ".", func(p string, d fs.DirEntry, err error) error {
|
||||
if err != nil || d.IsDir() {
|
||||
return err
|
||||
}
|
||||
b, err := fs.ReadFile(staticFS, p)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
sum := sha256.Sum256(b)
|
||||
staticHashes[p] = hex.EncodeToString(sum[:8])
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatalf("embedded static: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func initAssets() {
|
||||
if devMode {
|
||||
templateFS = os.DirFS("templates")
|
||||
staticFS = os.DirFS("static")
|
||||
staticHashes = nil
|
||||
return
|
||||
}
|
||||
var err error
|
||||
if templateFS, err = fs.Sub(embeddedAssets, "templates"); err != nil {
|
||||
log.Fatalf("embedded templates: %v", err)
|
||||
}
|
||||
if staticFS, err = fs.Sub(embeddedAssets, "static"); err != nil {
|
||||
log.Fatalf("embedded static: %v", err)
|
||||
}
|
||||
buildStaticHashes()
|
||||
}
|
||||
|
||||
type sortOption struct{ Key, Label string }
|
||||
|
||||
var sortOptions = func() []sortOption {
|
||||
opts := []sortOption{{"relevance", "Relevance"}}
|
||||
for _, s := range docSorts {
|
||||
if s.InBar {
|
||||
opts = append(opts, sortOption{s.Key, s.Label})
|
||||
}
|
||||
}
|
||||
return opts
|
||||
}()
|
||||
|
||||
func sortLabel(key string) string {
|
||||
switch key {
|
||||
case "relevance":
|
||||
return "Relevance"
|
||||
case "":
|
||||
return "Title"
|
||||
}
|
||||
if s := docSortByKey[key]; s != nil {
|
||||
return s.Label
|
||||
}
|
||||
return strings.ToUpper(key[:1]) + key[1:]
|
||||
}
|
||||
|
||||
var (
|
||||
// tmplSet holds one parsed template set PER PAGE (base + partials +
|
||||
// page).
|
||||
tmplSet map[string]*template.Template
|
||||
|
||||
funcMap = template.FuncMap{
|
||||
"buildInfo": func() map[string]string {
|
||||
return map[string]string{"Commit": buildCommit}
|
||||
},
|
||||
// articleKey is the key of the article page for a publication's
|
||||
// link. Every linked publication of a loaded document has one
|
||||
// (buildArticles), so the key needs no lookup - and no corpus.
|
||||
"articleKey": func(link string) string {
|
||||
if link == "" {
|
||||
return ""
|
||||
}
|
||||
return articleKeyFor(link)
|
||||
},
|
||||
"asset": func(name string) string {
|
||||
if h, ok := staticHashes[name]; ok {
|
||||
return "/static/" + name + "?v=" + h
|
||||
}
|
||||
return "/static/" + name
|
||||
},
|
||||
// wholeArchiveQS is the scope that names every collection, for the
|
||||
// links of a FOIA release on a page rendered once for all readers.
|
||||
"wholeArchiveQS": func() string { return wholeArchive.suffix() },
|
||||
"facetLinks": facetLinks,
|
||||
"add": func(a, b int) int { return a + b },
|
||||
"eqFold": sameValue,
|
||||
"sortOptions": func() []sortOption { return sortOptions },
|
||||
"sortLabel": sortLabel,
|
||||
"agencyName": agencyName,
|
||||
"dict": func(values ...interface{}) (map[string]interface{}, error) {
|
||||
if len(values)%2 != 0 {
|
||||
return nil, fmt.Errorf("dict: odd number of arguments (%d)", len(values))
|
||||
}
|
||||
dict := make(map[string]interface{}, len(values)/2)
|
||||
for i := 0; i < len(values); i += 2 {
|
||||
key, ok := values[i].(string)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("dict: argument %d is not a string key (%T)", i, values[i])
|
||||
}
|
||||
dict[key] = values[i+1]
|
||||
}
|
||||
return dict, nil
|
||||
},
|
||||
// pathEscape percent-encodes a value for use as a single URL path
|
||||
// segment (so values like "TOP SECRET//SI" don't split the path).
|
||||
"pathEscape": func(s string) template.URL {
|
||||
return template.URL(url.PathEscape(s))
|
||||
},
|
||||
"azID": azID,
|
||||
"pages": func(current, total int) []int {
|
||||
var out []int
|
||||
last := 0
|
||||
for i := 1; i <= total; i++ {
|
||||
if i == 1 || i == total || (i >= current-2 && i <= current+2) {
|
||||
if last != 0 && i-last > 1 {
|
||||
out = append(out, 0)
|
||||
}
|
||||
out = append(out, i)
|
||||
last = i
|
||||
}
|
||||
}
|
||||
return out
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
// Build identity, stamped at build time so mirrors are comparable at a
|
||||
// glance. Shown in the footer next to the data load time:
|
||||
//
|
||||
// go build -ldflags "-X main.buildCommit=$(git rev-parse --short HEAD)"
|
||||
var buildCommit = "dev"
|
||||
|
||||
// accelPrefix is set by -accel-files: the internal nginx location that maps
|
||||
// to the data directory. With it, handleFiles still decides whether a file is
|
||||
// served, but answers with an X-Accel-Redirect and nginx sends the bytes
|
||||
// (sendfile, ranges, slow readers) without them passing through this process.
|
||||
// Only set it behind nginx: any other client gets an empty 200.
|
||||
var accelPrefix string
|
||||
|
||||
// accelExpires is set by -accel-expires: how long the proxy may keep a page
|
||||
// this server marks public, sent as X-Accel-Expires (which nginx obeys ahead
|
||||
// of Cache-Control and strips from the response). Browsers still get the
|
||||
// short max-age. The corpus is read once at startup, so such a page cannot
|
||||
// change before the next restart; clear the proxy's cache after one.
|
||||
var accelExpires string
|
||||
|
||||
// apiEnabled is cleared by -no-api: public instances that don't want bulk
|
||||
// scraping can turn the JSON API off.
|
||||
var apiEnabled = true
|
||||
|
||||
func main() {
|
||||
rebuildRelated := flag.Bool("rebuild-related", false, "Rebuild related docs index and exit")
|
||||
flag.BoolVar(&devMode, "dev", false, "Dev mode: serve templates/ and static/ from the working directory and re-parse templates per request, no immutable static cache, verbose errors")
|
||||
baseFlag := flag.String("base", os.Getenv("SNOWCHIVE_BASE"),
|
||||
"Canonical base URL (e.g. https://archive.example.org) for the absolute URLs in robots.txt and the sitemap. "+
|
||||
"Without it those URLs are built from each request's Host header, which is fine only behind a proxy that pins Host. "+
|
||||
"Set this on anything internet-facing. Defaults to $SNOWCHIVE_BASE.")
|
||||
listen := flag.String("listen", envOr("SNOWCHIVE_LISTEN", "127.0.0.1:8080"),
|
||||
"TCP address to listen on. Defaults to $SNOWCHIVE_LISTEN, then 127.0.0.1:8080.")
|
||||
root := flag.String("root", envOr("SNOWCHIVE_ROOT", "."),
|
||||
"Archive directory holding data/ (the document corpus, one directory per document) and project/ "+
|
||||
"(the files about the archive as a whole: glossary.json, assemblies.json, changelog.json and related.json; only -rebuild-related writes there). "+
|
||||
"Defaults to $SNOWCHIVE_ROOT, then the working directory.")
|
||||
check := flag.Bool("check", false,
|
||||
"Load the corpus, print the warnings and the tally, then exit - non-zero when anything was reported. ")
|
||||
noAPI := flag.Bool("no-api", envBool("SNOWCHIVE_NO_API"),
|
||||
"Disable the read-only JSON API: every /api/ path answers a JSON 403 and the UI stops linking to it. Defaults on when $SNOWCHIVE_NO_API is true (1, t, true).")
|
||||
flag.StringVar(&accelPrefix, "accel-files", os.Getenv("SNOWCHIVE_ACCEL_FILES"),
|
||||
"Internal nginx location that aliases the data directory (e.g. /_files/). Document files are then answered with an X-Accel-Redirect into it and sent by nginx. "+
|
||||
"Only behind nginx. Defaults to $SNOWCHIVE_ACCEL_FILES.")
|
||||
accelTTL := flag.Duration("accel-expires", envDuration("SNOWCHIVE_ACCEL_EXPIRES"),
|
||||
"How long the proxy may cache a page served to a reader without cookies (e.g. 10m), sent as X-Accel-Expires; browsers keep the short max-age. "+
|
||||
"Clear the proxy cache after a restart with new data. Defaults to $SNOWCHIVE_ACCEL_EXPIRES.")
|
||||
flag.Parse()
|
||||
setRoot(*root)
|
||||
if accelPrefix != "" {
|
||||
if !strings.HasPrefix(accelPrefix, "/") || !strings.HasSuffix(accelPrefix, "/") {
|
||||
log.Fatalf("-accel-files: %q must begin and end with a slash", accelPrefix)
|
||||
}
|
||||
}
|
||||
if *accelTTL >= time.Second {
|
||||
accelExpires = strconv.Itoa(int(*accelTTL / time.Second))
|
||||
}
|
||||
if *baseFlag != "" {
|
||||
b, err := parseCanonicalBase(*baseFlag)
|
||||
if err != nil {
|
||||
log.Fatalf("-base: %v", err)
|
||||
}
|
||||
canonicalBase = b
|
||||
}
|
||||
apiEnabled = !*noAPI
|
||||
initAssets()
|
||||
// Before the load: it pre-renders the big page bodies from these.
|
||||
if !devMode {
|
||||
if err := parseTemplates(); err != nil {
|
||||
log.Fatalf("Fatal: Could not parse templates at startup: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
loadData()
|
||||
co := corpus()
|
||||
|
||||
if *check {
|
||||
if loadErrors > 0 || loadWarnings > 0 {
|
||||
log.Printf("check failed: %d errors, %d warnings", loadErrors, loadWarnings)
|
||||
os.Exit(1)
|
||||
}
|
||||
log.Printf("check passed: %d documents, nothing to report", len(co.archive.Docs))
|
||||
return
|
||||
}
|
||||
|
||||
if *rebuildRelated {
|
||||
log.Println("Rebuilding related docs index...")
|
||||
|
||||
existing := make(map[string][]string)
|
||||
if b, err := os.ReadFile(relatedDocsPath()); err == nil {
|
||||
if err := json.Unmarshal(b, &existing); err != nil {
|
||||
warnf("could not parse existing related.json: %v - starting fresh", err)
|
||||
existing = make(map[string][]string)
|
||||
} else {
|
||||
log.Printf("Loaded existing related.json with %d entries", len(existing))
|
||||
}
|
||||
}
|
||||
|
||||
stale := 0
|
||||
for id := range existing {
|
||||
if co.unknownDoc(id) {
|
||||
delete(existing, id)
|
||||
stale++
|
||||
}
|
||||
}
|
||||
var missing []Document
|
||||
for _, d := range co.archive.Docs {
|
||||
if ids, ok := existing[d.ID]; !ok || slices.ContainsFunc(ids, co.unknownDoc) {
|
||||
missing = append(missing, d)
|
||||
}
|
||||
}
|
||||
|
||||
if len(missing) == 0 && stale == 0 {
|
||||
log.Println("No new documents - related.json is up to date.")
|
||||
return
|
||||
}
|
||||
|
||||
log.Printf("Computing related docs for %d documents...", len(missing))
|
||||
maps.Copy(existing, co.buildRelatedDocs(missing, co.archive.Docs))
|
||||
|
||||
if err := saveRelatedDocs(existing); err != nil {
|
||||
log.Fatalf("Failed to save related docs: %v", err)
|
||||
}
|
||||
log.Printf("Saved related.json with %d entries (%d computed, %d stale dropped, other entries and manual edits preserved)",
|
||||
len(existing), len(missing), stale)
|
||||
return
|
||||
}
|
||||
|
||||
handler := withBaseHeaders(newMux())
|
||||
|
||||
srv := &http.Server{
|
||||
Addr: *listen,
|
||||
Handler: handler,
|
||||
ReadHeaderTimeout: 5 * time.Second,
|
||||
ReadTimeout: 15 * time.Second,
|
||||
WriteTimeout: writeTimeout, // downloads set their own: fileWriteBudget
|
||||
IdleTimeout: 60 * time.Second,
|
||||
MaxHeaderBytes: 64 << 10,
|
||||
}
|
||||
|
||||
warnNonLoopback(*listen)
|
||||
log.Printf("Archive Server listening on %s (data: %s, project: %s)", srv.Addr, dataDir, projectDir)
|
||||
|
||||
errCh := make(chan error, 1)
|
||||
go func() { errCh <- srv.ListenAndServe() }()
|
||||
sig := make(chan os.Signal, 1)
|
||||
signal.Notify(sig, os.Interrupt, syscall.SIGTERM)
|
||||
select {
|
||||
case err := <-errCh:
|
||||
log.Fatal(err)
|
||||
case s := <-sig:
|
||||
signal.Reset(os.Interrupt, syscall.SIGTERM)
|
||||
log.Printf("%s received - draining connections (again to stop now)", s)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), writeTimeout)
|
||||
defer cancel()
|
||||
if err := srv.Shutdown(ctx); err != nil {
|
||||
log.Printf("Shutdown: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (co *Corpus) prerenderPages() {
|
||||
co.prerendered = map[string]template.HTML{}
|
||||
co.docRowHTML = nil
|
||||
if devMode {
|
||||
return
|
||||
}
|
||||
exec := func(key, page, block string, data interface{}) {
|
||||
t := tmplSet[page]
|
||||
if t == nil {
|
||||
return
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
if err := t.ExecuteTemplate(&buf, block, data); err != nil {
|
||||
warnf("prerender %s/%s: %v - rendering it per request", page, block, err)
|
||||
return
|
||||
}
|
||||
co.prerendered[key] = template.HTML(buf.String())
|
||||
}
|
||||
|
||||
exec("ids", "ids.html", "id-rows", &idsPage{Rows: co.idRows})
|
||||
exec("archive-timeline", "browse.html", "archive-timeline",
|
||||
&browsePage{ArchiveYears: co.archiveTimeline})
|
||||
exec("glossary", "glossary.html", "glossary-groups",
|
||||
&glossaryPage{Groups: co.glossaryGroups})
|
||||
exec("articles", "articles.html", "article-groups",
|
||||
&articlesPage{Groups: co.articleGroups})
|
||||
for key, v := range co.codewordViews {
|
||||
exec("codewords:"+key, "codewords.html", "codeword-rows",
|
||||
&codewordsPage{Rows: v.Rows})
|
||||
}
|
||||
for set, tables := range co.exploreTables {
|
||||
for i, t := range tables {
|
||||
exec(exploreKey(set, i), "explore_table.html", "explore-rows",
|
||||
&exploreTablePage{Rows: t.Rows})
|
||||
}
|
||||
}
|
||||
co.prerenderCategoryRows()
|
||||
co.prerenderDocRows()
|
||||
total := 0
|
||||
for _, h := range co.prerendered {
|
||||
total += len(h)
|
||||
}
|
||||
log.Printf("Pre-rendered %d page bodies (%d KB)", len(co.prerendered), total/1024)
|
||||
}
|
||||
|
||||
func (co *Corpus) preHTML(key string) template.HTML { return co.prerendered[key] }
|
||||
|
||||
func newMux() *http.ServeMux {
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("GET /{$}", bind((*Corpus).handleIndex))
|
||||
mux.HandleFunc("GET /random", bind((*Corpus).handleRandom))
|
||||
mux.HandleFunc("GET /glossary", bind((*Corpus).handleGlossary))
|
||||
mux.HandleFunc("GET /preferences", handlePreferences)
|
||||
mux.HandleFunc("POST /preferences/save", handlePreferencesSave)
|
||||
mux.HandleFunc("GET /pinned", bind((*Corpus).handlePinnedPage))
|
||||
mux.HandleFunc("GET /pinned/export", bind((*Corpus).handlePinnedExport))
|
||||
mux.HandleFunc("POST /pin/{id}", bind((*Corpus).handlePinToggle))
|
||||
mux.HandleFunc("GET /articles", bind((*Corpus).handleArticles))
|
||||
mux.HandleFunc("GET /article/{key}", bind((*Corpus).handleArticleView))
|
||||
mux.HandleFunc("GET /changelog", bind((*Corpus).handleChangelog))
|
||||
mux.HandleFunc("POST /history/clear", handleHistoryClear)
|
||||
mux.HandleFunc("POST /pinsearch", bind((*Corpus).handleSearchPinToggle))
|
||||
mux.HandleFunc("GET /robots.txt", handleRobots)
|
||||
mux.HandleFunc("GET /sitemap.xml", bind((*Corpus).handleSitemap))
|
||||
mux.HandleFunc("/", notFound)
|
||||
mux.HandleFunc("GET /browse", bind((*Corpus).handleBrowseRoot))
|
||||
mux.HandleFunc("GET /browse/{category}", bind((*Corpus).handleCategoryList))
|
||||
mux.HandleFunc("GET /explore", bind((*Corpus).handleExploreIndex))
|
||||
mux.HandleFunc("GET /explore/{metric}", bind((*Corpus).handleExploreMetric))
|
||||
mux.HandleFunc("GET /explore/terms", bind((*Corpus).handleCodewords))
|
||||
mux.HandleFunc("GET /explore/ids", bind((*Corpus).handleExploreIDs))
|
||||
mux.HandleFunc("GET /explore/ids.txt", bind((*Corpus).handleExploreIDsText))
|
||||
mux.HandleFunc("GET /explore/classification", bind((*Corpus).handleClassPatterns))
|
||||
mux.HandleFunc("GET /explore/portions", bind((*Corpus).handlePortions))
|
||||
mux.HandleFunc("GET /explore/outlets", bind((*Corpus).handleOutlets))
|
||||
mux.HandleFunc("GET /explore/assemblies", bind((*Corpus).handleAssemblies))
|
||||
mux.HandleFunc("GET /explore/assemblies/{id}", bind((*Corpus).handleAssemblyView))
|
||||
mux.HandleFunc("GET /browse/{category}/{value}", bind((*Corpus).handleCategoryFilter))
|
||||
mux.HandleFunc("GET /doc/{id}", bind((*Corpus).handleDocView))
|
||||
mux.HandleFunc("GET /doc/{id}/pages", bind((*Corpus).handleDocPages))
|
||||
mux.HandleFunc("GET /doc/{id}/versions/{slug}", bind((*Corpus).handleVersionView))
|
||||
mux.HandleFunc("GET /doc/{id}/versions/{slug}/pages", bind((*Corpus).handleVersionPages))
|
||||
mux.HandleFunc("GET /search", bind((*Corpus).handleSearch))
|
||||
mux.HandleFunc("GET /about", handleAbout)
|
||||
mux.HandleFunc("GET /files/", bind((*Corpus).handleFiles))
|
||||
mux.HandleFunc("GET /static/", handleStatic)
|
||||
|
||||
// JSON API (read-only) - see api.go for endpoint documentation.
|
||||
if apiEnabled {
|
||||
mux.HandleFunc("GET /api/docs", bind((*Corpus).handleAPIDocs))
|
||||
mux.HandleFunc("GET /api/doc/{id}", bind((*Corpus).handleAPIDoc))
|
||||
mux.HandleFunc("GET /api/search", bind((*Corpus).handleAPISearch))
|
||||
mux.HandleFunc("GET /api/facets", bind((*Corpus).handleAPIFacets))
|
||||
mux.HandleFunc("/api/", func(w http.ResponseWriter, r *http.Request) {
|
||||
apiError(w, http.StatusNotFound, "unknown API endpoint")
|
||||
})
|
||||
} else {
|
||||
mux.HandleFunc("/api/", func(w http.ResponseWriter, r *http.Request) {
|
||||
apiError(w, http.StatusForbidden, "the JSON API is disabled on this instance")
|
||||
})
|
||||
}
|
||||
|
||||
return mux
|
||||
}
|
||||
|
||||
// withBaseHeaders wraps a handler with the headers every response carries.
|
||||
func withBaseHeaders(next http.Handler) http.Handler {
|
||||
const csp = "default-src 'self'; img-src 'self'; script-src 'self'; " +
|
||||
"style-src 'self' 'unsafe-inline'; object-src 'none'; base-uri 'none'; " +
|
||||
"form-action 'self'; frame-ancestors 'self'"
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
h := w.Header()
|
||||
h.Set("Content-Security-Policy", csp)
|
||||
h.Set("X-Frame-Options", "SAMEORIGIN")
|
||||
h.Set("X-Content-Type-Options", "nosniff")
|
||||
h.Set("Referrer-Policy", "no-referrer")
|
||||
switch r.Method {
|
||||
case http.MethodGet, http.MethodHead, http.MethodPost:
|
||||
default:
|
||||
h.Set("Allow", "GET, HEAD, POST")
|
||||
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
next.ServeHTTP(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
func envOr(key, fallback string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
return v
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
|
||||
func envBool(key string) bool {
|
||||
v := os.Getenv(key)
|
||||
if v == "" {
|
||||
return false
|
||||
}
|
||||
b, err := strconv.ParseBool(v)
|
||||
if err != nil {
|
||||
log.Fatalf("$%s=%q is not a boolean (use 1/0 or true/false)", key, v)
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func envDuration(key string) time.Duration {
|
||||
v := os.Getenv(key)
|
||||
if v == "" {
|
||||
return 0
|
||||
}
|
||||
d, err := time.ParseDuration(v)
|
||||
if err != nil {
|
||||
log.Fatalf("$%s=%q is not a duration (use e.g. 90s, 10m)", key, v)
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// accelPath is the X-Accel-Redirect target for a file: its path relative to
|
||||
// the data directory under accelPrefix, percent-encoded because nginx decodes
|
||||
// the header as a URI.
|
||||
func accelPath(file string) (string, bool) {
|
||||
rel, err := filepath.Rel(dataDir, file)
|
||||
if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
||||
return "", false
|
||||
}
|
||||
u := url.URL{Path: accelPrefix + filepath.ToSlash(rel)}
|
||||
return u.EscapedPath(), true
|
||||
}
|
||||
|
||||
// handleFiles serves the files the loader indexed for a document, and no
|
||||
// other file its directory may hold (Document.servedFile).
|
||||
func (co *Corpus) handleFiles(w http.ResponseWriter, r *http.Request) {
|
||||
rest := strings.TrimPrefix(r.URL.Path, "/files/")
|
||||
id, sub, ok := strings.Cut(rest, "/")
|
||||
if !ok || sub == "" {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
doc := co.docIndex[id]
|
||||
if doc == nil {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
|
||||
sub = path.Clean("/" + sub)
|
||||
rel := strings.TrimPrefix(sub, "/")
|
||||
if strings.Contains(sub, "..") || !doc.servedFile(rel) {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
|
||||
cacheIfFile(w, os.DirFS(doc.Dir), rel, "public, max-age=86400")
|
||||
if isTranscript(rel) {
|
||||
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
|
||||
}
|
||||
file := filepath.Join(doc.Dir, filepath.FromSlash(sub))
|
||||
if accelPrefix != "" {
|
||||
if target, ok := accelPath(file); ok {
|
||||
w.Header().Set("X-Accel-Redirect", target)
|
||||
return
|
||||
}
|
||||
}
|
||||
_ = http.NewResponseController(w).SetWriteDeadline(time.Now().Add(fileWriteBudget))
|
||||
http.ServeFile(w, r, file)
|
||||
}
|
||||
|
||||
const writeTimeout = 30 * time.Second
|
||||
|
||||
// fileWriteBudget is how long a single download may take in total. A 123 MB
|
||||
// PDF over a bad mobile link (~35 KB/s) fits in an hour, so in practice this
|
||||
// bound is only reached by a reader that has gone away.
|
||||
var fileWriteBudget = time.Hour
|
||||
|
||||
func cacheIfFile(w http.ResponseWriter, fsys fs.FS, name, policy string) {
|
||||
if fi, err := fs.Stat(fsys, name); err == nil && fi.Mode().IsRegular() {
|
||||
w.Header().Set("Cache-Control", policy)
|
||||
}
|
||||
}
|
||||
|
||||
func handleStatic(w http.ResponseWriter, r *http.Request) {
|
||||
if path.Ext(r.URL.Path) == "" {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
rel := strings.TrimPrefix(path.Clean("/"+strings.TrimPrefix(r.URL.Path, "/static/")), "/")
|
||||
if h, ok := staticHashes[rel]; ok {
|
||||
if r.URL.Query().Get("v") == h {
|
||||
w.Header().Set("Cache-Control", "public, max-age=31536000, immutable")
|
||||
}
|
||||
w.Header().Set("ETag", `"`+h+`"`)
|
||||
}
|
||||
http.StripPrefix("/static/", http.FileServerFS(staticFS)).ServeHTTP(w, r)
|
||||
}
|
||||
|
||||
func navSection(path string) string {
|
||||
switch {
|
||||
case path == "/":
|
||||
return "home"
|
||||
case path == "/browse" || strings.HasPrefix(path, "/browse/"):
|
||||
return "browse"
|
||||
case path == "/search":
|
||||
return "search"
|
||||
case path == "/articles" || strings.HasPrefix(path, "/article/"):
|
||||
return "articles"
|
||||
case path == "/glossary":
|
||||
return "glossary"
|
||||
case path == "/about":
|
||||
return "about"
|
||||
case path == "/pinned":
|
||||
return "pinned"
|
||||
case path == "/preferences":
|
||||
return "preferences"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func render(w http.ResponseWriter, r *http.Request, data pageData) {
|
||||
renderCode(w, r, data, http.StatusOK)
|
||||
}
|
||||
|
||||
func crossSitePOST(r *http.Request) bool {
|
||||
switch r.Header.Get("Sec-Fetch-Site") {
|
||||
case "same-origin", "none":
|
||||
return false
|
||||
case "cross-site", "same-site":
|
||||
return true
|
||||
}
|
||||
|
||||
if o := r.Header.Get("Origin"); o != "" && o != "null" {
|
||||
if u, err := url.Parse(o); err == nil && !originIsHost(u, r.Host) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func originIsHost(origin *url.URL, host string) bool {
|
||||
def := map[string]string{"http": "80", "https": "443"}[strings.ToLower(origin.Scheme)]
|
||||
op := origin.Port()
|
||||
if op == "" {
|
||||
op = def
|
||||
}
|
||||
h, hp, err := net.SplitHostPort(host)
|
||||
if err != nil { // no port
|
||||
h, hp = strings.Trim(host, "[]"), def
|
||||
}
|
||||
return strings.EqualFold(origin.Hostname(), h) && op == hp
|
||||
}
|
||||
|
||||
func noStore(w http.ResponseWriter) {
|
||||
w.Header().Set("Cache-Control", "private, no-store")
|
||||
}
|
||||
|
||||
func readPOSTForm(w http.ResponseWriter, r *http.Request) bool {
|
||||
if crossSitePOST(r) {
|
||||
http.Error(w, "cross-site request rejected", http.StatusForbidden)
|
||||
return false
|
||||
}
|
||||
r.Body = http.MaxBytesReader(w, r.Body, 1<<16)
|
||||
if err := r.ParseForm(); err != nil {
|
||||
http.Error(w, "bad form", http.StatusBadRequest)
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func crossSiteFetch(r *http.Request) bool {
|
||||
switch r.Header.Get("Sec-Fetch-Site") {
|
||||
case "cross-site", "same-site":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func renderCode(w http.ResponseWriter, r *http.Request, data pageData, code int) {
|
||||
name := data.templateName()
|
||||
if w.Header().Get("Cache-Control") == "" {
|
||||
if (r.Method == http.MethodGet || r.Method == http.MethodHead) &&
|
||||
(code == http.StatusOK || code == http.StatusNotFound) &&
|
||||
r.Header.Get("Cookie") == "" {
|
||||
w.Header().Set("Cache-Control", "public, max-age=10")
|
||||
w.Header().Set("Vary", "Cookie")
|
||||
if accelExpires != "" {
|
||||
w.Header().Set("X-Accel-Expires", accelExpires)
|
||||
}
|
||||
} else {
|
||||
w.Header().Set("Cache-Control", "private, no-cache")
|
||||
}
|
||||
}
|
||||
|
||||
b := data.base()
|
||||
b.Nav = navSection(r.URL.Path)
|
||||
b.APIEnabled = apiEnabled
|
||||
// A handler that read the reader's preferences passes them on
|
||||
// (pageWith); they are only read here for one that had no use for them.
|
||||
if !b.hasPrefs {
|
||||
b.Prefs = readPrefs(r)
|
||||
}
|
||||
b.Theme = b.Prefs.ThemeAttr()
|
||||
b.CardPagePreview = b.Prefs.CardPagePreview
|
||||
|
||||
var t *template.Template
|
||||
var err error
|
||||
|
||||
if devMode {
|
||||
t, err = parsePage(name)
|
||||
if err != nil {
|
||||
log.Printf("Template Parse Error (%s): %v", name, err)
|
||||
serverError(w, err)
|
||||
return
|
||||
}
|
||||
} else {
|
||||
t = tmplSet[name]
|
||||
if t == nil {
|
||||
log.Printf("Template not found: %s", name)
|
||||
serverError(w, fmt.Errorf("template not found: %s", name))
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
var buf bytes.Buffer
|
||||
if err := t.ExecuteTemplate(&buf, name, data); err != nil {
|
||||
log.Printf("Template Exec Error (%s): %v", name, err)
|
||||
serverError(w, err)
|
||||
return
|
||||
}
|
||||
if code != http.StatusOK {
|
||||
w.WriteHeader(code)
|
||||
}
|
||||
w.Write(bytes.TrimSpace(buf.Bytes()))
|
||||
w.Write([]byte("\n"))
|
||||
}
|
||||
|
||||
func notFound(w http.ResponseWriter, r *http.Request) {
|
||||
renderCode(w, r, ¬FoundPage{Path: r.URL.Path}, http.StatusNotFound)
|
||||
}
|
||||
|
||||
func parsePage(name string) (*template.Template, error) {
|
||||
return template.New(name).Funcs(funcMap).ParseFS(templateFS, "base.html", "partials.html", name)
|
||||
}
|
||||
|
||||
func parseTemplates() error {
|
||||
pages, err := fs.Glob(templateFS, "*.html")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmplSet = make(map[string]*template.Template)
|
||||
for _, name := range pages {
|
||||
if name == "base.html" || name == "partials.html" {
|
||||
continue
|
||||
}
|
||||
t, err := parsePage(name)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%s: %w", name, err)
|
||||
}
|
||||
tmplSet[name] = t
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func serverError(w http.ResponseWriter, err error) {
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
if devMode {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
|
||||
}
|
||||
|
||||
func warnNonLoopback(addr string) {
|
||||
host, _, err := net.SplitHostPort(addr)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if host == "localhost" {
|
||||
return
|
||||
}
|
||||
if ip := net.ParseIP(host); ip != nil && ip.IsLoopback() {
|
||||
return
|
||||
}
|
||||
warnf("-listen %s is not loopback - anything that can route to this host reaches the archive directly, bypassing your proxy's caching, rate limits and logs.", addr)
|
||||
}
|
||||
591
mdparser.go
Normal file
591
mdparser.go
Normal file
|
|
@ -0,0 +1,591 @@
|
|||
// mdparser.go - minimal dependency-free Markdown parser.
|
||||
//
|
||||
// Input trust: everything rendered through this parser is the archive's own
|
||||
// curator-produced OCR never user-supplied input.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"html"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
// -- Inline patterns ----------------------------------------------------
|
||||
|
||||
var (
|
||||
reInlineCode = regexp.MustCompile("`(.+?)`")
|
||||
reStrikethrough = regexp.MustCompile(`~~(.+?)~~`)
|
||||
reHardBreak = regexp.MustCompile(`(?i)<br\s*/?>`)
|
||||
reEscape = regexp.MustCompile("\\\\([!-/:-@\\[-`|{}~])")
|
||||
reOrderedItem = regexp.MustCompile(`^(\s*)(\d{1,9})\.(?:\s+(.*))?\s*$`)
|
||||
reUnorderedItem = regexp.MustCompile(`^(\s*)[-*+](?:\s+(.*))?\s*$`)
|
||||
reTableSep = regexp.MustCompile(`^\|[-| :]+\|$`)
|
||||
)
|
||||
|
||||
// parseInline converts inline Markdown to HTML.
|
||||
func parseInline(s string) string {
|
||||
s = strings.ReplaceAll(s, "\x00", "")
|
||||
s = strings.ReplaceAll(s, "\x01", "")
|
||||
s = html.EscapeString(s)
|
||||
|
||||
// Backslash escapes (CommonMark): "\*" is a literal "*", not emphasis
|
||||
// syntax. Pull escaped ASCII punctuation out to placeholders BEFORE the
|
||||
// emphasis passes so the character can't be interpreted as a marker,
|
||||
// then restore it at the end.
|
||||
var escaped []string
|
||||
if strings.IndexByte(s, '\\') >= 0 {
|
||||
s = reEscape.ReplaceAllStringFunc(s, func(m string) string {
|
||||
// m is "\X": EscapeString leaves the backslash alone. Keep
|
||||
// the char after it.
|
||||
ch := m[len(m)-1:]
|
||||
escaped = append(escaped, ch)
|
||||
return "\x01" + strconv.Itoa(len(escaped)-1) + "\x01"
|
||||
})
|
||||
}
|
||||
|
||||
// Extract inline code spans next and replace them with placeholders so
|
||||
// the strikethrough/bold/italic passes can't reformat code content
|
||||
// (e.g. `*x*` must render as literal *x*, not <em>x</em>). Escape
|
||||
// placeholders that landed INSIDE a code span are put back as their
|
||||
// original "\X" source text first: backslash escapes are not escapes
|
||||
// within code spans (CommonMark), so `\*` renders as \*, not *. The
|
||||
// escape pass still runs before this one on purpose - that ordering is
|
||||
// what lets \` be a literal backtick that cannot OPEN a code span.
|
||||
var codes []string
|
||||
if strings.IndexByte(s, '`') >= 0 {
|
||||
s = reInlineCode.ReplaceAllStringFunc(s, func(m string) string {
|
||||
inner := reInlineCode.FindStringSubmatch(m)[1]
|
||||
for i, ch := range escaped {
|
||||
inner = strings.ReplaceAll(inner, "\x01"+strconv.Itoa(i)+"\x01", "\\"+ch)
|
||||
}
|
||||
codes = append(codes, "<code>"+inner+"</code>")
|
||||
return "\x00" + strconv.Itoa(len(codes)-1) + "\x00"
|
||||
})
|
||||
}
|
||||
|
||||
// After the code-span pass, so `<br>` inside backticks stays literal.
|
||||
if strings.Contains(s, "<") {
|
||||
s = reHardBreak.ReplaceAllString(s, "<br>")
|
||||
}
|
||||
if strings.Contains(s, "~~") {
|
||||
s = reStrikethrough.ReplaceAllString(s, `<del>$1</del>`)
|
||||
}
|
||||
// ***x*** first, so it nests as <em><strong> rather than crossing.
|
||||
s = emphasize(s, '*', 3, "<em><strong>", "</strong></em>")
|
||||
s = emphasize(s, '*', 2, "<strong>", "</strong>")
|
||||
s = emphasize(s, '_', 2, "<strong>", "</strong>")
|
||||
s = emphasize(s, '*', 1, "<em>", "</em>")
|
||||
s = emphasize(s, '_', 1, "<em>", "</em>")
|
||||
|
||||
// Restore code spans.
|
||||
for i, code := range codes {
|
||||
s = strings.Replace(s, "\x00"+strconv.Itoa(i)+"\x00", code, 1)
|
||||
}
|
||||
// Restore backslash-escaped characters as their literal selves.
|
||||
// (Placeholders that sat inside a code span were already restored to
|
||||
// their "\X" source form up in the code-extraction pass and no longer
|
||||
// occur here; Replace on an absent placeholder is a no-op.)
|
||||
for i, ch := range escaped {
|
||||
s = strings.Replace(s, "\x01"+strconv.Itoa(i)+"\x01", ch, 1)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// emphasize wraps the text between two runs of k delimiters c ('*' or '_')
|
||||
// in open/close. It pairs them as CommonMark does: left to right, a run
|
||||
// that can close gives its first k delimiters to the nearest unpaired
|
||||
// opener before it, and a run that can open offers its last k to a later
|
||||
// closer. Which runs can open or close follows CommonMark's flanking rules
|
||||
// (delimiterRun).
|
||||
func emphasize(s string, c byte, k int, open, close string) string {
|
||||
var openers []int // offsets of unpaired openers, nearest last
|
||||
tags := map[int]string{} // offset of k paired delimiters -> their tag
|
||||
for i := 0; i < len(s); {
|
||||
if s[i] != c {
|
||||
i++
|
||||
continue
|
||||
}
|
||||
end := i
|
||||
for end < len(s) && s[end] == c {
|
||||
end++
|
||||
}
|
||||
canOpen, canClose := delimiterRun(s, i, end, c)
|
||||
n := end - i
|
||||
if n >= k && canClose && len(openers) > 0 {
|
||||
tags[openers[len(openers)-1]], tags[i] = open, close
|
||||
openers = openers[:len(openers)-1]
|
||||
n -= k
|
||||
}
|
||||
if n >= k && canOpen {
|
||||
openers = append(openers, end-k)
|
||||
}
|
||||
i = end
|
||||
}
|
||||
if len(tags) == 0 {
|
||||
return s
|
||||
}
|
||||
var b strings.Builder
|
||||
for i := 0; i < len(s); i++ {
|
||||
if t, ok := tags[i]; ok {
|
||||
b.WriteString(t)
|
||||
i += k - 1
|
||||
} else {
|
||||
b.WriteByte(s[i])
|
||||
}
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// delimiterRun reports whether the run of delimiters c at s[start:end]
|
||||
// can open and can close emphasis, by CommonMark's flanking rules, which
|
||||
// look at the whole run.
|
||||
func delimiterRun(s string, start, end int, c byte) (canOpen, canClose bool) {
|
||||
prev, next := ' ', ' '
|
||||
if start > 0 {
|
||||
prev, _ = utf8.DecodeLastRuneInString(s[:start])
|
||||
}
|
||||
if end < len(s) {
|
||||
next, _ = utf8.DecodeRuneInString(s[end:])
|
||||
}
|
||||
left := !unicode.IsSpace(next) &&
|
||||
(!isPunct(next) || unicode.IsSpace(prev) || isPunct(prev))
|
||||
right := !unicode.IsSpace(prev) &&
|
||||
(!isPunct(prev) || unicode.IsSpace(next) || isPunct(next))
|
||||
if c == '_' {
|
||||
return left && (!right || isPunct(prev)), right && (!left || isPunct(next))
|
||||
}
|
||||
return left, right
|
||||
}
|
||||
|
||||
// isPunct is CommonMark's punctuation (Unicode P and S). The \x00/\x01
|
||||
// placeholders count too: they stand for a code span (opened by a
|
||||
// backtick) or a backslash-escaped punctuation character.
|
||||
func isPunct(r rune) bool {
|
||||
return r == 0 || r == 1 || unicode.IsPunct(r) || unicode.IsSymbol(r)
|
||||
}
|
||||
|
||||
// -- Block-level parser --------------------------------------------
|
||||
|
||||
type parser struct {
|
||||
lines []string
|
||||
pos int
|
||||
out strings.Builder
|
||||
open []listLevel // the lists parseList is inside, outermost first
|
||||
// numbered holds the last number of the ordered list at each indent,
|
||||
// for as long as list lines keep coming (see endsItem).
|
||||
numbered map[int]int
|
||||
// top is the source's highest heading level and last the level of the
|
||||
// heading emitted last (see parseHeading).
|
||||
top, last int
|
||||
}
|
||||
|
||||
// listLevel is one open list: the indent of its items and their kind.
|
||||
type listLevel struct {
|
||||
indent int
|
||||
ordered bool
|
||||
}
|
||||
|
||||
func newParser(src string) *parser {
|
||||
src = strings.ReplaceAll(src, "\r\n", "\n")
|
||||
src = strings.ReplaceAll(src, "\r", "\n")
|
||||
p := &parser{lines: strings.Split(src, "\n"), top: 7, last: 1}
|
||||
fence := ""
|
||||
for _, line := range p.lines {
|
||||
switch {
|
||||
case fence != "":
|
||||
if strings.HasPrefix(line, fence) {
|
||||
fence = ""
|
||||
}
|
||||
case isFencedCodeStart(line):
|
||||
fence = line[:3]
|
||||
case isHeading(line):
|
||||
p.top = min(p.top, headingLevel(line))
|
||||
}
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
func (p *parser) peek() (string, bool) {
|
||||
if p.pos >= len(p.lines) {
|
||||
return "", false
|
||||
}
|
||||
return p.lines[p.pos], true
|
||||
}
|
||||
|
||||
func (p *parser) next() (string, bool) {
|
||||
l, ok := p.peek()
|
||||
if ok {
|
||||
p.pos++
|
||||
}
|
||||
return l, ok
|
||||
}
|
||||
|
||||
func (p *parser) emit(s string) { p.out.WriteString(s) }
|
||||
|
||||
// parse processes all lines and returns HTML.
|
||||
func (p *parser) parse() string {
|
||||
for p.pos < len(p.lines) {
|
||||
line, _ := p.peek()
|
||||
if !isListItem(line) {
|
||||
p.numbered = nil // the run of list lines is over
|
||||
}
|
||||
|
||||
switch {
|
||||
case isFencedCodeStart(line):
|
||||
p.parseFencedCode()
|
||||
case isHR(line):
|
||||
p.next()
|
||||
p.emit("<hr>\n")
|
||||
case isHeading(line):
|
||||
p.parseHeading()
|
||||
case isTableRow(line):
|
||||
p.parseTable()
|
||||
case isListItem(line):
|
||||
p.parseList(reOrderedItem.MatchString(line), indentOf(line))
|
||||
case strings.TrimSpace(line) == "":
|
||||
p.next()
|
||||
default:
|
||||
p.parseParagraph()
|
||||
}
|
||||
}
|
||||
return p.out.String()
|
||||
}
|
||||
|
||||
// -- Heading --------------------------------------------------
|
||||
|
||||
// headingLevel returns the heading level of an ATX heading line - one to
|
||||
// six '#' followed by whitespace or the end of the line - or 0 when the
|
||||
// line is not a heading.
|
||||
func headingLevel(s string) int {
|
||||
n := 0
|
||||
for n < len(s) && s[n] == '#' {
|
||||
n++
|
||||
}
|
||||
if n == 0 || n > 6 || (n < len(s) && s[n] != ' ' && s[n] != '\t') {
|
||||
return 0
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func isHeading(s string) bool { return headingLevel(s) > 0 }
|
||||
|
||||
// parseHeading emits a heading under the page's own <h1>: the source's top
|
||||
// level becomes <h2> and no heading skips a level. The md-hN class keeps the
|
||||
// size of the level the source wrote.
|
||||
func (p *parser) parseHeading() {
|
||||
line, _ := p.next()
|
||||
level := headingLevel(line)
|
||||
content := strings.TrimSpace(line[level:])
|
||||
tag := min(level-p.top+2, p.last+1, 6)
|
||||
p.last = tag
|
||||
p.emit(fmt.Sprintf("<h%d class=\"md-h%d\">%s</h%d>\n", tag, level, parseInline(content), tag))
|
||||
}
|
||||
|
||||
// -- Horizontal rule -------------------------------------------------
|
||||
|
||||
// isHR reports whether s is a thematic break: three or more of the same
|
||||
// '-', '*' or '_', optionally with spaces or tabs between them.
|
||||
func isHR(s string) bool {
|
||||
t := strings.TrimSpace(s)
|
||||
if t == "" || (t[0] != '-' && t[0] != '*' && t[0] != '_') {
|
||||
return false
|
||||
}
|
||||
n := 0
|
||||
for i := 0; i < len(t); i++ {
|
||||
switch t[i] {
|
||||
case t[0]:
|
||||
n++
|
||||
case ' ', '\t':
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
return n >= 3
|
||||
}
|
||||
|
||||
// Fenced code block ---------------------------------------------------------
|
||||
|
||||
func isFencedCodeStart(s string) bool {
|
||||
return strings.HasPrefix(s, "```") || strings.HasPrefix(s, "~~~")
|
||||
}
|
||||
|
||||
func (p *parser) parseFencedCode() {
|
||||
opener, _ := p.next()
|
||||
fence := opener[:3]
|
||||
|
||||
var body strings.Builder
|
||||
for {
|
||||
line, ok := p.next()
|
||||
if !ok || strings.HasPrefix(line, fence) {
|
||||
break
|
||||
}
|
||||
body.WriteString(html.EscapeString(line))
|
||||
body.WriteByte('\n')
|
||||
}
|
||||
p.emit("<pre><code>" + body.String() + "</code></pre>\n")
|
||||
}
|
||||
|
||||
// -- Lists (unified, handles nesting) -----------------------------------------
|
||||
|
||||
// indentOf returns the number of leading whitespace characters in s.
|
||||
func indentOf(s string) int {
|
||||
return len(s) - len(strings.TrimLeft(s, " \t"))
|
||||
}
|
||||
|
||||
// isListItem returns true if the line is a ul or ol item at any indent.
|
||||
// A thematic break wins over a bullet ("* * *" is a rule, not an item).
|
||||
func isListItem(s string) bool {
|
||||
// A list marker is the first non-space byte: rule out the rest (most
|
||||
// lines) before any regex runs.
|
||||
t := strings.TrimLeft(s, "\t\n\f\r ")
|
||||
if t == "" || (t[0] != '-' && t[0] != '*' && t[0] != '+' && (t[0] < '0' || t[0] > '9')) {
|
||||
return false
|
||||
}
|
||||
return !isHR(s) && (reUnorderedItem.MatchString(s) || reOrderedItem.MatchString(s))
|
||||
}
|
||||
|
||||
// itemContent strips the list marker and returns the item text (possibly
|
||||
// empty for a bare marker like "6.").
|
||||
func itemContent(s string) string {
|
||||
if sub := reUnorderedItem.FindStringSubmatch(s); sub != nil {
|
||||
return sub[2]
|
||||
}
|
||||
if sub := reOrderedItem.FindStringSubmatch(s); sub != nil {
|
||||
return sub[3]
|
||||
}
|
||||
return strings.TrimSpace(s)
|
||||
}
|
||||
|
||||
// orderedNum returns the source number of an ordered-list line ("7." -> 7)
|
||||
// and true, or false if the line isn't an ordered item. "0." is a real
|
||||
// number, so it cannot double as the "not ordered" answer.
|
||||
func orderedNum(s string) (int, bool) {
|
||||
if sub := reOrderedItem.FindStringSubmatch(s); sub != nil {
|
||||
n, _ := strconv.Atoi(sub[2]) // at most nine digits: cannot fail
|
||||
return n, true
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// startsList reports whether list-item line s may open a new list on the
|
||||
// line after text, where it would otherwise continue that text.
|
||||
func startsList(s string) bool {
|
||||
if itemContent(s) == "" {
|
||||
return false
|
||||
}
|
||||
n, ok := orderedNum(s)
|
||||
return !ok || n == 1
|
||||
}
|
||||
|
||||
// endsItem reports whether line s, met while collecting the lazily
|
||||
// continued text of a list item, ends that item rather than continuing it.
|
||||
func (p *parser) endsItem(s string) bool {
|
||||
if isHR(s) || isHeading(s) || isFencedCodeStart(s) || isTableRow(s) {
|
||||
return true
|
||||
}
|
||||
if !isListItem(s) {
|
||||
return false
|
||||
}
|
||||
if startsList(s) {
|
||||
return true
|
||||
}
|
||||
indent, ordered := indentOf(s), reOrderedItem.MatchString(s)
|
||||
for _, l := range p.open {
|
||||
if l.indent == indent && l.ordered == ordered {
|
||||
return true
|
||||
}
|
||||
}
|
||||
if n, ok := orderedNum(s); ok {
|
||||
for _, last := range p.numbered {
|
||||
if n > last && n <= last+2 {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// parseList recursively parses a ul or ol at the given indent level.
|
||||
func (p *parser) parseList(ordered bool, minIndent int) {
|
||||
tag := "ul"
|
||||
if ordered {
|
||||
tag = "ol"
|
||||
}
|
||||
p.emit("<" + tag + ">\n")
|
||||
p.open = append(p.open, listLevel{minIndent, ordered})
|
||||
defer func() { p.open = p.open[:len(p.open)-1] }()
|
||||
|
||||
expected := 0 // running ordered-list counter, for value="N" decisions
|
||||
|
||||
for {
|
||||
line, ok := p.peek()
|
||||
if !ok || strings.TrimSpace(line) == "" || !isListItem(line) {
|
||||
break
|
||||
}
|
||||
indent := indentOf(line)
|
||||
if indent < minIndent {
|
||||
// Belongs to a parent list - stop.
|
||||
break
|
||||
}
|
||||
if indent == minIndent && reOrderedItem.MatchString(line) != ordered {
|
||||
// Bullets after numbers, or numbers after bullets: a new
|
||||
// list (as in CommonMark). Kept in this one, a number would
|
||||
// lose itself to a bullet and a bullet would gain a number.
|
||||
break
|
||||
}
|
||||
if indent > minIndent {
|
||||
// Deeper indent mid-list without a parent <li> - treat as nested.
|
||||
p.parseList(reOrderedItem.MatchString(line), indent)
|
||||
continue
|
||||
}
|
||||
|
||||
p.next()
|
||||
|
||||
// Item body: the marker line plus any lazily-continued lines after it.
|
||||
var body []string
|
||||
if c := itemContent(line); c != "" {
|
||||
body = append(body, c)
|
||||
}
|
||||
for {
|
||||
cont, ok := p.peek()
|
||||
if !ok || strings.TrimSpace(cont) == "" || p.endsItem(cont) {
|
||||
break
|
||||
}
|
||||
p.next()
|
||||
body = append(body, strings.TrimSpace(cont))
|
||||
}
|
||||
|
||||
liOpen := "<li>"
|
||||
if ordered {
|
||||
n, _ := orderedNum(line)
|
||||
expected++
|
||||
if n != expected {
|
||||
liOpen = "<li value=\"" + strconv.Itoa(n) + "\">"
|
||||
expected = n
|
||||
}
|
||||
if p.numbered == nil {
|
||||
p.numbered = map[int]int{}
|
||||
}
|
||||
p.numbered[minIndent] = n
|
||||
}
|
||||
|
||||
var parts []string
|
||||
for _, l := range body {
|
||||
parts = append(parts, parseInline(l))
|
||||
}
|
||||
text := strings.Join(parts, "<br>\n")
|
||||
|
||||
// Peek: if next line is deeper, nest a list inside this <li> -
|
||||
// as many as follow, since bullets turning into numbers down
|
||||
// there start a second nested list, which belongs here too.
|
||||
next, hasNext := p.peek()
|
||||
if hasNext && isListItem(next) && indentOf(next) > indent {
|
||||
p.emit(liOpen + text + "\n")
|
||||
for hasNext && isListItem(next) && indentOf(next) > indent {
|
||||
p.parseList(reOrderedItem.MatchString(next), indentOf(next))
|
||||
next, hasNext = p.peek()
|
||||
}
|
||||
p.emit("</li>\n")
|
||||
} else {
|
||||
p.emit(liOpen + text + "</li>\n")
|
||||
}
|
||||
}
|
||||
|
||||
p.emit("</" + tag + ">\n")
|
||||
}
|
||||
|
||||
// -- Table ---------------------------------------------------------------------
|
||||
|
||||
func isTableRow(s string) bool {
|
||||
t := strings.TrimSpace(s)
|
||||
return len(t) >= 3 && t[0] == '|' && t[len(t)-1] == '|' && !strings.Contains(t, "\n")
|
||||
}
|
||||
|
||||
// splitCells splits a pipe-delimited row into trimmed cell strings.
|
||||
func splitCells(row string) []string {
|
||||
row = strings.TrimPrefix(strings.TrimSpace(row), "|")
|
||||
if strings.HasSuffix(row, "|") && !strings.HasSuffix(row, `\|`) {
|
||||
row = row[:len(row)-1]
|
||||
}
|
||||
var cells []string
|
||||
var cell strings.Builder
|
||||
for i := 0; i < len(row); i++ {
|
||||
switch {
|
||||
case row[i] == '\\' && i+1 < len(row) && row[i+1] == '|':
|
||||
cell.WriteByte('|')
|
||||
i++
|
||||
case row[i] == '|':
|
||||
cells = append(cells, strings.TrimSpace(cell.String()))
|
||||
cell.Reset()
|
||||
default:
|
||||
cell.WriteByte(row[i])
|
||||
}
|
||||
}
|
||||
return append(cells, strings.TrimSpace(cell.String()))
|
||||
}
|
||||
|
||||
func (p *parser) parseTable() {
|
||||
headerLine, _ := p.next()
|
||||
headers := splitCells(headerLine)
|
||||
|
||||
// Consume separator row (|---|---|) if present.
|
||||
if sep, ok := p.peek(); ok && reTableSep.MatchString(strings.TrimSpace(sep)) {
|
||||
p.next()
|
||||
}
|
||||
|
||||
p.emit("<div class=\"table-scroll\">\n<table>\n<thead>\n<tr>\n")
|
||||
for _, h := range headers {
|
||||
p.emit("<th>" + parseInline(h) + "</th>\n")
|
||||
}
|
||||
p.emit("</tr>\n</thead>\n<tbody>\n")
|
||||
|
||||
for {
|
||||
line, ok := p.peek()
|
||||
if !ok || !isTableRow(line) {
|
||||
break
|
||||
}
|
||||
p.next()
|
||||
p.emit("<tr>\n")
|
||||
for _, cell := range splitCells(line) {
|
||||
p.emit("<td>" + parseInline(cell) + "</td>\n")
|
||||
}
|
||||
p.emit("</tr>\n")
|
||||
}
|
||||
|
||||
p.emit("</tbody>\n</table>\n</div>\n")
|
||||
}
|
||||
|
||||
// -- Paragraph -----------------------------------------------------------------
|
||||
|
||||
func (p *parser) parseParagraph() {
|
||||
var lines []string
|
||||
for {
|
||||
line, ok := p.peek()
|
||||
if !ok || strings.TrimSpace(line) == "" {
|
||||
break
|
||||
}
|
||||
if isHeading(line) || isFencedCodeStart(line) || isHR(line) ||
|
||||
(isListItem(line) && startsList(line)) || isTableRow(line) {
|
||||
break
|
||||
}
|
||||
p.next()
|
||||
lines = append(lines, strings.TrimSpace(line))
|
||||
}
|
||||
if len(lines) > 0 {
|
||||
var parts []string
|
||||
for _, l := range lines {
|
||||
parts = append(parts, parseInline(l))
|
||||
}
|
||||
p.emit("<p>" + strings.Join(parts, "<br>\n") + "</p>\n")
|
||||
}
|
||||
}
|
||||
|
||||
// Render converts Markdown src to an HTML string.
|
||||
func Render(src string) string {
|
||||
return newParser(src).parse()
|
||||
}
|
||||
132
outlets.go
Normal file
132
outlets.go
Normal file
|
|
@ -0,0 +1,132 @@
|
|||
// outlets.go - /explore/outlets: publication activity per media outlet and
|
||||
// year.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"sort"
|
||||
)
|
||||
|
||||
type outletCell struct {
|
||||
Docs int
|
||||
Articles int
|
||||
URL string
|
||||
}
|
||||
|
||||
type outletRow struct {
|
||||
Name string
|
||||
URL string
|
||||
Total int
|
||||
Cells []outletCell
|
||||
}
|
||||
|
||||
type outletTable struct {
|
||||
Years []string
|
||||
YearLinks []facetEntry
|
||||
Rows []outletRow
|
||||
Totals []int
|
||||
}
|
||||
|
||||
func (co *Corpus) buildOutletTables() {
|
||||
co.outletTables = map[string]outletTable{
|
||||
"": co.buildOutletTable(""),
|
||||
collectionNews: co.buildOutletTable(collectionNews),
|
||||
}
|
||||
}
|
||||
|
||||
func (co *Corpus) buildOutletTable(coll string) outletTable {
|
||||
type key struct{ outlet, year string }
|
||||
docs := map[key]int{}
|
||||
stories := map[key]int{}
|
||||
totals := map[string]int{}
|
||||
yearSet := map[string]bool{}
|
||||
display := map[string]string{}
|
||||
canon := func(name string) string {
|
||||
k := valueKey(name)
|
||||
if d, ok := display[k]; ok {
|
||||
return d
|
||||
}
|
||||
display[k] = name
|
||||
return name
|
||||
}
|
||||
|
||||
for i := range co.archive.Docs {
|
||||
d := &co.archive.Docs[i]
|
||||
if coll != "" && d.Collection != coll {
|
||||
continue
|
||||
}
|
||||
for _, pub := range facetValues(d, "PU") {
|
||||
totals[canon(pub)]++
|
||||
}
|
||||
// A document counts for a publisher in the years of THAT publisher's
|
||||
// own entries: another outlet's earlier publication does not make it
|
||||
// this outlet's document of that year (publishedBy, the cell's page).
|
||||
counted := map[key]bool{}
|
||||
for _, p := range d.Publication {
|
||||
if p.ReportingOnly || p.Publisher == "" || len(p.DatePublished) < 4 {
|
||||
continue
|
||||
}
|
||||
k := key{canon(p.Publisher), p.DatePublished[:4]}
|
||||
if !counted[k] {
|
||||
counted[k] = true
|
||||
docs[k]++
|
||||
yearSet[k.year] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
for i := range co.articles {
|
||||
a := &co.articles[i]
|
||||
if a.Title == "" || a.Publisher == "" || len(a.Date) < 4 {
|
||||
continue
|
||||
}
|
||||
// A story counts in a collection's table when one of its documents
|
||||
// is in that collection.
|
||||
if coll != "" && !a.inCollection(co, coll) {
|
||||
continue
|
||||
}
|
||||
stories[key{canon(a.Publisher), a.Date[:4]}]++
|
||||
}
|
||||
|
||||
years := make([]string, 0, len(yearSet))
|
||||
for y := range yearSet {
|
||||
years = append(years, y)
|
||||
}
|
||||
sort.Strings(years)
|
||||
|
||||
names := rankCounts(totals, -1)
|
||||
|
||||
scope := wholeArchive
|
||||
if coll != "" {
|
||||
scope = scopeOf(map[string]bool{coll: true})
|
||||
}
|
||||
t := outletTable{Years: years, Totals: make([]int, len(years))}
|
||||
for _, y := range years {
|
||||
t.YearLinks = append(t.YearLinks, facetEntry{Name: y, URL: scope.link(categoryValueURL("published", y))})
|
||||
}
|
||||
for _, name := range names {
|
||||
row := outletRow{Name: name, URL: scope.link(categoryValueURL("publisher", name)), Total: totals[name]}
|
||||
for yi, y := range years {
|
||||
k := key{name, y}
|
||||
cell := outletCell{Docs: docs[k], Articles: stories[k]}
|
||||
if cell.Docs > 0 {
|
||||
cell.URL = scope.link(publisherYearURL(name, y))
|
||||
}
|
||||
t.Totals[yi] += cell.Docs
|
||||
row.Cells = append(row.Cells, cell)
|
||||
}
|
||||
t.Rows = append(t.Rows, row)
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
func (co *Corpus) handleOutlets(w http.ResponseWriter, r *http.Request) {
|
||||
coll := ""
|
||||
if r.URL.Query().Get("coll") == collectionNews {
|
||||
coll = collectionNews
|
||||
}
|
||||
render(w, r, &outletsPage{
|
||||
Table: co.outletTables[coll],
|
||||
NewsOnly: coll == collectionNews,
|
||||
})
|
||||
}
|
||||
393
pages.go
Normal file
393
pages.go
Normal file
|
|
@ -0,0 +1,393 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"html/template"
|
||||
"net/url"
|
||||
)
|
||||
|
||||
// pageData is a page struct, it embeds Base and names its template.
|
||||
type pageData interface {
|
||||
base() *Base
|
||||
templateName() string
|
||||
}
|
||||
|
||||
// Base is what base.html reads. renderCode fills in everything but the
|
||||
// category boxes.
|
||||
type Base struct {
|
||||
Nav string
|
||||
APIEnabled bool
|
||||
Theme string
|
||||
CardPagePreview bool
|
||||
Prefs Prefs
|
||||
hasPrefs bool
|
||||
ShowCatBoxes bool
|
||||
CatCounts map[string]int
|
||||
ActiveCategory string
|
||||
}
|
||||
|
||||
func (b *Base) base() *Base { return b }
|
||||
|
||||
func pageWith(prefs Prefs) Base { return Base{Prefs: prefs, hasPrefs: true} }
|
||||
|
||||
type cardView struct {
|
||||
Docs []*Document
|
||||
ShowThumbs bool
|
||||
CardPagePreview bool
|
||||
HideClassification bool
|
||||
HideJournalists bool
|
||||
HidePeople bool
|
||||
HidePrograms bool
|
||||
HideSummary bool
|
||||
FOIAQS string // see Prefs.foiaLinkQS
|
||||
ShowUnpin bool
|
||||
ActiveCategory string
|
||||
ActiveValue string
|
||||
MarkReported bool // the "reported" badge on individually reported SIDtoday issues
|
||||
}
|
||||
|
||||
// listView is a page of documents as cards or as a table
|
||||
type listView struct {
|
||||
Docs []*Document
|
||||
Total int
|
||||
Page int
|
||||
TotalPages int
|
||||
View string
|
||||
Sort string
|
||||
Order string
|
||||
DefaultSort string
|
||||
BaseURL string
|
||||
SortURL string
|
||||
TableHeads []sortHead
|
||||
RowsHTML template.HTML // the table view's rows, pre-rendered
|
||||
Cards cardView
|
||||
// MarkReported badges the individually reported SIDtoday issues, which
|
||||
// only the SIDtoday tab tells apart from the bulk release.
|
||||
MarkReported bool
|
||||
}
|
||||
|
||||
// docTableRow is a row of the document table
|
||||
type docTableRow struct {
|
||||
*Document
|
||||
MarkReported bool
|
||||
}
|
||||
|
||||
func (lv listView) TableRows() []docTableRow {
|
||||
rows := make([]docTableRow, len(lv.Docs))
|
||||
for i, d := range lv.Docs {
|
||||
rows[i] = docTableRow{d, lv.MarkReported}
|
||||
}
|
||||
return rows
|
||||
}
|
||||
|
||||
type viewer struct {
|
||||
Images []string
|
||||
Sizes []imageSize
|
||||
ImageBase string
|
||||
Content template.HTML
|
||||
Version *DocVersion
|
||||
}
|
||||
|
||||
func (v viewer) ImageSize(i int) template.HTMLAttr {
|
||||
if i < len(v.Sizes) {
|
||||
return v.Sizes[i].Attr()
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
type notFoundPage struct {
|
||||
Base
|
||||
Path string
|
||||
}
|
||||
|
||||
type aboutPage struct{ Base }
|
||||
|
||||
type indexPage struct {
|
||||
Base
|
||||
homeFigures
|
||||
TrackRecent bool
|
||||
RecentDocs []*Document
|
||||
SnowdenTotal int
|
||||
ArchiveTotal int
|
||||
FOIATotal int
|
||||
IncludeFOIA bool
|
||||
ReportedTotal int
|
||||
WithSIDtoday bool // the figures include the SIDtoday archive (/?sidtoday=1)
|
||||
}
|
||||
|
||||
// browsePage is /browse
|
||||
type browsePage struct {
|
||||
Base
|
||||
listView
|
||||
Collection string
|
||||
CollectionCounts map[string]int
|
||||
SIDTodayCount int
|
||||
SelectedTypes map[string]bool
|
||||
AllTypes map[string]int
|
||||
Params url.Values
|
||||
SeriesList []categoryEntry
|
||||
ArchiveYears []archiveYear
|
||||
TimelineHTML template.HTML
|
||||
}
|
||||
|
||||
// categoryPage is /browse/{category}
|
||||
type categoryPage struct {
|
||||
Base
|
||||
Title string
|
||||
Blurb string
|
||||
Heads []sortHead
|
||||
RowsHTML template.HTML
|
||||
Entries []categoryEntry
|
||||
CollCols bool
|
||||
ShowCollCols bool
|
||||
ShowAZ bool
|
||||
Letters []string
|
||||
SeeAlso []seeAlsoLink
|
||||
}
|
||||
|
||||
// listPage is /browse/{category}/{value}
|
||||
type listPage struct {
|
||||
Base
|
||||
listView
|
||||
Title string
|
||||
CatLabel string
|
||||
Blurb string
|
||||
SearchLink string
|
||||
SelectedTypes map[string]bool
|
||||
AllTypes map[string]int
|
||||
Params url.Values
|
||||
GlossaryEntry *GlossaryEntry
|
||||
Refinements []searchFacet
|
||||
Breakdowns []bdGroup
|
||||
Timeline []timelineBar
|
||||
PubTimeline []timelineBar
|
||||
YearScope string
|
||||
YearScopeAllURL string
|
||||
MonthNav []monthNavEntry
|
||||
MonthNavYear string
|
||||
MonthNavYearURL string
|
||||
MonthNavYearActive bool
|
||||
MonthlessCount int
|
||||
}
|
||||
|
||||
type docPage struct {
|
||||
Base
|
||||
viewer
|
||||
Doc *Document
|
||||
BrowseQS string
|
||||
OGImageURL string
|
||||
DocAbsURL string
|
||||
Embeds docEmbeds
|
||||
SeriesGroup *seriesGroup
|
||||
Assemblies []*Assembly
|
||||
SerialPrev *Document
|
||||
SerialNext *Document
|
||||
PublishedTogether []*Document
|
||||
PublishedShown int
|
||||
PublishedMore []*Article
|
||||
PublishedCards cardView
|
||||
RelatedDocs []*Document
|
||||
RelatedCards cardView
|
||||
Pinned bool
|
||||
}
|
||||
|
||||
type versionPage struct {
|
||||
Base
|
||||
viewer
|
||||
Doc *Document
|
||||
Publications []Publication
|
||||
PDFURL string
|
||||
}
|
||||
|
||||
// pagesPage is the page-image grid of a document or of one of its versions.
|
||||
type pagesPage struct {
|
||||
Base
|
||||
viewer
|
||||
Doc *Document
|
||||
TextOnly bool
|
||||
}
|
||||
|
||||
type articlesPage struct {
|
||||
Base
|
||||
Groups []articleYearGroup
|
||||
GroupsHTML template.HTML
|
||||
Total int
|
||||
Books int
|
||||
}
|
||||
|
||||
type articlePage struct {
|
||||
Base
|
||||
Article Article
|
||||
Total int
|
||||
BrowseQS string
|
||||
PublisherURL string
|
||||
Cards cardView
|
||||
}
|
||||
|
||||
type assembliesPage struct {
|
||||
Base
|
||||
Assemblies []*Assembly
|
||||
TextOnly bool
|
||||
}
|
||||
|
||||
type assemblyPage struct {
|
||||
Base
|
||||
Assembly *Assembly
|
||||
TextOnly bool
|
||||
}
|
||||
|
||||
type changelogPage struct {
|
||||
Base
|
||||
Entries []ChangeEntry
|
||||
Total int
|
||||
}
|
||||
|
||||
type codewordsPage struct {
|
||||
Base
|
||||
Rows []codewordRow
|
||||
RowsHTML template.HTML
|
||||
Total int
|
||||
Shown int
|
||||
Heads []sortHead
|
||||
}
|
||||
|
||||
type explorePage struct {
|
||||
Base
|
||||
Metrics []exploreMetric
|
||||
HasAssemblies bool
|
||||
}
|
||||
|
||||
type exploreTablePage struct {
|
||||
Base
|
||||
Metric *exploreMetric
|
||||
Rows []exploreRow
|
||||
RowsHTML template.HTML
|
||||
Total int
|
||||
Shown int
|
||||
Heads []sortHead
|
||||
}
|
||||
|
||||
type classPatternsPage struct {
|
||||
Base
|
||||
Cols []facetEntry
|
||||
Levels []string
|
||||
Strips []clsYearRow
|
||||
Sparse []clsYearRow
|
||||
MinMarks int
|
||||
HeatRows []heatRow
|
||||
}
|
||||
|
||||
type idsPage struct {
|
||||
Base
|
||||
Rows []idRow
|
||||
RowsHTML template.HTML
|
||||
Shown int
|
||||
}
|
||||
|
||||
type glossaryPage struct {
|
||||
Base
|
||||
Groups []glossaryGroup
|
||||
GroupsHTML template.HTML
|
||||
Total int
|
||||
}
|
||||
|
||||
type outletsPage struct {
|
||||
Base
|
||||
Table outletTable
|
||||
NewsOnly bool
|
||||
}
|
||||
|
||||
type portionsPage struct {
|
||||
Base
|
||||
Title string
|
||||
Stats portionSummary
|
||||
RowLimit int
|
||||
}
|
||||
|
||||
type pinnedPage struct {
|
||||
Base
|
||||
Total int
|
||||
Searches []recentSearch
|
||||
Full bool // a pin was refused: the cookie is full
|
||||
Cards cardView
|
||||
}
|
||||
|
||||
type preferencesPage struct {
|
||||
Base
|
||||
MaxCards int
|
||||
Saved bool
|
||||
Reset bool
|
||||
}
|
||||
|
||||
type searchPage struct {
|
||||
Base
|
||||
FieldOptions []searchField
|
||||
Facets []searchFacet
|
||||
Queries []string
|
||||
Fields []string
|
||||
Conjs []string
|
||||
Results []searchResult
|
||||
SearchTypeCounts map[string]int
|
||||
SelTypeSet map[string]bool
|
||||
FormParams url.Values
|
||||
Savable bool
|
||||
ShowThumbs bool
|
||||
FOIAQS string
|
||||
Total int
|
||||
Page int
|
||||
TotalPages int
|
||||
RecentSearches []recentSearch
|
||||
SelColl map[string]bool
|
||||
CollFiltered bool
|
||||
CanonicalQS string
|
||||
SearchPinned bool
|
||||
Searched bool
|
||||
QueryLabel string
|
||||
DateDocStart string
|
||||
DateDocEnd string
|
||||
DatePubStart string
|
||||
DatePubEnd string
|
||||
Sort string
|
||||
Order string
|
||||
SearchBaseURL string
|
||||
DateHist []timelineBar
|
||||
SnippetsCapped bool
|
||||
SnippetLimit int
|
||||
OverClauses int
|
||||
ClauseLimit int
|
||||
OverWords int
|
||||
WordLimit int
|
||||
}
|
||||
|
||||
func (*notFoundPage) templateName() string { return "404.html" }
|
||||
func (*aboutPage) templateName() string { return "about.html" }
|
||||
func (*indexPage) templateName() string { return "index.html" }
|
||||
func (*browsePage) templateName() string { return "browse.html" }
|
||||
func (*categoryPage) templateName() string { return "category.html" }
|
||||
func (*listPage) templateName() string { return "list.html" }
|
||||
func (*docPage) templateName() string { return "doc.html" }
|
||||
func (*versionPage) templateName() string { return "version.html" }
|
||||
func (*pagesPage) templateName() string { return "pages.html" }
|
||||
func (*articlesPage) templateName() string { return "articles.html" }
|
||||
func (*articlePage) templateName() string { return "article.html" }
|
||||
func (*assembliesPage) templateName() string { return "assemblies.html" }
|
||||
func (*assemblyPage) templateName() string { return "assembly.html" }
|
||||
func (*changelogPage) templateName() string { return "changelog.html" }
|
||||
func (*codewordsPage) templateName() string { return "codewords.html" }
|
||||
func (*explorePage) templateName() string { return "explore.html" }
|
||||
func (*exploreTablePage) templateName() string { return "explore_table.html" }
|
||||
func (*classPatternsPage) templateName() string { return "classpatterns.html" }
|
||||
func (*idsPage) templateName() string { return "ids.html" }
|
||||
func (*glossaryPage) templateName() string { return "glossary.html" }
|
||||
func (*outletsPage) templateName() string { return "outlets.html" }
|
||||
func (*portionsPage) templateName() string { return "portions.html" }
|
||||
func (*pinnedPage) templateName() string { return "pinned.html" }
|
||||
func (*preferencesPage) templateName() string { return "preferences.html" }
|
||||
func (*searchPage) templateName() string { return "search.html" }
|
||||
|
||||
var pageTypes = []pageData{
|
||||
¬FoundPage{}, &aboutPage{}, &indexPage{}, &browsePage{}, &categoryPage{}, &listPage{},
|
||||
&docPage{}, &versionPage{}, &pagesPage{}, &articlesPage{}, &articlePage{}, &assembliesPage{},
|
||||
&assemblyPage{}, &changelogPage{}, &codewordsPage{}, &explorePage{}, &exploreTablePage{},
|
||||
&classPatternsPage{}, &idsPage{}, &glossaryPage{}, &outletsPage{}, &portionsPage{},
|
||||
&pinnedPage{}, &preferencesPage{}, &searchPage{},
|
||||
}
|
||||
145
pins.go
Normal file
145
pins.go
Normal file
|
|
@ -0,0 +1,145 @@
|
|||
// pins.go - pinned documents (a reading list). Cookie-local.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/url"
|
||||
"slices"
|
||||
)
|
||||
|
||||
const (
|
||||
pinnedCookie = "pinned_docs"
|
||||
pinnedMax = 200
|
||||
pinnedSearchesCookie = "pinned_searches"
|
||||
pinnedSearchesMax = 30
|
||||
)
|
||||
|
||||
func (co *Corpus) readPinned(r *http.Request) []string {
|
||||
var out []string
|
||||
for _, id := range readIDListCookie(r, pinnedCookie, pinnedMax) {
|
||||
if co.docIndex[id] != nil {
|
||||
out = append(out, id)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func toggleEntry(list []string, e string, max int) []string {
|
||||
var next []string
|
||||
removed := false
|
||||
for _, x := range list {
|
||||
if x == e {
|
||||
removed = true
|
||||
continue
|
||||
}
|
||||
next = append(next, x)
|
||||
}
|
||||
if !removed && len(next) < max {
|
||||
next = append(next, e)
|
||||
}
|
||||
return next
|
||||
}
|
||||
|
||||
func isPinned(r *http.Request, id string) bool {
|
||||
return slices.Contains(readIDListCookie(r, pinnedCookie, pinnedMax), id)
|
||||
}
|
||||
|
||||
func (co *Corpus) handlePinToggle(w http.ResponseWriter, r *http.Request) {
|
||||
if !readPOSTForm(w, r) {
|
||||
return
|
||||
}
|
||||
id := r.PathValue("id")
|
||||
if co.docIndex[id] == nil {
|
||||
notFound(w, r)
|
||||
return
|
||||
}
|
||||
|
||||
pinned := co.readPinned(r)
|
||||
stored := writeListCookie(w, pinnedCookie, toggleEntry(pinned, id, pinnedMax))
|
||||
if pinRefused(pinned, stored, id) {
|
||||
http.Redirect(w, r, "/pinned?full=1", http.StatusSeeOther)
|
||||
return
|
||||
}
|
||||
|
||||
back := safeBackPath(r.FormValue("back"))
|
||||
if back == "/" {
|
||||
back = docPath(id)
|
||||
}
|
||||
http.Redirect(w, r, back, http.StatusSeeOther)
|
||||
}
|
||||
|
||||
func readPinnedSearchEntries(r *http.Request) []string {
|
||||
return readIDListCookie(r, pinnedSearchesCookie, pinnedSearchesMax)
|
||||
}
|
||||
|
||||
func isPinnedSearch(r *http.Request, canonicalQS string) bool {
|
||||
return slices.Contains(readPinnedSearchEntries(r), url.QueryEscape(canonicalQS))
|
||||
}
|
||||
|
||||
func (co *Corpus) handleSearchPinToggle(w http.ResponseWriter, r *http.Request) {
|
||||
if !readPOSTForm(w, r) {
|
||||
return
|
||||
}
|
||||
qs := r.FormValue("qs")
|
||||
if qs == "" {
|
||||
http.Error(w, "bad query", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
v, err := url.ParseQuery(qs)
|
||||
if err != nil {
|
||||
http.Error(w, "bad query", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
entry := url.QueryEscape(qs)
|
||||
pinned := readPinnedSearchEntries(r)
|
||||
if !co.savableSearch(v) && !slices.Contains(pinned, entry) {
|
||||
http.Error(w, "this search cannot be pinned: it has no terms or filters, or is too long", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
stored := writeListCookie(w, pinnedSearchesCookie, toggleEntry(pinned, entry, pinnedSearchesMax))
|
||||
if pinRefused(pinned, stored, entry) {
|
||||
http.Redirect(w, r, "/pinned?full=1", http.StatusSeeOther)
|
||||
return
|
||||
}
|
||||
http.Redirect(w, r, safeBackPath(r.FormValue("back")), http.StatusSeeOther)
|
||||
}
|
||||
|
||||
func pinRefused(before, stored []string, e string) bool {
|
||||
return !slices.Contains(before, e) && !slices.Contains(stored, e)
|
||||
}
|
||||
|
||||
func (co *Corpus) handlePinnedPage(w http.ResponseWriter, r *http.Request) {
|
||||
prefs := readPrefs(r)
|
||||
var docs []*Document
|
||||
for _, id := range co.readPinned(r) {
|
||||
docs = append(docs, co.docIndex[id])
|
||||
}
|
||||
searches := co.parseSearchEntries(readPinnedSearchEntries(r))
|
||||
|
||||
data := &pinnedPage{
|
||||
Base: pageWith(prefs),
|
||||
Total: len(docs),
|
||||
Searches: searches,
|
||||
Full: r.URL.Query().Get("full") != "",
|
||||
Cards: newCardView(prefs, docs),
|
||||
}
|
||||
data.Cards.ShowUnpin = true
|
||||
noStore(w)
|
||||
render(w, r, data)
|
||||
}
|
||||
|
||||
func (co *Corpus) handlePinnedExport(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Cache-Control", "private, no-store")
|
||||
w.Header().Set("Content-Disposition", `attachment; filename="snowchive-pinned.json"`)
|
||||
ids := co.readPinned(r)
|
||||
out := make([]apiDocSummary, 0, len(ids))
|
||||
for _, id := range ids {
|
||||
out = append(out, docSummary(*co.docIndex[id]))
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]interface{}{
|
||||
"count": len(out),
|
||||
"docs": out,
|
||||
})
|
||||
}
|
||||
230
portions.go
Normal file
230
portions.go
Normal file
|
|
@ -0,0 +1,230 @@
|
|||
// portions.go - the portion markings inside the transcripts.
|
||||
package main
|
||||
|
||||
import (
|
||||
"html/template"
|
||||
"log"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// A marking opens a line, optionally behind a bullet, a heading hash or a
|
||||
// list number: "(U)", "* (S//SI) ...", "## (U//FOUO) Heading".
|
||||
var rePortion = regexp.MustCompile(`^[ \t]{0,6}(?:[-*+][ \t]+|#{1,6}[ \t]+|\d+[.)][ \t]+)?\(([^)\n]{1,42})\)`)
|
||||
|
||||
var portionLevels = map[string]string{
|
||||
"U": "U", "UNCLASSIFIED": "U", "FOUO": "U",
|
||||
"C": "C", "CONFIDENTIAL": "C",
|
||||
"S": "S", "SECRET": "S",
|
||||
"TS": "TS", "TOP SECRET": "TS",
|
||||
}
|
||||
|
||||
var portionControls = map[string]bool{
|
||||
"SI": true, "TK": true, "GAMMA": true, "SI-GAMMA": true, "STLW": true, "TSP": true,
|
||||
"HCS": true, "KLONDIKE": true, "ECI": true, "SI-ECI": true, "COMINT": true,
|
||||
"FOUO": true, "NF": true, "NOFORN": true, "OC": true, "ORCON": true,
|
||||
"PROPIN": true, "PR": true, "RELIDO": true, "RSEN": true, "FISA": true,
|
||||
"IMCON": true, "DSEN": true, "REL": true, "FVEY": true, "NATO": true,
|
||||
"X1": true,
|
||||
}
|
||||
|
||||
var portionLevelNames = []string{"U", "C", "S", "TS"}
|
||||
|
||||
var portionLevelLabels = map[string]string{
|
||||
"U": "UNCLASSIFIED", "C": "CONFIDENTIAL", "S": "SECRET", "TS": "TOP SECRET",
|
||||
}
|
||||
|
||||
// reSlashRun folds "/", "//" and "///" to one separator.
|
||||
var reSlashRun = regexp.MustCompile(`[ \t]*/+[ \t]*`)
|
||||
|
||||
// portionMark is one marking, canonicalised.
|
||||
type portionMark struct {
|
||||
Raw string // as the transcript writes it, without the parentheses
|
||||
Canonical string // "TS//SI//NF"
|
||||
Level string // "TS"
|
||||
Controls []string // ["SI", "NF"], in the order the marking writes them
|
||||
OK bool // every part recognised
|
||||
}
|
||||
|
||||
func parsePortion(raw string) portionMark {
|
||||
m := portionMark{Raw: strings.TrimSpace(raw)}
|
||||
written := strings.Split(reSlashRun.ReplaceAllString(strings.Join(strings.Fields(m.Raw), " "), "//"), "//")
|
||||
if lvl := strings.TrimSpace(written[0]); lvl == "" || lvl != strings.ToUpper(lvl) {
|
||||
return m
|
||||
}
|
||||
parts := make([]string, len(written))
|
||||
for i := range written {
|
||||
parts[i] = strings.ToUpper(strings.TrimSpace(written[i]))
|
||||
}
|
||||
lvl, isLevel := portionLevels[parts[0]]
|
||||
if !isLevel {
|
||||
return m
|
||||
}
|
||||
m.Level = lvl
|
||||
if parts[0] == "FOUO" {
|
||||
m.Controls = append(m.Controls, "FOUO")
|
||||
}
|
||||
known := true
|
||||
for _, p := range parts[1:] {
|
||||
if p == "" {
|
||||
continue
|
||||
}
|
||||
m.Controls = append(m.Controls, p)
|
||||
if !portionControlKnown(p) {
|
||||
known = false
|
||||
}
|
||||
}
|
||||
m.OK = known
|
||||
m.Canonical = m.Level
|
||||
if len(m.Controls) > 0 {
|
||||
m.Canonical += "//" + strings.Join(m.Controls, "//")
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func portionControlKnown(p string) bool {
|
||||
if portionControls[p] {
|
||||
return true
|
||||
}
|
||||
if strings.HasPrefix(p, "REL") {
|
||||
return true
|
||||
}
|
||||
if strings.HasPrefix(p, "ECI ") || strings.HasPrefix(p, "SI-ECI") {
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func extractPortions(text string) []portionMark {
|
||||
var out []portionMark
|
||||
fenced := false
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
t := strings.TrimSpace(line)
|
||||
if strings.HasPrefix(t, "```") || strings.HasPrefix(t, "~~~") {
|
||||
fenced = !fenced
|
||||
continue
|
||||
}
|
||||
if fenced || strings.HasPrefix(t, ">") {
|
||||
continue
|
||||
}
|
||||
m := rePortion.FindStringSubmatch(line)
|
||||
if m == nil {
|
||||
continue
|
||||
}
|
||||
if p := parsePortion(m[1]); p.Level != "" {
|
||||
out = append(out, p)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// --- the corpus view -------------------------------------------------------
|
||||
|
||||
type portionRow struct {
|
||||
Name string // the canonical marking, a level, or a control
|
||||
Label string // the spelled-out level, where there is one
|
||||
Marks int // occurrences across the corpus
|
||||
Docs int // documents carrying it
|
||||
SearchURL template.URL
|
||||
}
|
||||
|
||||
type portionSummary struct {
|
||||
Docs int // documents carrying at least one marking
|
||||
Marks int // markings found
|
||||
Distinct int // distinct canonical markings
|
||||
Unrecog int // markings whose parts are not all known
|
||||
UnrecogEg []portionRow
|
||||
Levels []portionRow
|
||||
Markings []portionRow
|
||||
Controls []portionRow
|
||||
}
|
||||
|
||||
const portionRows = 200
|
||||
|
||||
func (co *Corpus) buildPortions() {
|
||||
co.portionStats = portionSummary{}
|
||||
markMarks, lvlMarks, ctlMarks := map[string]int{}, map[string]int{}, map[string]int{}
|
||||
docsBy := map[string]map[string]int{"PL": {}, "PM": {}, "PC": {}}
|
||||
unrecogMarks := map[string]int{}
|
||||
|
||||
for i := range co.archive.Docs {
|
||||
d := &co.archive.Docs[i]
|
||||
d.PortionLevels, d.PortionMarkings, d.PortionControls = nil, nil, nil
|
||||
marks := extractPortions(d.FullText)
|
||||
if len(marks) == 0 {
|
||||
continue
|
||||
}
|
||||
co.portionStats.Docs++
|
||||
co.portionStats.Marks += len(marks)
|
||||
for _, m := range marks {
|
||||
if !m.OK {
|
||||
co.portionStats.Unrecog++
|
||||
unrecogMarks[m.Raw]++
|
||||
continue
|
||||
}
|
||||
markMarks[m.Canonical]++
|
||||
lvlMarks[m.Level]++
|
||||
d.PortionMarkings = append(d.PortionMarkings, m.Canonical)
|
||||
d.PortionLevels = append(d.PortionLevels, m.Level)
|
||||
for _, c := range m.Controls {
|
||||
ctlMarks[c]++
|
||||
d.PortionControls = append(d.PortionControls, c)
|
||||
}
|
||||
}
|
||||
d.PortionLevels = normalizeList(d.PortionLevels)
|
||||
d.PortionMarkings = normalizeList(d.PortionMarkings)
|
||||
d.PortionControls = normalizeList(d.PortionControls)
|
||||
for code, docs := range docsBy {
|
||||
for _, v := range facetValues(d, code) {
|
||||
docs[v]++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
co.portionStats.Distinct = len(markMarks)
|
||||
for _, l := range portionLevelNames {
|
||||
if lvlMarks[l] == 0 {
|
||||
continue
|
||||
}
|
||||
co.portionStats.Levels = append(co.portionStats.Levels, portionRow{
|
||||
Name: l, Label: portionLevelLabels[l], Marks: lvlMarks[l], Docs: docsBy["PL"][l],
|
||||
SearchURL: portionSearchURL("PL", l),
|
||||
})
|
||||
}
|
||||
co.portionStats.Markings = portionRowsOf(markMarks, docsBy["PM"], portionRows)
|
||||
for i := range co.portionStats.Markings {
|
||||
r := &co.portionStats.Markings[i]
|
||||
r.SearchURL = portionSearchURL("PM", r.Name)
|
||||
}
|
||||
co.portionStats.Controls = portionRowsOf(ctlMarks, docsBy["PC"], portionRows)
|
||||
for i := range co.portionStats.Controls {
|
||||
r := &co.portionStats.Controls[i]
|
||||
r.SearchURL = portionSearchURL("PC", r.Name)
|
||||
}
|
||||
co.portionStats.UnrecogEg = portionRowsOf(unrecogMarks, nil, 25)
|
||||
|
||||
log.Printf("Portion markings: %d in %d documents, %d distinct (%d unrecognised)",
|
||||
co.portionStats.Marks, co.portionStats.Docs, co.portionStats.Distinct, co.portionStats.Unrecog)
|
||||
}
|
||||
|
||||
func portionSearchURL(code, value string) template.URL {
|
||||
return template.URL(wholeArchive.link(searchURL([]searchClause{{Field: code, Value: exactClause(value)}})))
|
||||
}
|
||||
|
||||
func portionRowsOf(marks, docs map[string]int, n int) []portionRow {
|
||||
names := rankCounts(marks, n)
|
||||
out := make([]portionRow, len(names))
|
||||
for i, k := range names {
|
||||
out[i] = portionRow{Name: k, Marks: marks[k], Docs: docs[k]}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (co *Corpus) handlePortions(w http.ResponseWriter, r *http.Request) {
|
||||
render(w, r, &portionsPage{
|
||||
Title: "Portion markings",
|
||||
Stats: co.portionStats,
|
||||
RowLimit: portionRows,
|
||||
})
|
||||
}
|
||||
262
prefs.go
Normal file
262
prefs.go
Normal file
|
|
@ -0,0 +1,262 @@
|
|||
// prefs.go - per-browser UI preferences stored in cookies, plus shared
|
||||
// cookie helpers used by the pins and history features.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// setCookie / delCookie are the single place cookie attributes are decided.
|
||||
|
||||
func setCookie(w http.ResponseWriter, name, val string) {
|
||||
http.SetCookie(w, &http.Cookie{
|
||||
Name: name,
|
||||
Value: val,
|
||||
Path: "/",
|
||||
MaxAge: 365 * 24 * 3600,
|
||||
SameSite: http.SameSiteLaxMode,
|
||||
HttpOnly: true,
|
||||
})
|
||||
}
|
||||
|
||||
func delCookie(w http.ResponseWriter, name string) {
|
||||
http.SetCookie(w, &http.Cookie{
|
||||
Name: name,
|
||||
Value: "",
|
||||
Path: "/",
|
||||
MaxAge: -1,
|
||||
SameSite: http.SameSiteLaxMode,
|
||||
HttpOnly: true,
|
||||
})
|
||||
}
|
||||
|
||||
const cookieValueMax = 3800
|
||||
|
||||
func writeListCookie(w http.ResponseWriter, name string, entries []string) []string {
|
||||
n, size := 0, 0
|
||||
for _, e := range entries {
|
||||
add := len(e)
|
||||
if n > 0 {
|
||||
add++ // the "|" separator
|
||||
}
|
||||
if size+add > cookieValueMax {
|
||||
break
|
||||
}
|
||||
size += add
|
||||
n++
|
||||
}
|
||||
entries = entries[:n]
|
||||
if len(entries) == 0 {
|
||||
delCookie(w, name)
|
||||
return entries
|
||||
}
|
||||
setCookie(w, name, strings.Join(entries, "|"))
|
||||
return entries
|
||||
}
|
||||
|
||||
type Prefs struct {
|
||||
DocView string // "images" (default), "split", "text"
|
||||
Theme string // "auto" (default), "light", "dark"
|
||||
PerPage int // docs per page (default 20); 0 means "all", which cardsPerPage bounds
|
||||
|
||||
ShowClassification bool // classification/dissemination marking on document cards (default true)
|
||||
ShowJournalists bool // journalist names on document cards (default true)
|
||||
ShowPeople bool // people line on document cards (default true)
|
||||
ShowPrograms bool // program names on document cards (default true)
|
||||
ShowSummary bool // short summary (description) on document cards (default true)
|
||||
ShowThumbs bool // thumbnails on search results & cards (default true)
|
||||
ShowCollCols bool // category tables: per-collection count columns always visible (default false, toggle on the page)
|
||||
TextOnly bool // suppress thumbnails and page images (default false)
|
||||
CardPagePreview bool // desktop: full page preview on card-thumb hover (default false, opt-in)
|
||||
ScrollDoc bool // doc view: cap pane height and scroll (default false)
|
||||
InfoSidebar bool // doc view: metadata beside the document instead of above (default false); cookie is "above"/"beside", not a 0/1 flag
|
||||
ProgramsAsList bool // doc view: programs one-per-line (default false, comma-separated)
|
||||
TagsAsList bool // doc view: tags one-per-line (default false, comma-separated)
|
||||
OpenInfo bool // doc view: "More info" expanded by default (default false)
|
||||
CollapsePubTogether bool // doc view: collapse "published in the same report" (default false)
|
||||
CollapseRelated bool // doc view: collapse related documents (default false)
|
||||
TrackRecent bool // record recently viewed documents (default false)
|
||||
TrackSearches bool // record recent searches (default false)
|
||||
IncludeFOIA bool // FOIA releases in search, category totals, /random and related documents when the URL names no collection (default false, opt-in)
|
||||
}
|
||||
|
||||
func (p Prefs) thumbsOn() bool { return p.ShowThumbs && !p.TextOnly }
|
||||
|
||||
const maxCardsPerPage = 500
|
||||
|
||||
func (p Prefs) cardsPerPage() int {
|
||||
if p.PerPage <= 0 || p.PerPage > maxCardsPerPage {
|
||||
return maxCardsPerPage
|
||||
}
|
||||
return p.PerPage
|
||||
}
|
||||
|
||||
func newCardView(p Prefs, docs []*Document) cardView {
|
||||
return cardView{
|
||||
Docs: docs,
|
||||
ShowThumbs: p.thumbsOn(),
|
||||
CardPagePreview: p.CardPagePreview,
|
||||
HideClassification: !p.ShowClassification,
|
||||
HideJournalists: !p.ShowJournalists,
|
||||
HidePeople: !p.ShowPeople,
|
||||
HidePrograms: !p.ShowPrograms,
|
||||
HideSummary: !p.ShowSummary,
|
||||
FOIAQS: p.foiaLinkQS(),
|
||||
}
|
||||
}
|
||||
|
||||
func (p Prefs) foiaLinkQS() string {
|
||||
return pageScope(nil, p, false).including(collectionFOIA).suffix()
|
||||
}
|
||||
|
||||
var boolPrefs = []struct {
|
||||
cookie string
|
||||
form string
|
||||
invert bool
|
||||
field func(*Prefs) *bool
|
||||
}{
|
||||
{"pref_hide_classification", "hide_classification", true, func(p *Prefs) *bool { return &p.ShowClassification }},
|
||||
{"pref_hide_journalists", "hide_journalists", true, func(p *Prefs) *bool { return &p.ShowJournalists }},
|
||||
{"pref_hide_people", "hide_people", true, func(p *Prefs) *bool { return &p.ShowPeople }},
|
||||
{"pref_hide_programs", "hide_programs", true, func(p *Prefs) *bool { return &p.ShowPrograms }},
|
||||
{"pref_hide_summary", "hide_summary", true, func(p *Prefs) *bool { return &p.ShowSummary }},
|
||||
{"pref_hide_thumbs", "hide_thumbs", true, func(p *Prefs) *bool { return &p.ShowThumbs }},
|
||||
{"pref_show_coll_cols", "show_coll_cols", false, func(p *Prefs) *bool { return &p.ShowCollCols }},
|
||||
{"pref_text_only", "text_only", false, func(p *Prefs) *bool { return &p.TextOnly }},
|
||||
{"pref_page_preview", "page_preview", false, func(p *Prefs) *bool { return &p.CardPagePreview }},
|
||||
{"pref_scroll_doc", "scroll_doc", false, func(p *Prefs) *bool { return &p.ScrollDoc }},
|
||||
{"pref_programs_list", "programs_list", false, func(p *Prefs) *bool { return &p.ProgramsAsList }},
|
||||
{"pref_tags_list", "tags_list", false, func(p *Prefs) *bool { return &p.TagsAsList }},
|
||||
{"pref_open_info", "open_info", false, func(p *Prefs) *bool { return &p.OpenInfo }},
|
||||
{"pref_collapse_pub", "collapse_pub", false, func(p *Prefs) *bool { return &p.CollapsePubTogether }},
|
||||
{"pref_collapse_related", "collapse_related", false, func(p *Prefs) *bool { return &p.CollapseRelated }},
|
||||
{"pref_track_recent", "track_recent", false, func(p *Prefs) *bool { return &p.TrackRecent }},
|
||||
{"pref_track_searches", "track_searches", false, func(p *Prefs) *bool { return &p.TrackSearches }},
|
||||
{"pref_include_foia", "include_foia", false, func(p *Prefs) *bool { return &p.IncludeFOIA }},
|
||||
}
|
||||
|
||||
var valuePrefs = []struct {
|
||||
cookie string
|
||||
form string
|
||||
allowed []string
|
||||
def string
|
||||
apply func(*Prefs, string)
|
||||
}{
|
||||
{"pref_doc_view", "doc_view", []string{"images", "split", "text"}, "images",
|
||||
func(p *Prefs, v string) { p.DocView = v }},
|
||||
{"pref_theme", "theme", []string{"auto", "light", "dark"}, "auto",
|
||||
func(p *Prefs, v string) { p.Theme = v }},
|
||||
{"pref_per_page", "per_page", []string{"10", "20", "50", "100", "all"}, "20",
|
||||
func(p *Prefs, v string) {
|
||||
if v == "all" {
|
||||
p.PerPage = 0
|
||||
return
|
||||
}
|
||||
if n, err := strconv.Atoi(v); err == nil {
|
||||
p.PerPage = n
|
||||
}
|
||||
}},
|
||||
{"pref_info_position", "info_position", []string{"above", "beside"}, "above",
|
||||
func(p *Prefs, v string) { p.InfoSidebar = v == "beside" }},
|
||||
}
|
||||
|
||||
var prefCookies = func() []string {
|
||||
var out []string
|
||||
for _, vp := range valuePrefs {
|
||||
out = append(out, vp.cookie)
|
||||
}
|
||||
for _, bp := range boolPrefs {
|
||||
out = append(out, bp.cookie)
|
||||
}
|
||||
return append(out, recentDocsCookie, recentSearchesCookie)
|
||||
}()
|
||||
|
||||
func (p Prefs) ThemeAttr() string {
|
||||
if p.Theme == "auto" {
|
||||
return ""
|
||||
}
|
||||
return p.Theme
|
||||
}
|
||||
|
||||
func readPrefs(r *http.Request) Prefs {
|
||||
var p Prefs
|
||||
cookie := map[string]string{}
|
||||
for _, c := range r.Cookies() {
|
||||
cookie[c.Name] = c.Value
|
||||
}
|
||||
|
||||
for i := range boolPrefs {
|
||||
bp := &boolPrefs[i]
|
||||
v := cookie[bp.cookie] == "1"
|
||||
if bp.invert {
|
||||
v = !v
|
||||
}
|
||||
*bp.field(&p) = v
|
||||
}
|
||||
|
||||
for i := range valuePrefs {
|
||||
vp := &valuePrefs[i]
|
||||
v := cookie[vp.cookie]
|
||||
if !slices.Contains(vp.allowed, v) {
|
||||
v = vp.def
|
||||
}
|
||||
vp.apply(&p, v)
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
func handlePreferences(w http.ResponseWriter, r *http.Request) {
|
||||
noStore(w)
|
||||
q := r.URL.Query()
|
||||
render(w, r, &preferencesPage{
|
||||
Base: pageWith(readPrefs(r)),
|
||||
MaxCards: maxCardsPerPage,
|
||||
Saved: q.Get("saved") == "1",
|
||||
Reset: q.Get("reset") == "1",
|
||||
})
|
||||
}
|
||||
|
||||
func handlePreferencesSave(w http.ResponseWriter, r *http.Request) {
|
||||
if !readPOSTForm(w, r) {
|
||||
return
|
||||
}
|
||||
|
||||
if r.FormValue("reset") == "1" {
|
||||
for _, name := range prefCookies {
|
||||
delCookie(w, name)
|
||||
}
|
||||
http.Redirect(w, r, "/preferences?reset=1", http.StatusSeeOther)
|
||||
return
|
||||
}
|
||||
|
||||
for _, vp := range valuePrefs {
|
||||
v := r.FormValue(vp.form)
|
||||
switch {
|
||||
case v == vp.def:
|
||||
delCookie(w, vp.cookie)
|
||||
case slices.Contains(vp.allowed, v):
|
||||
setCookie(w, vp.cookie, v)
|
||||
}
|
||||
}
|
||||
|
||||
for _, bp := range boolPrefs {
|
||||
if r.FormValue(bp.form) == "1" {
|
||||
setCookie(w, bp.cookie, "1")
|
||||
} else {
|
||||
delCookie(w, bp.cookie)
|
||||
}
|
||||
}
|
||||
|
||||
if r.FormValue("track_recent") != "1" {
|
||||
delCookie(w, recentDocsCookie)
|
||||
}
|
||||
if r.FormValue("track_searches") != "1" {
|
||||
delCookie(w, recentSearchesCookie)
|
||||
}
|
||||
|
||||
http.Redirect(w, r, "/preferences?saved=1", http.StatusSeeOther)
|
||||
}
|
||||
233
seo.go
Normal file
233
seo.go
Normal file
|
|
@ -0,0 +1,233 @@
|
|||
// seo.go - robots.txt, sitemap.xml, and the per-document metadata embeds.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
"html/template"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
)
|
||||
|
||||
var canonicalBase string
|
||||
|
||||
func requestBase(r *http.Request) string {
|
||||
if canonicalBase != "" {
|
||||
return canonicalBase
|
||||
}
|
||||
scheme := "http"
|
||||
if r.TLS != nil || r.Header.Get("X-Forwarded-Proto") == "https" {
|
||||
scheme = "https"
|
||||
}
|
||||
return scheme + "://" + r.Host
|
||||
}
|
||||
|
||||
func handleRobots(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Cache-Control", "public, max-age=86400")
|
||||
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
|
||||
fmt.Fprintf(w, `User-agent: *
|
||||
Allow: /
|
||||
Disallow: /search
|
||||
Disallow: /api/
|
||||
Disallow: /random
|
||||
Disallow: /pinned
|
||||
Disallow: /preferences
|
||||
|
||||
Sitemap: %s/sitemap.xml
|
||||
`, requestBase(r))
|
||||
}
|
||||
|
||||
type sitemapURL struct {
|
||||
Loc string `xml:"loc"`
|
||||
}
|
||||
|
||||
type sitemapSet struct {
|
||||
XMLName xml.Name `xml:"urlset"`
|
||||
Xmlns string `xml:"xmlns,attr"`
|
||||
URLs []sitemapURL `xml:"url"`
|
||||
}
|
||||
|
||||
func (co *Corpus) buildSitemapPaths() {
|
||||
paths := []string{
|
||||
"/", "/browse", "/articles", "/glossary", "/changelog", "/about",
|
||||
}
|
||||
for _, c := range categories {
|
||||
paths = append(paths, "/browse/"+c.Slug)
|
||||
}
|
||||
paths = append(paths, "/explore")
|
||||
for _, m := range exploreMetrics {
|
||||
paths = append(paths, "/explore/"+m.Slug)
|
||||
}
|
||||
paths = append(paths, "/explore/terms")
|
||||
paths = append(paths, "/explore/classification")
|
||||
paths = append(paths, "/explore/portions")
|
||||
paths = append(paths, "/explore/outlets")
|
||||
if len(co.assemblies) > 0 {
|
||||
paths = append(paths, assembliesPath)
|
||||
for _, a := range co.assemblies {
|
||||
paths = append(paths, assemblyPath(a.ID))
|
||||
}
|
||||
}
|
||||
for _, d := range co.archive.Docs {
|
||||
paths = append(paths, docPath(d.ID))
|
||||
}
|
||||
for _, a := range co.articles {
|
||||
if a.Title != "" {
|
||||
paths = append(paths, "/article/"+a.Key)
|
||||
}
|
||||
}
|
||||
|
||||
co.sitemapPaths = paths
|
||||
}
|
||||
|
||||
func (co *Corpus) marshalSitemap(base string) ([]byte, error) {
|
||||
set := sitemapSet{Xmlns: "http://www.sitemaps.org/schemas/sitemap/0.9"}
|
||||
set.URLs = make([]sitemapURL, len(co.sitemapPaths))
|
||||
for i, p := range co.sitemapPaths {
|
||||
set.URLs[i] = sitemapURL{Loc: base + p}
|
||||
}
|
||||
out, err := xml.MarshalIndent(set, "", " ")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return append([]byte(xml.Header), out...), nil
|
||||
}
|
||||
|
||||
func parseCanonicalBase(base string) (string, error) {
|
||||
u, err := url.Parse(base)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("invalid base URL %q: %w", base, err)
|
||||
}
|
||||
if (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" || u.User != nil ||
|
||||
(u.Path != "" && u.Path != "/") || u.RawQuery != "" || u.Fragment != "" {
|
||||
return "", fmt.Errorf("base must be scheme://host with no path/query/fragment/userinfo, e.g. https://archive.example.org (got %q)", base)
|
||||
}
|
||||
return u.Scheme + "://" + u.Host, nil
|
||||
}
|
||||
|
||||
const sitemapBaseMark = "{base url}"
|
||||
|
||||
func (co *Corpus) buildCanonicalSitemap() {
|
||||
co.sitemapParts = nil
|
||||
if canonicalBase == "" {
|
||||
if b, err := co.marshalSitemap(sitemapBaseMark); err == nil {
|
||||
co.sitemapParts = bytes.Split(b, []byte(sitemapBaseMark))
|
||||
}
|
||||
return
|
||||
}
|
||||
b, err := co.marshalSitemap(canonicalBase)
|
||||
if err != nil {
|
||||
errorf("sitemap: %v - building it per request instead", err)
|
||||
return
|
||||
}
|
||||
co.canonicalSitemap = b
|
||||
}
|
||||
|
||||
func (co *Corpus) handleSitemap(w http.ResponseWriter, r *http.Request) {
|
||||
body := co.canonicalSitemap
|
||||
var base bytes.Buffer
|
||||
if body == nil && co.sitemapParts != nil {
|
||||
xml.EscapeText(&base, []byte(requestBase(r)))
|
||||
} else if body == nil {
|
||||
var err error
|
||||
body, err = co.marshalSitemap(requestBase(r))
|
||||
if err != nil {
|
||||
serverError(w, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
w.Header().Set("Cache-Control", "public, max-age=3600")
|
||||
w.Header().Set("Content-Type", "application/xml; charset=utf-8")
|
||||
if body != nil {
|
||||
w.Write(body)
|
||||
return
|
||||
}
|
||||
for i, part := range co.sitemapParts {
|
||||
if i > 0 {
|
||||
w.Write(base.Bytes())
|
||||
}
|
||||
w.Write(part)
|
||||
}
|
||||
}
|
||||
|
||||
// docEmbeds is handed to doc.html as data["Embeds"].
|
||||
type docEmbeds struct {
|
||||
JSONLDTag template.HTML
|
||||
COinS string
|
||||
}
|
||||
|
||||
func buildDocEmbeds(doc *Document, absURL, pdfURL string) docEmbeds {
|
||||
type entity struct {
|
||||
Type string `json:"@type"`
|
||||
Name string `json:"name"`
|
||||
}
|
||||
type media struct {
|
||||
Type string `json:"@type"`
|
||||
ContentURL string `json:"contentUrl"`
|
||||
EncodingFormat string `json:"encodingFormat"`
|
||||
}
|
||||
keywords := strings.Join(append(append([]string{}, doc.DocMeta.Programs...), doc.Tags...), ", ")
|
||||
ld := struct {
|
||||
Context string `json:"@context"`
|
||||
Type string `json:"@type"`
|
||||
Name string `json:"name"`
|
||||
URL string `json:"url"`
|
||||
Identifier string `json:"identifier"`
|
||||
Description string `json:"description,omitempty"`
|
||||
DateCreated string `json:"dateCreated,omitempty"`
|
||||
DatePublished string `json:"datePublished,omitempty"`
|
||||
InLanguage string `json:"inLanguage"`
|
||||
Keywords string `json:"keywords,omitempty"`
|
||||
Author *entity `json:"author,omitempty"`
|
||||
Publisher entity `json:"publisher"`
|
||||
AssociatedMedia media `json:"associatedMedia"`
|
||||
}{
|
||||
Context: "https://schema.org",
|
||||
Type: "DigitalDocument",
|
||||
Name: doc.Title,
|
||||
URL: absURL,
|
||||
Identifier: absURL,
|
||||
Description: doc.Description,
|
||||
DateCreated: doc.DocMeta.DateCreated,
|
||||
DatePublished: oldestPubDate(doc),
|
||||
InLanguage: "en",
|
||||
Keywords: keywords,
|
||||
Publisher: entity{Type: "Organization", Name: "Snowden Archive"},
|
||||
AssociatedMedia: media{
|
||||
Type: "MediaObject",
|
||||
ContentURL: pdfURL,
|
||||
EncodingFormat: "application/pdf",
|
||||
},
|
||||
}
|
||||
if doc.DocMeta.Creator != "" {
|
||||
ld.Author = &entity{Type: "Organization", Name: doc.DocMeta.Creator}
|
||||
}
|
||||
|
||||
var tag template.HTML
|
||||
if b, err := json.Marshal(ld); err == nil {
|
||||
tag = template.HTML(`<script type="application/ld+json">` + string(b) + `</script>`)
|
||||
}
|
||||
|
||||
v := url.Values{}
|
||||
v.Set("ctx_ver", "Z39.88-2004")
|
||||
v.Set("rft_val_fmt", "info:ofi/fmt:kev:mtx:dc")
|
||||
v.Set("rft.title", doc.Title)
|
||||
if doc.DocMeta.Creator != "" {
|
||||
v.Set("rft.creator", doc.DocMeta.Creator)
|
||||
}
|
||||
if doc.DocMeta.DateCreated != "" {
|
||||
v.Set("rft.date", doc.DocMeta.DateCreated)
|
||||
}
|
||||
v.Set("rft.type", "Text")
|
||||
v.Set("rft.format", "application/pdf")
|
||||
v.Set("rft.identifier", absURL)
|
||||
v.Set("rft.publisher", "Snowden Archive")
|
||||
for _, p := range doc.DocMeta.Programs {
|
||||
v.Add("rft.subject", p)
|
||||
}
|
||||
return docEmbeds{JSONLDTag: tag, COinS: v.Encode()}
|
||||
}
|
||||
415
static/main.js
Normal file
415
static/main.js
Normal file
|
|
@ -0,0 +1,415 @@
|
|||
// saved custom CSS; ?nocss=1 clears it
|
||||
(function () {
|
||||
try {
|
||||
if (/[?&]nocss=1(?:&|$)/.test(location.search)) {
|
||||
localStorage.removeItem('snowchive-custom-css');
|
||||
return;
|
||||
}
|
||||
var css = localStorage.getItem('snowchive-custom-css') || '';
|
||||
if (!css) return;
|
||||
var el = document.createElement('style');
|
||||
el.id = 'snowchive-custom-css';
|
||||
el.textContent = css;
|
||||
document.head.appendChild(el);
|
||||
} catch (e) {}
|
||||
})();
|
||||
|
||||
function snowchiveStorage() {
|
||||
try { return window.localStorage || null; } catch (e) { return null; }
|
||||
}
|
||||
|
||||
function isTyping(el) {
|
||||
return !!el && (/^(INPUT|TEXTAREA|SELECT)$/.test(el.tagName) || el.isContentEditable);
|
||||
}
|
||||
|
||||
// animated thumb rollover
|
||||
(function () {
|
||||
if (window.matchMedia && matchMedia('(prefers-reduced-motion: reduce)').matches) return;
|
||||
const gifThumbs = Array.from(document.querySelectorAll('.thumb img[data-gif], .result-thumb img[data-gif]'));
|
||||
gifThumbs.forEach(img => {
|
||||
img.dataset.still = img.getAttribute('src');
|
||||
const gif = img.getAttribute('data-gif');
|
||||
|
||||
img.addEventListener('mouseenter', () => img.setAttribute('src', gif));
|
||||
img.addEventListener('mouseleave', () => img.setAttribute('src', img.dataset.still));
|
||||
|
||||
img.addEventListener('touchstart', () => {
|
||||
gifThumbs.forEach(o => {
|
||||
if (o !== img) o.setAttribute('src', o.dataset.still);
|
||||
});
|
||||
img.setAttribute('src', gif);
|
||||
}, { passive: true });
|
||||
});
|
||||
|
||||
})();
|
||||
|
||||
// card page-preview panel (opt-in, desktop)
|
||||
(function () {
|
||||
if (!window.matchMedia || !matchMedia('(hover: hover) and (pointer: fine)').matches) return;
|
||||
if (!document.querySelector('[data-pp-id]')) return;
|
||||
|
||||
var panel = null, activeEl = null, pages = [], idx = 0, docId = '', timer = null, dragging = false, pinned = false;
|
||||
var openTimer = null;
|
||||
var WKEY = 'snowchive.pp.width', WMIN = 300, WDEFAULT = 460;
|
||||
|
||||
function imgURL(name) { return '/files/' + docId + '/images/' + name; }
|
||||
function cancelClose() { if (timer) { clearTimeout(timer); timer = null; } }
|
||||
function scheduleClose() { if (dragging || pinned) return; cancelClose(); timer = setTimeout(close_, 220); }
|
||||
|
||||
function maxW() { return Math.round(window.innerWidth * 0.9); }
|
||||
function clampW(w) { return Math.max(WMIN, Math.min(w, maxW())); }
|
||||
function loadW() { try { var v = parseInt(localStorage.getItem(WKEY), 10); if (v) return v; } catch (e) {} return WDEFAULT; }
|
||||
function saveW(w) { try { localStorage.setItem(WKEY, String(w)); } catch (e) {} }
|
||||
function applyW(w) { if (panel) panel.style.width = clampW(w) + 'px'; }
|
||||
|
||||
function build() {
|
||||
panel = document.createElement('div');
|
||||
panel.className = 'pp-panel';
|
||||
panel.innerHTML =
|
||||
'<div class="pp-resizer" title="Drag to resize (press s to pin first)"></div>' +
|
||||
'<div class="pp-stage"><img class="pp-img" alt=""></div>' +
|
||||
'<div class="pp-strip"></div>' +
|
||||
'<div class="pp-foot">' +
|
||||
'<span class="pp-counter"></span>' +
|
||||
'<span class="pp-hint pp-hint-free">h / ← prev · l / → next · <b>s</b> pin to resize</span>' +
|
||||
'<span class="pp-hint pp-hint-pinned">📌 pinned · h / ← prev · l / → next · <b>esc</b> close</span>' +
|
||||
'</div>';
|
||||
document.body.appendChild(panel);
|
||||
applyW(loadW());
|
||||
initResize();
|
||||
panel.addEventListener('mouseenter', cancelClose);
|
||||
panel.addEventListener('mouseleave', scheduleClose);
|
||||
}
|
||||
|
||||
function initResize() {
|
||||
var resizer = panel.querySelector('.pp-resizer');
|
||||
function onMove(e) { applyW(window.innerWidth - e.clientX); }
|
||||
function onUp() {
|
||||
dragging = false;
|
||||
resizer.classList.remove('pp-dragging');
|
||||
document.body.style.userSelect = '';
|
||||
document.removeEventListener('mousemove', onMove);
|
||||
document.removeEventListener('mouseup', onUp);
|
||||
saveW(parseInt(panel.style.width, 10));
|
||||
}
|
||||
resizer.addEventListener('mousedown', function (e) {
|
||||
e.preventDefault();
|
||||
dragging = true;
|
||||
cancelClose();
|
||||
resizer.classList.add('pp-dragging');
|
||||
document.body.style.userSelect = 'none';
|
||||
document.addEventListener('mousemove', onMove);
|
||||
document.addEventListener('mouseup', onUp);
|
||||
});
|
||||
}
|
||||
|
||||
function go(n) {
|
||||
if (!pages.length) return;
|
||||
idx = n < 0 ? 0 : (n >= pages.length ? pages.length - 1 : n);
|
||||
panel.querySelector('.pp-img').src = imgURL(pages[idx]);
|
||||
panel.querySelector('.pp-counter').textContent = (idx + 1) + ' / ' + pages.length;
|
||||
var strip = panel.querySelector('.pp-strip'), i;
|
||||
for (i = 0; i < strip.children.length; i++) {
|
||||
strip.children[i].classList.toggle('pp-cur', i === idx);
|
||||
}
|
||||
var cur = strip.children[idx];
|
||||
if (cur) {
|
||||
var cr = cur.getBoundingClientRect(), sr = strip.getBoundingClientRect();
|
||||
strip.scrollLeft += (cr.left + cr.width / 2) - (sr.left + sr.width / 2);
|
||||
}
|
||||
}
|
||||
|
||||
function buildStrip() {
|
||||
var strip = panel.querySelector('.pp-strip');
|
||||
strip.textContent = '';
|
||||
pages.forEach(function (name, i) {
|
||||
var t = new Image();
|
||||
t.className = 'pp-th';
|
||||
t.loading = 'lazy';
|
||||
t.src = imgURL(name);
|
||||
t.alt = 'Page ' + (i + 1);
|
||||
t.addEventListener('click', function () { go(i); });
|
||||
strip.appendChild(t);
|
||||
});
|
||||
}
|
||||
|
||||
function open_(el) {
|
||||
cancelClose();
|
||||
if (el === activeEl) return;
|
||||
var imgs = (el.getAttribute('data-pp-imgs') || '').split(' ').filter(Boolean);
|
||||
if (!imgs.length) { close_(); return; }
|
||||
activeEl = el;
|
||||
docId = el.getAttribute('data-pp-id');
|
||||
pages = imgs;
|
||||
if (!panel) build();
|
||||
buildStrip();
|
||||
go(0);
|
||||
panel.classList.add('pp-open');
|
||||
}
|
||||
|
||||
function close_() {
|
||||
cancelClose();
|
||||
pinned = false;
|
||||
if (panel) { panel.classList.remove('pp-open'); panel.classList.remove('pp-pinned'); }
|
||||
activeEl = null;
|
||||
}
|
||||
|
||||
document.addEventListener('mouseover', function (e) {
|
||||
if (dragging || pinned) return;
|
||||
var el = e.target.closest('[data-pp-id]');
|
||||
if (!el) return;
|
||||
if (el === activeEl) { cancelClose(); return; }
|
||||
clearTimeout(openTimer);
|
||||
openTimer = setTimeout(function () { open_(el); }, 150);
|
||||
});
|
||||
document.addEventListener('mouseout', function (e) {
|
||||
var el = e.target.closest('[data-pp-id]');
|
||||
if (!el) return;
|
||||
var to = e.relatedTarget;
|
||||
if (to && (el.contains(to) || (panel && panel.contains(to)))) return;
|
||||
clearTimeout(openTimer);
|
||||
scheduleClose();
|
||||
});
|
||||
document.addEventListener('keydown', function (e) {
|
||||
if (!activeEl) return;
|
||||
if (e.altKey || e.ctrlKey || e.metaKey) return;
|
||||
if (isTyping(document.activeElement)) return;
|
||||
switch (e.key) {
|
||||
case 'h': case 'ArrowLeft': go(idx - 1); e.preventDefault(); break;
|
||||
case 'l': case 'ArrowRight': go(idx + 1); e.preventDefault(); break;
|
||||
case 'Home': if (pinned) { go(0); e.preventDefault(); } break;
|
||||
case 'End': if (pinned) { go(pages.length - 1); e.preventDefault(); } break;
|
||||
case 's': case 'S':
|
||||
pinned = !pinned;
|
||||
panel.classList.toggle('pp-pinned', pinned);
|
||||
if (!pinned && !(panel.matches(':hover') || (activeEl && activeEl.matches(':hover')))) close_();
|
||||
e.preventDefault();
|
||||
break;
|
||||
case 'Escape': close_(); break;
|
||||
}
|
||||
});
|
||||
window.addEventListener('resize', function () { if (panel) applyW(parseInt(panel.style.width, 10) || WDEFAULT); });
|
||||
})();
|
||||
|
||||
// lightbox keyboard and swipe navigation
|
||||
(function () {
|
||||
var total = document.querySelectorAll('.lightbox:not(.lb-alt)').length;
|
||||
if (!total) return;
|
||||
document.addEventListener('keydown', function (e) {
|
||||
if (e.altKey || e.ctrlKey || e.metaKey) return;
|
||||
if (isTyping(e.target)) return;
|
||||
if (e.key === 'Escape' && /^#lb-a\d+-\d+$/.test(location.hash)) {
|
||||
location.hash = '#lb-closed';
|
||||
e.preventDefault();
|
||||
return;
|
||||
}
|
||||
var m = /^#lb-(\d+)$/.exec(location.hash);
|
||||
if (!m) return;
|
||||
var n = parseInt(m[1], 10);
|
||||
switch (e.key) {
|
||||
case 'ArrowLeft': case 'h':
|
||||
if (n > 1) location.hash = '#lb-' + (n - 1);
|
||||
e.preventDefault();
|
||||
break;
|
||||
case 'ArrowRight': case 'l':
|
||||
if (n < total) location.hash = '#lb-' + (n + 1);
|
||||
e.preventDefault();
|
||||
break;
|
||||
case 'Escape':
|
||||
location.hash = '#lb-closed';
|
||||
e.preventDefault();
|
||||
break;
|
||||
}
|
||||
});
|
||||
|
||||
window.addEventListener('hashchange', function (e) {
|
||||
var was = /#lb-(\d+)$/.exec(e.oldURL), is = /#lb-(\d+)$/.exec(e.newURL);
|
||||
if (is) {
|
||||
var close = document.querySelector('#lb-' + is[1] + ' .lb-close');
|
||||
if (close) close.focus({ preventScroll: true });
|
||||
} else if (was) {
|
||||
var thumbs = document.querySelectorAll('a[href="#lb-' + was[1] + '"]');
|
||||
for (var i = 0; i < thumbs.length; i++) {
|
||||
if (!thumbs[i].closest('.lightbox')) { thumbs[i].focus(); break; }
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
var tx = 0, ty = 0;
|
||||
document.addEventListener('touchstart', function (e) {
|
||||
if (e.touches.length > 1) { tx = NaN; return; }
|
||||
tx = e.changedTouches[0].clientX;
|
||||
ty = e.changedTouches[0].clientY;
|
||||
}, { passive: true });
|
||||
document.addEventListener('touchend', function (e) {
|
||||
var m = /^#lb-(\d+)$/.exec(location.hash);
|
||||
if (!m) return;
|
||||
var dx = e.changedTouches[0].clientX - tx;
|
||||
var dy = e.changedTouches[0].clientY - ty;
|
||||
if (!(Math.abs(dx) >= 50) || Math.abs(dy) > Math.abs(dx)) return;
|
||||
var n = parseInt(m[1], 10);
|
||||
if (dx < 0 && n < total) location.hash = '#lb-' + (n + 1);
|
||||
else if (dx > 0 && n > 1) location.hash = '#lb-' + (n - 1);
|
||||
}, { passive: true });
|
||||
})();
|
||||
|
||||
// private per-document notes
|
||||
(function () {
|
||||
var box = document.getElementById('note-box');
|
||||
var localStorage = snowchiveStorage();
|
||||
if (!box || !localStorage) return;
|
||||
box.hidden = false;
|
||||
|
||||
var key = 'snowchive-note:' + box.dataset.docId;
|
||||
var ta = document.getElementById('note-text');
|
||||
var status = document.getElementById('note-status');
|
||||
|
||||
try {
|
||||
var saved = localStorage.getItem(key);
|
||||
if (saved) {
|
||||
var obj = JSON.parse(saved);
|
||||
ta.value = obj.text || '';
|
||||
if (ta.value) { box.open = true; status.textContent = '(saved)'; }
|
||||
}
|
||||
} catch (e) {}
|
||||
|
||||
var t = null;
|
||||
function save() {
|
||||
clearTimeout(t);
|
||||
t = null;
|
||||
try {
|
||||
if (ta.value.trim() === '') {
|
||||
localStorage.removeItem(key);
|
||||
status.textContent = '';
|
||||
} else {
|
||||
localStorage.setItem(key, JSON.stringify({
|
||||
title: box.dataset.docTitle,
|
||||
text: ta.value,
|
||||
updated: new Date().toISOString()
|
||||
}));
|
||||
status.textContent = '(saved)';
|
||||
}
|
||||
} catch (e) { status.textContent = '(could not save)'; }
|
||||
}
|
||||
ta.addEventListener('input', function () {
|
||||
status.textContent = '...';
|
||||
clearTimeout(t);
|
||||
t = setTimeout(save, 400);
|
||||
});
|
||||
function flush() { if (t !== null) save(); }
|
||||
ta.addEventListener('change', flush);
|
||||
window.addEventListener('pagehide', flush);
|
||||
})();
|
||||
|
||||
// the Pinned page's note listing
|
||||
(function () {
|
||||
var box = document.getElementById('note-list-box');
|
||||
var list = document.getElementById('note-list');
|
||||
if (!box || !list) return;
|
||||
var localStorage = snowchiveStorage();
|
||||
if (!localStorage) return;
|
||||
|
||||
var PREFIX = 'snowchive-note:';
|
||||
var items = [];
|
||||
try {
|
||||
for (var i = 0; i < localStorage.length; i++) {
|
||||
var k = localStorage.key(i);
|
||||
if (!k || k.indexOf(PREFIX) !== 0) continue;
|
||||
try {
|
||||
var obj = JSON.parse(localStorage.getItem(k));
|
||||
items.push({ id: k.slice(PREFIX.length), title: obj.title || k.slice(PREFIX.length),
|
||||
text: obj.text || '', updated: obj.updated || '' });
|
||||
} catch (e) {}
|
||||
}
|
||||
} catch (e) {}
|
||||
if (!items.length) return;
|
||||
items.sort(function (a, b) { return a.updated < b.updated ? 1 : -1; });
|
||||
|
||||
box.hidden = false;
|
||||
|
||||
items.forEach(function (it) {
|
||||
var li = document.createElement('li');
|
||||
li.className = 'note-item';
|
||||
|
||||
var a = document.createElement('a');
|
||||
a.href = '/doc/' + encodeURIComponent(it.id);
|
||||
a.textContent = it.title;
|
||||
li.appendChild(a);
|
||||
|
||||
if (it.updated) {
|
||||
var d = document.createElement('span');
|
||||
d.className = 'faint note-date';
|
||||
d.textContent = ' ' + it.updated.slice(0, 10);
|
||||
li.appendChild(d);
|
||||
}
|
||||
|
||||
var del = document.createElement('button');
|
||||
del.type = 'button';
|
||||
del.className = 'link-btn unpin-btn';
|
||||
del.textContent = 'delete';
|
||||
del.addEventListener('click', function () {
|
||||
if (!confirm('Delete this note?')) return;
|
||||
try { localStorage.removeItem(PREFIX + it.id); } catch (e) { return; }
|
||||
items.splice(items.indexOf(it), 1);
|
||||
li.remove();
|
||||
if (!list.children.length) box.hidden = true;
|
||||
});
|
||||
li.appendChild(del);
|
||||
|
||||
var p = document.createElement('div');
|
||||
p.className = 'note-text';
|
||||
p.textContent = it.text;
|
||||
li.appendChild(p);
|
||||
|
||||
list.appendChild(li);
|
||||
});
|
||||
|
||||
document.getElementById('note-export').addEventListener('click', function () {
|
||||
var blob = new Blob([JSON.stringify(items, null, 2)], { type: 'application/json' });
|
||||
var a = document.createElement('a');
|
||||
a.href = URL.createObjectURL(blob);
|
||||
a.download = 'snowchive-notes.json';
|
||||
a.click();
|
||||
// revoked later: Safari starts the download after click() returns
|
||||
setTimeout(function () { URL.revokeObjectURL(a.href); }, 40000);
|
||||
});
|
||||
})();
|
||||
|
||||
// the Preferences page's custom-CSS editor
|
||||
(function () {
|
||||
var localStorage = snowchiveStorage();
|
||||
if (!localStorage) return;
|
||||
var fs = document.getElementById('custom-css-details');
|
||||
var ta = document.getElementById('custom-css');
|
||||
var status = document.getElementById('custom-css-status');
|
||||
if (!fs || !ta || !status) return;
|
||||
fs.hidden = false;
|
||||
|
||||
var KEY = 'snowchive-custom-css';
|
||||
try {
|
||||
ta.value = localStorage.getItem(KEY) || '';
|
||||
} catch (e) {}
|
||||
|
||||
function apply() {
|
||||
var existing = document.getElementById('snowchive-custom-css');
|
||||
if (existing) existing.remove();
|
||||
if (ta.value) {
|
||||
var el = document.createElement('style');
|
||||
el.id = 'snowchive-custom-css';
|
||||
el.textContent = ta.value;
|
||||
document.head.appendChild(el);
|
||||
}
|
||||
}
|
||||
|
||||
document.getElementById('custom-css-save').addEventListener('click', function () {
|
||||
try {
|
||||
if (ta.value) {
|
||||
localStorage.setItem(KEY, ta.value);
|
||||
} else {
|
||||
localStorage.removeItem(KEY);
|
||||
}
|
||||
apply();
|
||||
status.textContent = ta.value ? 'Saved and applied.' : 'Cleared - custom CSS is off.';
|
||||
} catch (e) { status.textContent = 'Could not save.'; }
|
||||
});
|
||||
})();
|
||||
2353
static/style.css
Normal file
2353
static/style.css
Normal file
File diff suppressed because it is too large
Load diff
14
templates/404.html
Normal file
14
templates/404.html
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
{{ define "title" }}Page not found - Snowden Archive{{ end }}
|
||||
{{ define "content" }}
|
||||
<div class="notfound">
|
||||
<h1>404 - Page not found</h1>
|
||||
<p class="muted">Nothing lives at <code>{{ .Path }}</code>. It may have been moved, removed, or mistyped.</p>
|
||||
|
||||
<form action="/search" method="GET" class="home-search-form" role="search">
|
||||
<input type="text" name="q" class="home-search" placeholder="Search the archive... (Enter to search)" aria-label="Search the archive">
|
||||
</form>
|
||||
|
||||
<p><a href="/">← Home</a> · <a href="/browse">Browse the archive</a></p>
|
||||
</div>
|
||||
{{ end }}
|
||||
{{ template "base.html" . }}
|
||||
544
templates/about.html
Normal file
544
templates/about.html
Normal file
|
|
@ -0,0 +1,544 @@
|
|||
{{ define "title" }}About - Snowden Archive{{ end }}
|
||||
{{ define "content" }}
|
||||
<div class="about-page">
|
||||
<h1>About this archive</h1>
|
||||
|
||||
<p>This is a searchable archive of all published documents from the 2013
|
||||
Snowden disclosures. The archive includes documents published alongside
|
||||
news reporting, The Intercept's bulk SIDtoday release, and FOIA material
|
||||
(official releases from FOIA litigation and FISC and ODNI
|
||||
declassifications).</p>
|
||||
<p>Every document carries its original PDF, page images, extracted text,
|
||||
and curated metadata - agencies, programs, countries, classification
|
||||
markings, people, and the journalists and articles that published it.</p>
|
||||
<p>Use <a href="/browse">Browse</a> to explore by category,
|
||||
<a href="/search">Search</a> for fielded boolean queries across text and
|
||||
metadata, <a href="/articles">Articles</a> to follow the disclosures story
|
||||
by story, the <a href="/glossary">Glossary</a> to decode codenames and
|
||||
acronyms, and <a href="/explore">Explore</a> to dig deeper into ranked
|
||||
views and lists.</p>
|
||||
|
||||
<nav class="toc" aria-label="Table of contents">
|
||||
<strong>Contents</strong>
|
||||
<ul>
|
||||
<li><a href="#privacy">Privacy</a></li>
|
||||
<li><a href="#whats-in">What's in this archive?</a></li>
|
||||
<li><a href="#search">How to search</a></li>
|
||||
{{ if .APIEnabled }}<li><a href="#api">Downloading data as JSON</a>
|
||||
<ul><li><a href="#metadata">Metadata structure</a></li></ul>
|
||||
</li>{{ else }}<li><a href="#metadata">Metadata structure</a></li>{{ end }}
|
||||
<li><a href="#download-archive">Downloading the archive</a>
|
||||
<ul>
|
||||
<li><a href="#downloading-all-data">Downloading all data</a></li>
|
||||
<li><a href="#downloading-texts">Downloading only extracted texts and metadata</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li><a href="#own-copy">Can I run my own copy or mirror this?</a></li>
|
||||
<li><a href="#contact">Contact</a></li>
|
||||
<li><a href="#contributing">Contributing</a></li>
|
||||
<li><a href="#thanks">Thank you</a></li>
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
<section id="privacy">
|
||||
<h2>Privacy</h2>
|
||||
<p>By default, the server keeps no state about its visitors. Pins and
|
||||
history live only in your own browser's cookies, and history features
|
||||
are off unless you opt in via <a href="/preferences">Preferences</a>.
|
||||
Anonymous page views set no cookies at all. All core features on this
|
||||
website work without JavaScript.</p>
|
||||
<p>Note that you can't verify that this server doesn't log and analyze
|
||||
your visits. Use Tor to hide your real IP address, or
|
||||
<a href="#own-copy">set up</a> your own instance on your machine to
|
||||
browse fully offline.</p>
|
||||
</section>
|
||||
|
||||
<section id="whats-in">
|
||||
<h2>What's in this archive?</h2>
|
||||
<p>By default, two collections are shown as tabs on the
|
||||
<a href="/browse">Browse</a> page: <strong>Reported</strong> documents
|
||||
(leaked files as the outlets that broke the stories presented them) and
|
||||
the <strong>SIDtoday Archive</strong> (The Intercept's bulk release of
|
||||
the NSA's internal SIDtoday newsletters). The third collection,
|
||||
<strong>FOIA</strong> (official releases from FOIA litigation and FISC
|
||||
and ODNI declassifications), can be included via
|
||||
<a href="/preferences">Preferences</a>.</p>
|
||||
<p>The metadata and many of the extracted texts are curated by hand and
|
||||
may contain errors or omissions, so treat them as a guide to the
|
||||
documents rather than as established fact, and check the original
|
||||
documents and reporting before relying on them.</p>
|
||||
</section>
|
||||
|
||||
<section id="search">
|
||||
<h2>How to search</h2>
|
||||
<p>The <a href="/search">Search</a> page supports fielded boolean
|
||||
queries. Restrict a term to a field (title, full text, programs,
|
||||
countries, journalists, and so on), combine clauses with AND/OR, quote
|
||||
<code>"text"</code> to find it anywhere, match exact whole words with
|
||||
<code>=...</code> (<code>publisher:="The Guardian"</code> matches the
|
||||
whole value), and use <code>*</code> wildcards (<code>tap*</code>,
|
||||
<code>*apping</code>).</p>
|
||||
|
||||
<p>Ways to write a value, in any row or after any operator:</p>
|
||||
<ul>
|
||||
<li><code>cable tapping</code> finds both words anywhere, in their other forms too (<em>tapping</em> also finds <em>tapped</em>).</li>
|
||||
<li><code>"apping"</code> finds that text anywhere, even inside a word, like <code>grep -i</code> (it finds, for example, <em>mapping</em> and <em>tapping</em>).</li>
|
||||
<li><code>=tap</code> or <code>="cable tapping"</code> finds exactly those whole words, so <code>=tap</code> skips <em>taps</em> and <em>strap</em>. On a metadata field it matches one whole value, so <code>agency:=NSA</code> finds documents whose agency is NSA but not <em>NSA, GCHQ</em>.</li>
|
||||
<li>The <code>*</code> wildcard finds part of a word: <code>tap*</code> finds words starting with <em>tap</em>, <code>*land</code> words ending in <em>land</em>, and <code>*keys*</code> words containing <em>keys</em>. A leading <code>*</code> needs at least three letters after it.</li>
|
||||
<li><code>"upstream collection"~10</code> finds the words within 10 words of each other (in any order, common stopwords ignored).</li>
|
||||
<li>A leading <code>-</code> excludes within its row: <code>xkeyscore -tempora</code> finds <em>xkeyscore</em> without <em>tempora</em>, and the row keeps that meaning under OR and NOT.</li>
|
||||
</ul>
|
||||
|
||||
<p>The simple search box (on the home page, on the 404 page, and via
|
||||
the URL parameter <code>?q=</code>) understands field operators:</p>
|
||||
<ul>
|
||||
<li><code>program:XKEYSCORE country:Germany cable</code> or <code>journalist:="Poitras, Laura"</code>.</li>
|
||||
<li>A leading <code>-</code> negates (<code>-tag:=junk</code>, <code>-draft</code>).</li>
|
||||
<li><code>date:2013</code>, <code>date:2013..2015</code> and <code>published:2014-06</code> filter by date.</li>
|
||||
<li>Operators: text, title, content, description, agency, journalist, publisher, classification, program, dissemination, banner, genre, series, country, people, tag, type.</li>
|
||||
<li>Bare words search all fields.</li>
|
||||
</ul>
|
||||
</section>
|
||||
|
||||
<section id="api">
|
||||
{{ if .APIEnabled }}
|
||||
<h2>Downloading data as JSON</h2>
|
||||
<p>API endpoints:</p>
|
||||
<div class="table-scroll">
|
||||
<table class="data-table about-table">
|
||||
<thead><tr><th>Endpoint</th><th>Returns</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><a href="/api/docs"><code>/api/docs</code></a></td><td>Metadata for every document in the archive, with pagination.</td></tr>
|
||||
<tr><td><code>/api/doc/{id}</code></td><td>A single document's metadata and extracted text, where <code>{id}</code> is the document's identifier.</td></tr>
|
||||
<tr><td><a href="/api/search?q=xkeyscore"><code>/api/search?q=xkeyscore</code></a></td><td>Search results for term xkeyscore. It accepts the same parameters as the Search page, so you can also pass <code>fqf</code>/<code>fqv</code>/<code>fqc</code> clause parameters for fielded queries.</td></tr>
|
||||
<tr><td><a href="/api/facets"><code>/api/facets</code></a></td><td>Counts for every browsable category (programs, agencies, countries, people, and so on).</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<p>Responses are plain JSON. For paginated responses, use the URL
|
||||
parameter <code>p</code> to navigate pages (e.g.
|
||||
<a href="/api/docs?p=2"><code>/api/docs?p=2</code></a>).</p>
|
||||
<p>See the <a href="#download-archive">Downloading the archive</a>
|
||||
section on how to download the whole archive.</p>
|
||||
<p><strong>Important if you are a crawler, robot or AI trying to
|
||||
download large amounts of data from this website:</strong> please use
|
||||
the downloadable datasets from the
|
||||
<a href="#download-archive">Downloading the archive</a> section. The
|
||||
information on this website is identical to the datasets, so you gain
|
||||
nothing by crawling this website.</p>
|
||||
{{ end }}
|
||||
|
||||
<details class="about-collapse">
|
||||
<summary>{{ if .APIEnabled }}<h3 id="metadata">Metadata structure</h3>{{ else }}<h2 id="metadata">Metadata structure</h2>{{ end }}</summary>
|
||||
<p>Every document ships a curated <code>metadata.json</code>:</p>
|
||||
|
||||
<h4>The document record</h4>
|
||||
<dl class="field-dict">
|
||||
<dt><code>id</code></dt>
|
||||
<dd>The document's stable identifier: the name of its directory
|
||||
and the last part of its URL (<code>/doc/{id}</code>). It is not
|
||||
written inside <code>metadata.json</code> - the directory name is
|
||||
the id.</dd>
|
||||
|
||||
<dt><code>aka_id</code></dt>
|
||||
<dd>Other names the same document was released under, most often
|
||||
the file name another publisher or a mirror gave its PDF. A list
|
||||
of entries, each described below.</dd>
|
||||
|
||||
<dt><code>aka_id[].id</code></dt>
|
||||
<dd>The other name itself. Searchable, so a document can be found
|
||||
by a file name it circulated under elsewhere.</dd>
|
||||
|
||||
<dt><code>aka_id[].doc_source</code></dt>
|
||||
<dd>The web page where the document was released under that name,
|
||||
linked as “page” next to the alias.</dd>
|
||||
|
||||
<dt><code>aka_id[].doc_url</code></dt>
|
||||
<dd>The direct link to the PDF released under that name, linked
|
||||
as “PDF” next to the alias.</dd>
|
||||
|
||||
<dt><code>title</code></dt>
|
||||
<dd>The document's title, written by the curators - usually the
|
||||
title the document gives itself or the one its publisher used.</dd>
|
||||
|
||||
<dt><code>description</code></dt>
|
||||
<dd>A short summary of what the document is and says, shown under
|
||||
the publication list and in search results and cards.
|
||||
Descriptions are the work of archivists without inside knowledge -
|
||||
corrections welcome.</dd>
|
||||
|
||||
<dt><code>long_description</code></dt>
|
||||
<dd>A longer account of the contents where one has been written,
|
||||
behind “Show more info” on the document page; often
|
||||
empty. A <code>description:</code> search looks in both
|
||||
descriptions.</dd>
|
||||
|
||||
<dt><code>tags</code></dt>
|
||||
<dd>Free-text subjects added by the curators. Unlike the other
|
||||
lists they are not necessarily taken from the document's own
|
||||
vocabulary, so they are the loosest thing to browse or search by.
|
||||
</dd>
|
||||
|
||||
<dt><code>type</code></dt>
|
||||
<dd>The kind of release: <code>REPORTED</code> (a leaked document
|
||||
as an outlet published it), <code>SIDTODAY</code> (an article of
|
||||
the NSA's internal SIDtoday newsletter from The Intercept's bulk
|
||||
release) or <code>FOIA</code> (obtained through a public-records
|
||||
request). When it is empty, a document filed in the SIDtoday or FOIA
|
||||
folder takes that type and any other is <code>REPORTED</code>.</dd>
|
||||
|
||||
<dt><code>collection</code></dt>
|
||||
<dd>Which of the three collections - the tabs on
|
||||
<a href="/browse">Browse</a> - the document belongs to:
|
||||
<code>news</code> (Reported), <code>sidtoday-archive</code> or
|
||||
<code>foia</code>. Documents filed under the SIDtoday and FOIA
|
||||
folders belong to those collections whatever this says; elsewhere
|
||||
an empty value means Reported, or FOIA for a document of type
|
||||
<code>FOIA</code>. FOIA releases are left out of
|
||||
browsing and search unless you include them in
|
||||
<a href="/preferences">Preferences</a>.</dd>
|
||||
|
||||
<dt><code>pages</code></dt>
|
||||
<dd>The page count of the PDF. When it is missing, the number of
|
||||
page images is used instead.</dd>
|
||||
|
||||
<dt><code>pdf_sha256</code></dt>
|
||||
<dd>The SHA-256 of the PDF as archived.</dd>
|
||||
|
||||
<dt><code>archive_comment</code></dt>
|
||||
<dd>A note from the curators about this archive's copy - how it
|
||||
was assembled, where it was found, what differs between releases,
|
||||
or a known problem with it.</dd>
|
||||
</dl>
|
||||
|
||||
<h4>The document itself: <code>document</code></h4>
|
||||
<dl class="field-dict">
|
||||
<dt><code>document.creator</code></dt>
|
||||
<dd>The agency that <em>wrote</em> the document, in its short form
|
||||
(<code>NSA</code>, <code>GCHQ</code>, <code>FISC</code>,
|
||||
<code>DOJ</code>, <code>CSE</code> ...) - not the agency it is
|
||||
about and not the one that published it. Empty when the author is
|
||||
not known.</dd>
|
||||
|
||||
<dt><code>document.date_created</code></dt>
|
||||
<dd>When the document was written, as precise as the document
|
||||
allows: <code>2013-03-20</code>, <code>2013-03</code>,
|
||||
<code>2013</code>, or empty when unknown. Nothing is invented to
|
||||
fill the gaps, and nothing is guessed from the publication date.
|
||||
</dd>
|
||||
|
||||
<dt><code>document.banner</code></dt>
|
||||
<dd>The marking line <strong>verbatim</strong> - what the
|
||||
document says about itself, qualifiers and all. Intranet pages
|
||||
often carry a ceiling banner (“DYNAMIC PAGE - HIGHEST
|
||||
POSSIBLE CLASSIFICATION IS ...”) while their content is
|
||||
portion-marked unclassified; the banner records the claim, not an
|
||||
assessment. Empty means the artifact has no marking line - nothing
|
||||
is reconstructed from portion marks.</dd>
|
||||
|
||||
<dt><code>document.classification</code></dt>
|
||||
<dd>The curator's <strong>assessment</strong> of the content's
|
||||
classification, as component values: the level
|
||||
(<code>TOP SECRET</code>, <code>SECRET</code>,
|
||||
<code>CONFIDENTIAL</code>, <code>UNCLASSIFIED</code>) and any
|
||||
control systems (<code>COMINT</code>, <code>SI</code>,
|
||||
<code>TK</code>, <code>STRAP 1</code> ...), judged from portion
|
||||
marks. If a document contains multiple classifications, highest is
|
||||
used here.</dd>
|
||||
|
||||
<dt><code>document.dissemination</code></dt>
|
||||
<dd>The curator's assessment of the handling controls, as
|
||||
component values (<code>FOUO</code>, <code>NOFORN</code>,
|
||||
<code>ORCON</code>, <code>REL TO ...</code>). Assessed the same
|
||||
way as the classification.</dd>
|
||||
|
||||
<dt><code>document.genre</code></dt>
|
||||
<dd>The document's form, from a fixed vocabulary:
|
||||
<code>slides</code>, <code>report</code>, <code>wiki</code>,
|
||||
<code>manual</code>, <code>guide</code>,
|
||||
<code>newsletter</code>, <code>other</code>.</dd>
|
||||
|
||||
<dt><code>document.series.name</code></dt>
|
||||
<dd>For a document that prints a series roster (currently only in
|
||||
SIDtoday): the series name verbatim, as the document prints it.
|
||||
Documents with the same name are listed together on the document
|
||||
page and under <a href="/browse/series">Series</a>.</dd>
|
||||
|
||||
<dt><code>document.series.part</code></dt>
|
||||
<dd>This document's 1-based position in the printed roster. Order
|
||||
comes from the artifact; parts of a series not yet in the archive
|
||||
show as gaps in the numbering, never renumbered away.</dd>
|
||||
|
||||
<dt><code>document.programs</code></dt>
|
||||
<dd>The codenames and programs the document names
|
||||
(<code>XKEYSCORE</code>, <code>PRISM</code>, ...), as curated
|
||||
values for browsing and search; the
|
||||
<a href="/glossary">Glossary</a> explains many of them.</dd>
|
||||
|
||||
<dt><code>document.countries</code></dt>
|
||||
<dd>The countries the document mentions, by their English names.
|
||||
A mention, not a claim that the country was a target or a partner.
|
||||
</dd>
|
||||
|
||||
<dt><code>document.people</code></dt>
|
||||
<dd>The people who appear in the document: named in its text, or
|
||||
recognised in its images - for example
|
||||
<a href="/doc/gchq-mobile-networks-in-my-noc-world">Mobile Networks
|
||||
in My NOC World</a> lists King Charles and Queen Camilla,
|
||||
who appear only in a photograph on one of its slides. Usually
|
||||
written “Surname, Given name”. A person appearing in
|
||||
these documents implies nothing about them beyond that appearance.
|
||||
The journalists who published the document are under
|
||||
<code>publication[].reporters</code>, not here.</dd>
|
||||
</dl>
|
||||
|
||||
<h4>Where it was published: <code>publication[]</code></h4>
|
||||
<p>One entry per story or book that published (or reported) the
|
||||
document. Browse-by-publication-date counts a document under its
|
||||
<em>oldest</em> dated entry; a publication-date search matches
|
||||
<em>any</em> entry's date - or, combined with a journalist or
|
||||
publisher, that entry's date.</p>
|
||||
<dl class="field-dict">
|
||||
<dt><code>publication[].publisher</code></dt>
|
||||
<dd>Who published it: a news outlet, a book's publisher, or for a
|
||||
FOIA release the organization that obtained and released it.
|
||||
Browsable under <a href="/browse/publisher">Publishers</a> - except
|
||||
entries marked <code>reporting_only</code>.</dd>
|
||||
|
||||
<dt><code>publication[].title</code></dt>
|
||||
<dd>The headline of the article, or the book's title. Shown on the
|
||||
document page and names the story on
|
||||
<a href="/articles">Articles</a>.</dd>
|
||||
|
||||
<dt><code>publication[].link</code></dt>
|
||||
<dd>The URL of the article. It is the story's identity: documents
|
||||
whose entries share a link were published in the same report and
|
||||
are listed together on its article page.</dd>
|
||||
|
||||
<dt><code>publication[].date_published</code></dt>
|
||||
<dd>When this publication appeared, in the same
|
||||
<code>YYYY-MM-DD</code> / <code>YYYY-MM</code> /
|
||||
<code>YYYY</code> forms as the document date.</dd>
|
||||
|
||||
<dt><code>publication[].reporters</code></dt>
|
||||
<dd>The journalists bylined on the story, written “Surname,
|
||||
Given name”. Browsable under
|
||||
<a href="/browse/journalist">Journalists</a> and searchable with
|
||||
<code>journalist:</code>.</dd>
|
||||
|
||||
<dt><code>publication[].contributors</code></dt>
|
||||
<dd>People credited on the story without a byline - additional
|
||||
reporting, research, technical analysis. Shown on the document
|
||||
page and found by an all-fields search, but not listed among the
|
||||
journalists.</dd>
|
||||
|
||||
<dt><code>publication[].note</code></dt>
|
||||
<dd>Anything particular to this publication: that it showed only
|
||||
some of the pages, published them in a different order, redacted
|
||||
differently, or printed them in a book appendix.</dd>
|
||||
|
||||
<dt><code>publication[].version</code></dt>
|
||||
<dd>When this publication released a different copy from the one
|
||||
shown on the document page, the name of that copy in the
|
||||
document's <code>versions/</code> folder; its entry then links to
|
||||
“view this copy”. Without it, the entry published the
|
||||
copy shown on the page.</dd>
|
||||
|
||||
<dt><code>publication[].doc_source</code></dt>
|
||||
<dd>The web page where this publication presented the document
|
||||
(the outlet's own document page, a DocumentCloud page), linked as
|
||||
“document page”. Where a copy came from is recorded
|
||||
here, on the publication that released it, not on the
|
||||
document.</dd>
|
||||
|
||||
<dt><code>publication[].doc_url</code></dt>
|
||||
<dd>The direct link to the PDF file this publication released,
|
||||
linked as “original PDF”.</dd>
|
||||
|
||||
<dt><code>publication[].reporting_only</code></dt>
|
||||
<dd><code>true</code> when the outlet reported the story without
|
||||
publishing the document - a reporting partner. The entry keeps its
|
||||
byline and its article, but the outlet is not counted as one of
|
||||
the document's publishers and no copy is linked from it.</dd>
|
||||
</dl>
|
||||
|
||||
<h4>Other copies: <code>versions</code></h4>
|
||||
<p>Other published copies of a document (e.g. less redacted, with extra
|
||||
pages, or a different scan) are kept in its <code>versions/</code>
|
||||
folder, one sub-folder per copy, each with its own PDF, text and page
|
||||
images. The <code>versions</code> object adds details about them,
|
||||
keyed by the copy's folder name.</p>
|
||||
<dl class="field-dict">
|
||||
<dt><code>versions.{name}.changed_pages</code></dt>
|
||||
<dd>The pages of that copy, by its own page numbers, that differ
|
||||
from the copy shown on the document page. They are linked as
|
||||
“changed pages” next to the copy.</dd>
|
||||
|
||||
<dt><code>versions.{name}.pdf_sha256</code></dt>
|
||||
<dd>The SHA-256 of that copy's PDF.</dd>
|
||||
</dl>
|
||||
</details>
|
||||
</section>
|
||||
|
||||
<section id="download-archive">
|
||||
<h2>Downloading the archive</h2>
|
||||
<p>Each release comes in two packages:</p>
|
||||
<ul>
|
||||
<li><strong>full</strong>, ~18 GB: all the PDFs, page images (the
|
||||
PDF pages as JPEG files), extracted texts, and metadata,</li>
|
||||
<li><strong>pdfs</strong>, ~3.5 GB: the same without the page
|
||||
images.</li>
|
||||
</ul>
|
||||
<p>Both extract to <code>data/</code> and <code>project/</code>, ready
|
||||
to <a href="#own-copy">run your own copy</a>. Each release also
|
||||
includes the source of the server version it was published with, and
|
||||
the last six releases stay online. For only the extracted texts and
|
||||
metadata (~58 MB), see <a href="#downloading-texts">below</a>.</p>
|
||||
|
||||
<h3 id="downloading-all-data">Downloading all data</h3>
|
||||
|
||||
<h4>HTTP</h4>
|
||||
<p><a href="https://data.snowchive.org/files/">https://data.snowchive.org/files/</a></p>
|
||||
<details>
|
||||
<summary>See a curl example</summary>
|
||||
<pre>
|
||||
# download the latest release:
|
||||
rel=$(curl -fsS https://data.snowchive.org/files/latest.txt)
|
||||
base="https://data.snowchive.org/files/$rel"
|
||||
curl -f -C - -O "$base/snowchive-$rel-full.tar" # "pdfs" instead of "full" for no page images
|
||||
|
||||
# verify (optional):
|
||||
curl -fsS -O "$base/SHA256SUMS" -O "$base/SHA256SUMS.gpg"
|
||||
curl -fsS -O "https://libroot.org/public/libroot.asc"
|
||||
# "gpg --show-keys libroot.asc" should show 3B054BC2E7B2CC825FF23E94108F97B8FC5B2EBC
|
||||
gpg --import libroot.asc
|
||||
gpg --verify SHA256SUMS.gpg SHA256SUMS
|
||||
sha256sum --check --ignore-missing SHA256SUMS
|
||||
|
||||
# extract to ./snowchive-data/:
|
||||
mkdir -p snowchive-data && tar -xf "snowchive-$rel-full.tar" -C snowchive-data
|
||||
</pre>
|
||||
</details>
|
||||
|
||||
<h4>Torrent</h4>
|
||||
<p>Each release folder on
|
||||
<a href="https://data.snowchive.org/files/">https://data.snowchive.org/files/</a>
|
||||
has a <code>.torrent</code> for both packages.</p>
|
||||
|
||||
<h3 id="downloading-texts">Downloading only extracted texts and metadata</h3>
|
||||
<p>Repository for the extracted texts and metadata:
|
||||
<a href="https://git.disroot.org/libroot/snowchive-meta">https://git.disroot.org/libroot/snowchive-meta</a></p>
|
||||
<pre>
|
||||
git clone https://git.disroot.org/libroot/snowchive-meta
|
||||
cd snowchive-meta/data
|
||||
|
||||
# grep only the extracted texts:
|
||||
grep "fiber optic" -ri --include=*.md
|
||||
|
||||
# grep only the metadata:
|
||||
grep "fiber optic" -ri --include=*.json
|
||||
</pre>
|
||||
</section>
|
||||
|
||||
<section id="own-copy">
|
||||
<h2>Can I run my own copy or mirror this?</h2>
|
||||
<p>Yes. The server software behind this archive is free software with
|
||||
no database and no external dependencies, designed to run offline or,
|
||||
for example, as a Tor hidden service.</p>
|
||||
<p>First, get a <a href="#downloading-all-data">release package</a>
|
||||
and extract it: <strong>full</strong> for everything, or
|
||||
<strong>pdfs</strong> for a site that works the same but shows no page
|
||||
images.</p>
|
||||
<p>Then get the server (Go required). Each release folder has the
|
||||
source of the server version it was published with,
|
||||
<code>snowchive-server-<commit>.tar.gz</code>, covered by the same
|
||||
<code>SHA256SUMS</code> - the version known to work with that data:</p>
|
||||
<pre>
|
||||
tar -xzf snowchive-server-<commit>.tar.gz
|
||||
cd snowchive-server-<commit>
|
||||
go run . -root=/path/to/snowchive-data/ -listen=127.0.0.1:8081
|
||||
</pre>
|
||||
<p>Or clone the latest version from the
|
||||
<a href="https://git.disroot.org/libroot/snowchive-server">snowchive-server</a>
|
||||
repository:</p>
|
||||
<pre>
|
||||
git clone https://git.disroot.org/libroot/snowchive-server
|
||||
cd snowchive-server
|
||||
go run . -root=/path/to/snowchive-data/ -listen=127.0.0.1:8081
|
||||
</pre>
|
||||
<p>Recommended minimal setup for production instances:</p>
|
||||
<pre>
|
||||
go build -ldflags "-X main.buildCommit=$(git rev-parse --short HEAD)" -o snowchive .
|
||||
./snowchive -root=/path/to/snowchive-data/ -listen=127.0.0.1:8081 -base=https://archive.example.com
|
||||
</pre>
|
||||
<p>System requirements: at least 300 MB of RAM (1 GB or more
|
||||
recommended for production).</p>
|
||||
<p>The maintainers of this project do not endorse any public Snowchive
|
||||
instance other than the official snowchive.org instance (unless you
|
||||
are a reputable organization that wants to host a public
|
||||
instance).</p>
|
||||
<p>See the
|
||||
<a href="https://git.disroot.org/libroot/snowchive-server">snowchive-server README</a>
|
||||
for the full documentation.</p>
|
||||
</section>
|
||||
|
||||
<section id="contact">
|
||||
<h2>Contact</h2>
|
||||
<p>If the archive is missing a document, or some metadata is wrong,
|
||||
please submit an issue on the project repository:
|
||||
<a href="https://git.disroot.org/libroot/snowchive-meta/issues">https://git.disroot.org/libroot/snowchive-meta/issues</a></p>
|
||||
<p>You can also report issues and errors or give feedback directly to
|
||||
the project maintainers via email:
|
||||
<a href="https://libroot.org/about">https://libroot.org/about</a></p>
|
||||
<p>Please submit server-related issues to the Snowchive server
|
||||
repository:
|
||||
<a href="https://git.disroot.org/libroot/snowchive-server/issues">https://git.disroot.org/libroot/snowchive-server/issues</a></p>
|
||||
</section>
|
||||
|
||||
<section id="contributing">
|
||||
<h2>Contributing</h2>
|
||||
<p>We welcome anyone who wants to contribute to the project, for
|
||||
example by fixing errors in the metadata or in the extracted texts.
|
||||
Please see the <a href="#contact">Contact</a> section above.</p>
|
||||
<p><small>Running this project doesn't cost much, but if you wish, you
|
||||
can send a donation to:</small></p>
|
||||
<dl class="field-dict">
|
||||
<dt><small>XMR</small></dt>
|
||||
<dd><small><code style="word-break:break-all;">85gTBgYuLB4Mvo1ZCGo62126hBQQ678UeESTuSi3L9SECHXr6S7mXE28yAKvURgdJneGKqbPDcBydFfTkK7H5bvh2WXE4oT</code></small></dd>
|
||||
<dt><small>BTC</small></dt>
|
||||
<dd><small><code style="word-break:break-all;">bc1q5f5mc4pytajrn2xrp3lsftx64fw6hsdxxvs3ha</code></small></dd>
|
||||
</dl>
|
||||
</section>
|
||||
|
||||
<section id="thanks">
|
||||
<h2>Thank you</h2>
|
||||
<p>We want to thank</p>
|
||||
<ul>
|
||||
<li>all the journalists, reporters, and other people who broke the
|
||||
stories or helped bring them to light,</li>
|
||||
<li>other archives and the people behind them, such as
|
||||
<ul>
|
||||
<li>University of Toronto's and Canadian Journalists for
|
||||
Free Expression's The Snowden Surveillance Archive (shut
|
||||
down in mid-2024),</li>
|
||||
<li>Courage Foundation's Snowden Revelations (shut down in
|
||||
early 2022),</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>organizations, resources and blogs such as
|
||||
<ul>
|
||||
<li><a href="https://www.eff.org/nsa-spying/nsadocs">EFF's NSA document collection</a>,</li>
|
||||
<li><a href="https://www.aclu.org/nsa-documents-released-to-the-public-since-june-2013">ACLU's NSA document collection</a>,</li>
|
||||
<li><a href="https://christopher-parsons.com/resources/the-sigint-summaries/">Christopher Parsons' SIGINT Summaries</a>,</li>
|
||||
<li><a href="https://electrospaces.net">Electrospaces</a>,</li>
|
||||
<li><a href="https://github.com/iamcryptoki/snowden-archive">iamcryptoki/snowden-archive</a>,</li>
|
||||
<li><a href="https://cryptome.org">Cryptome</a>,</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>and of course, Edward Snowden himself.</li>
|
||||
</ul>
|
||||
</section>
|
||||
|
||||
</div>
|
||||
{{ end }}
|
||||
{{ template "base.html" . }}
|
||||
25
templates/article.html
Normal file
25
templates/article.html
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
{{ define "title" }}{{ with .Article.Title }}{{ . }}{{ else }}{{ if .Article.Book }}Book{{ else }}Article{{ end }} at {{ .Article.Host }}{{ end }} - Snowden Archive{{ end }}
|
||||
{{ define "content" }}
|
||||
<header class="list-header article-header">
|
||||
<p class="muted article-kicker">{{ if .Article.Book }}Book{{ else }}Article{{ end }}</p>
|
||||
<h1>{{ if .Article.Title }}{{ .Article.Title }}{{ else }}<span class="muted">[untitled]</span> {{ .Article.Host }}{{ end }}</h1>
|
||||
<p class="article-byline muted">
|
||||
{{ with .Article.Publisher }}{{ if $.PublisherURL }}<a href="{{ $.PublisherURL }}" class="meta-link-soft">{{ . }}</a>{{ else }}{{ . }}{{ end }}{{ end }}
|
||||
{{ with .Article.Date }} · {{ . }}{{ end }}
|
||||
{{ if .Article.Journalists }} ·
|
||||
{{ facetLinks (dict "Vals" .Article.Journalists "Cat" "journalist" "Class" "meta-link-soft" "QS" .BrowseQS) }}
|
||||
{{ end }}
|
||||
</p>
|
||||
<p>
|
||||
<a href="{{ .Article.Link }}" target="_blank" rel="noopener" class="doc-top-btn doc-pdf-link">{{ if .Article.Book }}About the book at{{ else }}Read the article at{{ end }} {{ .Article.Host }} ↗</a>
|
||||
</p>
|
||||
</header>
|
||||
|
||||
<h2>Documents published with this {{ if .Article.Book }}book{{ else }}article{{ end }} <small class="count-note">{{ .Total }}</small></h2>
|
||||
{{ if .Cards.Docs }}
|
||||
{{ template "doc-cards" .Cards }}
|
||||
{{ else }}
|
||||
<p class="muted">No documents found for this {{ if .Article.Book }}book{{ else }}article{{ end }}.</p>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
{{ template "base.html" . }}
|
||||
39
templates/articles.html
Normal file
39
templates/articles.html
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
{{ define "title" }}Articles - Snowden Archive{{ end }}
|
||||
{{ define "content" }}
|
||||
<header class="list-header">
|
||||
<h1>Articles
|
||||
<small class="count-note">{{ .Total }} articles{{ with .Books }} and {{ . }} book{{ if ne . 1 }}s{{ end }}{{ end }}</small>
|
||||
</h1>
|
||||
</header>
|
||||
<p class="muted tab-desc">
|
||||
The news stories, and the few books, these documents were published with, newest first.
|
||||
A document covered by several stories appears under each.
|
||||
</p>
|
||||
|
||||
{{ if .Groups }}
|
||||
{{ if .GroupsHTML }}{{ .GroupsHTML }}{{ else }}{{ template "article-groups" . }}{{ end }}{{ else }}
|
||||
<p class="muted">No articles.</p>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
{{ template "base.html" . }}
|
||||
|
||||
{{ define "article-groups" }}{{ range .Groups }}
|
||||
<section class="article-year">
|
||||
<h2>{{ .Year }} <small class="count-note">{{ len .Articles }}</small></h2>
|
||||
<ul class="article-list">
|
||||
{{ range .Articles }}
|
||||
<li class="article-item">
|
||||
<span class="article-date faint">{{ if .Date }}{{ .Date }}{{ else }}-{{ end }}</span>
|
||||
<span class="article-main">
|
||||
<a href="/article/{{ .Key }}" class="article-title">{{ if .Title }}{{ .Title }}{{ else }}<span class="muted">[untitled]</span> {{ .Host }}{{ end }}</a>
|
||||
<span class="article-meta muted">
|
||||
{{ if .Book }}<span class="article-kind">Book</span> · {{ end }}{{ with .Publisher }}{{ . }}{{ end }}{{ if and .Publisher .Journalists }} · {{ end }}{{ range $i, $j := .Journalists }}{{ if $i }}, {{ end }}{{ $j }}{{ end }}
|
||||
· {{ len .DocIDs }} doc{{ if ne (len .DocIDs) 1 }}s{{ end }}
|
||||
</span>
|
||||
</span>
|
||||
</li>
|
||||
{{ end }}
|
||||
</ul>
|
||||
</section>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
32
templates/assemblies.html
Normal file
32
templates/assemblies.html
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
{{ define "title" }}Assemblies - Snowden Archive{{ end }}
|
||||
{{ define "content" }}
|
||||
<header class="list-header">
|
||||
<p class="crumb"><a href="/explore">← Explore</a></p>
|
||||
<h1>Assemblies
|
||||
<small class="count-note">{{ len .Assemblies }}</small>
|
||||
</h1>
|
||||
</header>
|
||||
<p class="muted tab-desc">
|
||||
Sets of pages this archive put together from documents and copies published
|
||||
separately, so they can be read in one place. An assembly is our own
|
||||
arrangement, not a document that was leaked or published in this form.
|
||||
</p>
|
||||
|
||||
{{ if .Assemblies }}
|
||||
<ul class="assembly-list">
|
||||
{{ range .Assemblies }}
|
||||
<li>
|
||||
{{ if not $.TextOnly }}<a href="/explore/assemblies/{{ .ID }}" class="assembly-thumb"><img src="{{ (index .Slides 0).Image }}"{{ (index .Slides 0).Size.Attr }} alt="" loading="lazy"></a>{{ end }}
|
||||
<div>
|
||||
<a href="/explore/assemblies/{{ .ID }}"><strong>{{ .Title }}</strong></a>
|
||||
<div class="muted">{{ len .Slides }} slide{{ if ne (len .Slides) 1 }}s{{ end }} from {{ .Copies }} cop{{ if eq .Copies 1 }}y{{ else }}ies{{ end }}</div>
|
||||
{{ with .Note }}<p class="assembly-note">{{ . }}</p>{{ end }}
|
||||
</div>
|
||||
</li>
|
||||
{{ end }}
|
||||
</ul>
|
||||
{{ else }}
|
||||
<p class="muted">No assemblies yet.</p>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
{{ template "base.html" . }}
|
||||
60
templates/assembly.html
Normal file
60
templates/assembly.html
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
{{ define "title" }}{{ .Assembly.Title }} - Snowden Archive{{ end }}
|
||||
|
||||
{{ define "assembly-credit" }}{{ with .Source }}{{ . }} · {{ end }}<a href="{{ .Path }}">{{ .DocTitle }}</a>{{ with .Version }} (copy: {{ . }}){{ end }}, p. {{ .Page }}{{ end }}
|
||||
|
||||
{{ define "content" }}
|
||||
{{ $a := .Assembly }}{{ $n := len $a.Slides }}
|
||||
<header class="list-header">
|
||||
<p class="crumb"><a href="/explore/assemblies">← Assemblies</a></p>
|
||||
<h1>{{ $a.Title }} <small class="count-note">{{ $n }} slide{{ if ne $n 1 }}s{{ end }}</small></h1>
|
||||
<p class="muted">Put together by this archive from {{ $a.Copies }} separately published cop{{ if eq $a.Copies 1 }}y{{ else }}ies{{ end }}. This is our arrangement, not a document in its own right.</p>
|
||||
{{ with $a.Note }}<p class="assembly-note">{{ . }}</p>{{ end }}
|
||||
</header>
|
||||
|
||||
{{ if .TextOnly }}
|
||||
<p class="muted textonly-note">Text-only mode is on - page images are not loaded. <a href="/preferences">Preferences</a></p>
|
||||
<ul class="page-link-list">
|
||||
{{ range $a.Slides }}
|
||||
<li><a href="{{ .Image }}" target="_blank" rel="noopener">Slide {{ .Pos }}</a> <span class="muted">{{ template "assembly-credit" . }}</span>
|
||||
{{- with .Note }} <span class="muted">- {{ . }}</span>{{ end }}
|
||||
{{- range .Alts }}<br><a href="{{ .Image }}" target="_blank" rel="noopener">Other copy</a> <span class="muted">{{ template "assembly-credit" . }}</span>{{ end }}
|
||||
</li>
|
||||
{{ end }}
|
||||
</ul>
|
||||
{{ else }}
|
||||
<ul class="page-sheet assembly-sheet">
|
||||
{{ range $s := $a.Slides }}
|
||||
<li>
|
||||
<a href="#lb-{{ $s.Pos }}" title="Open slide {{ $s.Pos }} full screen">
|
||||
<img src="{{ $s.Image }}"{{ $s.Size.Attr }} alt="Slide {{ $s.Pos }}" loading="lazy">
|
||||
<span class="page-num faint">{{ $s.Pos }}</span>
|
||||
</a>
|
||||
<p class="assembly-cap">{{ template "assembly-credit" $s }}
|
||||
{{- range $k, $alt := $s.Alts }}<br><a href="#lb-a{{ $s.Pos }}-{{ add $k 1 }}">Other copy{{ with $alt.Source }}: {{ . }}{{ end }}</a>{{ end }}
|
||||
</p>
|
||||
</li>
|
||||
{{ end }}
|
||||
</ul>
|
||||
|
||||
{{ range $i, $s := $a.Slides }}
|
||||
<div class="lightbox" id="lb-{{ $s.Pos }}" aria-label="Slide {{ $s.Pos }} of {{ $n }}">
|
||||
<a class="lb-backdrop" href="#lb-closed" tabindex="-1" aria-hidden="true"></a>
|
||||
<a class="lb-close" href="#lb-closed" title="Close">×</a>
|
||||
{{ if $i }}<a class="lb-prev" href="#lb-{{ $i }}" title="Previous slide">‹</a>{{ end }}
|
||||
{{ if lt $s.Pos $n }}<a class="lb-next" href="#lb-{{ add $s.Pos 1 }}" title="Next slide">›</a>{{ end }}
|
||||
<img src="{{ $s.Image }}"{{ $s.Size.Attr }} alt="Slide {{ $s.Pos }} of {{ $n }}" loading="lazy" decoding="async">
|
||||
<span class="lb-counter lb-caption">{{ $s.Pos }} / {{ $n }} · {{ template "assembly-credit" $s }}{{ with $s.Note }} · {{ . }}{{ end }}{{ range $k, $alt := $s.Alts }} · <a href="#lb-a{{ $s.Pos }}-{{ add $k 1 }}">Other copy{{ with $alt.Source }}: {{ . }}{{ end }}</a>{{ end }}</span>
|
||||
</div>
|
||||
{{ range $k, $alt := $s.Alts }}
|
||||
<div class="lightbox lb-alt" id="lb-a{{ $s.Pos }}-{{ add $k 1 }}" aria-label="Other copy of slide {{ $s.Pos }}">
|
||||
<a class="lb-backdrop" href="#lb-closed" tabindex="-1" aria-hidden="true"></a>
|
||||
<a class="lb-close" href="#lb-closed" title="Close">×</a>
|
||||
<img src="{{ $alt.Image }}"{{ $alt.Size.Attr }} alt="Other copy of slide {{ $s.Pos }}" loading="lazy" decoding="async">
|
||||
<span class="lb-counter lb-caption">Other copy of slide {{ $s.Pos }} · {{ template "assembly-credit" $alt }} · <a href="#lb-{{ $s.Pos }}">Back to slide {{ $s.Pos }}</a></span>
|
||||
</div>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
|
||||
{{ template "base.html" . }}
|
||||
35
templates/base.html
Normal file
35
templates/base.html
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en"{{ if .Theme }} data-theme="{{ .Theme }}"{{ end }}>
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>{{ block "title" . }}Snowden Archive{{ end }}</title>
|
||||
<link rel="stylesheet" href="{{ asset "style.css" }}">
|
||||
<script src="{{ asset "main.js" }}" defer></script>
|
||||
{{ block "head" . }}{{ end }}
|
||||
</head>
|
||||
<body>
|
||||
<a class="skip-link" href="#main">Skip to main content</a>
|
||||
<nav class="site-nav">
|
||||
<a href="/" class="{{ if eq .Nav "home" }}active{{ end }}">Home</a>
|
||||
<a href="/browse" class="{{ if eq .Nav "browse" }}active{{ end }}">Browse</a>
|
||||
<a href="/search" class="{{ if eq .Nav "search" }}active{{ end }}">Search</a>
|
||||
<a href="/articles" class="{{ if eq .Nav "articles" }}active{{ end }}">Articles</a>
|
||||
<a href="/glossary" class="{{ if eq .Nav "glossary" }}active{{ end }}">Glossary</a>
|
||||
<a href="/random">Random</a>
|
||||
<a href="/about" class="{{ if eq .Nav "about" }}active{{ end }}">About</a>
|
||||
<a href="/pinned" class="nav-right{{ if eq .Nav "pinned" }} active{{ end }}">Pinned</a>
|
||||
<a href="/preferences" class="{{ if eq .Nav "preferences" }}active{{ end }}">Preferences</a>
|
||||
</nav>
|
||||
{{ if .ShowCatBoxes }}{{ template "cat-boxes" . }}{{ end }}
|
||||
<main id="main" tabindex="-1">
|
||||
{{ block "content" . }}{{ end }}
|
||||
</main>
|
||||
<footer>
|
||||
{{ $b := buildInfo }}
|
||||
<p>Snowden Archive · <a href="/changelog">changelog</a>
|
||||
<span class="build-info faint">· rev {{ $b.Commit }}</span>
|
||||
</p>
|
||||
</footer>
|
||||
</body>
|
||||
</html>
|
||||
155
templates/browse.html
Normal file
155
templates/browse.html
Normal file
|
|
@ -0,0 +1,155 @@
|
|||
{{ define "title" }}{{ if eq .Collection "foia" }}FOIA{{ else if eq .Collection "sidtoday-archive" }}SIDtoday Archive{{ else }}Reported{{ end }} documents - Snowden Archive{{ end }}
|
||||
{{ define "content" }}
|
||||
|
||||
<hr class="section-rule">
|
||||
|
||||
<nav class="collection-tabs" aria-label="Collections">
|
||||
<a href="/browse"
|
||||
class="collection-tab {{ if eq .Collection "news" }}active-tab{{ end }}"
|
||||
{{ if eq .Collection "news" }}aria-current="page"{{ end }}>
|
||||
Reported <span class="tab-count">{{ index .CollectionCounts "news" }}</span>
|
||||
</a>
|
||||
<a href="/browse?collection=sidtoday-archive"
|
||||
class="collection-tab {{ if eq .Collection "sidtoday-archive" }}active-tab{{ end }}"
|
||||
{{ if eq .Collection "sidtoday-archive" }}aria-current="page"{{ end }}>
|
||||
SIDtoday Archive <span class="tab-count">{{ .SIDTodayCount }}</span>
|
||||
</a>
|
||||
{{ if or .Prefs.IncludeFOIA (eq .Collection "foia") }}
|
||||
<a href="/browse?collection=foia"
|
||||
class="collection-tab {{ if eq .Collection "foia" }}active-tab{{ end }}"
|
||||
{{ if eq .Collection "foia" }}aria-current="page"{{ end }}>
|
||||
FOIA <span class="tab-count">{{ index .CollectionCounts "foia" }}</span>
|
||||
</a>
|
||||
{{ end }}
|
||||
</nav>
|
||||
|
||||
{{ if eq .Collection "sidtoday-archive" }}
|
||||
|
||||
<header class="tab-header">
|
||||
<h1 class="tab-title">SIDtoday Archive <small>({{ .Total }} issues)</small></h1>
|
||||
<p class="tab-desc">
|
||||
All NSA SIDtoday newsletters in chronological order. This section
|
||||
includes The Intercept's bulk release plus issues that were individually
|
||||
reported by media (marked <span class="issue-reported">reported</span>; those
|
||||
also appear in the <a href="/browse">Reported</a> collection).
|
||||
</p>
|
||||
{{ with .SeriesList }}
|
||||
<details class="series-index">
|
||||
<summary>Series <span class="count">{{ len . }}</span> <span class="muted pref-note">- recurring columns and multi-part runs</span></summary>
|
||||
<ul class="series-index-list">
|
||||
{{ range . }}<li><a href="{{ .URL }}">{{ .Name }}</a> <span class="count">{{ .Count }}</span></li>{{ end }}
|
||||
</ul>
|
||||
<p class="muted pref-note"><a href="/browse/series">Full series table</a> - sortable, with per-collection counts.</p>
|
||||
</details>
|
||||
{{ end }}
|
||||
<div class="browse-controls{{ if eq .View "table" }} browse-controls-spaced{{ end }}">
|
||||
<div class="view-toggle">
|
||||
<a href="/browse?collection=sidtoday-archive" class="{{ if and (ne .View "card") (ne .View "table") }}active-toggle{{ end }}">Timeline</a>
|
||||
<a href="/browse?collection=sidtoday-archive&view=card" class="{{ if eq .View "card" }}active-toggle{{ end }}">Card View</a>
|
||||
<a href="/browse?collection=sidtoday-archive&view=table" class="{{ if eq .View "table" }}active-toggle{{ end }}">Table View</a>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
{{ if eq .View "card" }}
|
||||
<div class="sort-bar-spaced">{{ template "sort-bar" (dict "SortURL" .BaseURL "SortBy" .Sort "Order" .Order "ShowRelevance" false "DefaultSort" .DefaultSort) }}</div>
|
||||
<h2 class="sr-only">Documents</h2>
|
||||
{{ template "doc-cards" .Cards }}
|
||||
{{ template "pagination" . }}
|
||||
{{ else if eq .View "table" }}
|
||||
{{ template "doc-table" . }}
|
||||
{{ else }}
|
||||
{{ if .ArchiveYears }}
|
||||
{{ if .TimelineHTML }}{{ .TimelineHTML }}{{ else }}{{ template "archive-timeline" . }}{{ end }}{{ else }}
|
||||
<p class="muted">No documents in this collection yet.</p>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
|
||||
{{ else }}
|
||||
<header class="browse-header">
|
||||
<h1 class="browse-title">
|
||||
{{ if eq .Collection "foia" }}FOIA Documents{{ else }}Reported Documents{{ end }}
|
||||
<small class="count-note">({{ .Total }} total)</small>
|
||||
</h1>
|
||||
|
||||
{{ $cardURL := .SortURL }}
|
||||
{{ $tableURL := print .SortURL "view=table" }}
|
||||
{{ if .Sort }}
|
||||
{{ $cardURL = print $cardURL "sort=" .Sort }}
|
||||
{{ $tableURL = print $tableURL "&sort=" .Sort }}
|
||||
{{ if .Order }}
|
||||
{{ $cardURL = print $cardURL "&order=" .Order }}
|
||||
{{ $tableURL = print $tableURL "&order=" .Order }}
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
<div class="browse-controls">
|
||||
<div class="view-toggle">
|
||||
<a href="{{ $cardURL }}" class="{{ if ne .View "table" }}active-toggle{{ end }}">Card View</a>
|
||||
<a href="{{ $tableURL }}" class="{{ if eq .View "table" }}active-toggle{{ end }}">Table View</a>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<p class="tab-desc collection-desc">
|
||||
{{ if eq .Collection "foia" }}
|
||||
Documents released by government agencies under Freedom of Information Act
|
||||
requests, rather than through the Snowden disclosures - obtained by
|
||||
journalists, researchers, and transparency organizations and added here
|
||||
for context.
|
||||
{{ else }}
|
||||
Documents published alongside news reporting on the Snowden disclosures.
|
||||
{{ end }}
|
||||
</p>
|
||||
|
||||
{{ if ne .View "table" }}
|
||||
{{ template "sort-bar" (dict "SortURL" .SortURL "SortBy" .Sort "Order" .Order "ShowRelevance" false "DefaultSort" .DefaultSort) }}
|
||||
{{ end }}
|
||||
|
||||
{{ if eq .Collection "news" }}
|
||||
{{ template "type-filter" (dict
|
||||
"Action" "/browse"
|
||||
"Params" .Params
|
||||
"AllTypes" .AllTypes
|
||||
"SelectedTypes" .SelectedTypes
|
||||
) }}
|
||||
{{ end }}
|
||||
|
||||
{{ if eq .View "table" }}
|
||||
{{ template "doc-table" . }}
|
||||
{{ else }}
|
||||
<h2 class="sr-only">Documents</h2>
|
||||
{{ template "doc-cards" .Cards }}
|
||||
{{ template "pagination" . }}
|
||||
{{ end }}
|
||||
|
||||
{{ end }}
|
||||
|
||||
{{ end }}
|
||||
{{ template "base.html" . }}
|
||||
|
||||
{{ define "archive-timeline" }}<nav class="year-nav" aria-label="Jump to year">
|
||||
<span class="year-nav-label">Jump to year:</span>
|
||||
{{ range .ArchiveYears }}<a href="#y{{ .Year }}">{{ .Year }}</a>{{ end }}
|
||||
</nav>
|
||||
|
||||
{{ range .ArchiveYears }}
|
||||
<section class="archive-year" id="y{{ .Year }}">
|
||||
<h2>{{ .Year }} <small>({{ .Count }} issue{{ if ne .Count 1 }}s{{ end }})</small></h2>
|
||||
{{ range .Months }}
|
||||
<div class="archive-month">
|
||||
{{ with .Name }}<h3>{{ . }}</h3>{{ end }}
|
||||
<ul class="archive-issues">
|
||||
{{ range .Docs }}
|
||||
<li>
|
||||
<span class="issue-date">{{ with .DocMeta.DateCreated }}{{ . }}{{ else }}-{{ end }}</span>
|
||||
<a href="/doc/{{ .ID }}">{{ .Title }}</a>
|
||||
<a href="/files/{{ .ID }}/{{ pathEscape .PDFFile }}" target="_blank" rel="noopener" class="doc-pdf" title="Open PDF">[pdf]</a>
|
||||
{{ if .ReportedIssue }}<span class="issue-reported" title="Individually reported - also in the Reported collection">reported</span>{{ end }}
|
||||
</li>
|
||||
{{ end }}
|
||||
</ul>
|
||||
</div>
|
||||
{{ end }}
|
||||
</section>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
59
templates/category.html
Normal file
59
templates/category.html
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
{{ define "title" }}Browse by {{ .Title }} - Snowden Archive{{ end }}
|
||||
{{ define "content" }}
|
||||
|
||||
<h1>Browse by {{ .Title }}</h1>
|
||||
|
||||
{{ with .Blurb }}<p class="tab-desc cat-blurb">{{ . }}</p>{{ end }}
|
||||
|
||||
{{ if .SeeAlso }}
|
||||
<p class="muted see-also">See also: {{ range $i, $l := .SeeAlso }}{{ if $i }} · {{ end }}<a href="{{ $l.URL }}">{{ $l.Text }}</a>{{ end }}.</p>
|
||||
{{ end }}
|
||||
|
||||
{{ if .ShowAZ }}
|
||||
<nav class="az-nav" aria-label="Jump to letter">
|
||||
<span class="az-label">Jump to:</span>
|
||||
{{ range .Letters }}<a href="#{{ azID . }}">{{ . }}</a>{{ end }}
|
||||
</nav>
|
||||
{{ end }}
|
||||
|
||||
{{ if .CollCols }}
|
||||
<input type="checkbox" id="coll-cols" class="coll-cols-checkbox"{{ if .ShowCollCols }} checked{{ end }}>
|
||||
<label for="coll-cols" class="coll-cols-label">
|
||||
<span class="collapsed-text">[+] Show per-type counts</span>
|
||||
<span class="expanded-text">[−] Hide per-type counts</span>
|
||||
<span class="sr-only">Toggle the per-type count columns</span>
|
||||
</label>
|
||||
{{ end }}
|
||||
<div class="table-scroll">
|
||||
<table class="cat-table cat-table-colls">
|
||||
<thead>
|
||||
<tr>{{ range .Heads }}{{ template "sort-th" . }}{{ end }}</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{{ if .RowsHTML }}{{ .RowsHTML }}{{ else }}{{ template "category-rows" . }}{{ end }} </tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
{{ end }}
|
||||
{{ template "base.html" . }}
|
||||
|
||||
{{ define "category-row" }}
|
||||
<tr>
|
||||
<td>
|
||||
<a href="{{ .URL }}">
|
||||
{{ .Name }}
|
||||
</a>
|
||||
{{- with .Note }} <small class="muted cat-note">{{ . }}</small>{{ end }}
|
||||
</td>
|
||||
<td class="num-col">{{ .Count }}</td>
|
||||
<td class="num-col coll-col">{{ if .News }}{{ .News }}{{ else }}<span class="faint">-</span>{{ end }}</td>
|
||||
<td class="num-col coll-col">{{ if .Archive }}{{ .Archive }}{{ else }}<span class="faint">-</span>{{ end }}</td>
|
||||
{{- if not .NoFOIA }}
|
||||
<td class="num-col coll-col">{{ if .FOIA }}{{ .FOIA }}{{ else }}<span class="faint">-</span>{{ end }}</td>
|
||||
{{- end }}
|
||||
</tr>
|
||||
{{ end }}
|
||||
{{ define "category-az-row" }}
|
||||
<tr class="az-row"><th id="{{ azID . }}" colspan="5">{{ . }}</th></tr>
|
||||
{{ end }}
|
||||
{{ define "category-rows" }}{{ $prevL := "" }}{{ range .Entries }}{{ if $.ShowAZ }}{{ if ne .FirstLetter $prevL }}{{ template "category-az-row" .FirstLetter }}{{ $prevL = .FirstLetter }}{{ end }}{{ end }}{{ template "category-row" . }}{{ end }}{{ end }}
|
||||
25
templates/changelog.html
Normal file
25
templates/changelog.html
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
{{ define "title" }}Changelog - Snowden Archive{{ end }}
|
||||
{{ define "content" }}
|
||||
<h1>Changelog
|
||||
{{ if .Total }}<small class="count-note">{{ .Total }} entries</small>{{ end }}
|
||||
</h1>
|
||||
|
||||
{{ if .Entries }}
|
||||
<ul class="changelog-list">
|
||||
{{ range .Entries }}
|
||||
<li class="changelog-entry">
|
||||
<span class="changelog-date faint">{{ if ge (len .Date) 10 }}{{ slice .Date 0 10 }}{{ else }}{{ .Date }}{{ end }}</span>
|
||||
<span class="changelog-type changelog-type-{{ .Type }}">{{ .Type }}</span>
|
||||
{{ if and .ID (ne .Type "removed") }}
|
||||
<a href="{{ .DocURL }}">{{ .Headline }}</a>
|
||||
{{ else }}
|
||||
{{ .Headline }}
|
||||
{{ end }}
|
||||
</li>
|
||||
{{ end }}
|
||||
</ul>
|
||||
{{ else }}
|
||||
<p class="muted">No changes recorded.</p>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
{{ template "base.html" . }}
|
||||
50
templates/classpatterns.html
Normal file
50
templates/classpatterns.html
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
{{ define "title" }}Classification patterns - Snowden Archive{{ end }}
|
||||
|
||||
{{ define "content" }}
|
||||
<header class="list-header">
|
||||
<p class="crumb"><a href="/explore">← Explore</a></p>
|
||||
<h1>Classification patterns</h1>
|
||||
</header>
|
||||
|
||||
<p class="muted">Two views of the marking metadata. Every number counts the
|
||||
documents carrying that exact classification component. A document carrying
|
||||
several markings counts once per marking.</p>
|
||||
|
||||
<h2>Mix by document year</h2>
|
||||
<div class="cls-legend">
|
||||
{{ range $i, $v := .Levels }}<span class="cls-key"><span class="cls-swatch seg-{{ $i }}"></span>{{ $v }}</span>{{ end }}
|
||||
</div>
|
||||
{{ range .Strips }}
|
||||
<div class="cls-row">
|
||||
<a class="cls-year" href="{{ .URL }}">{{ .Year }}</a>
|
||||
<span class="cls-strip">
|
||||
{{ range .Segs }}
|
||||
<a class="cls-seg {{ .Class }}" href="{{ .URL }}" style="width: {{ .Pct }}%" title="{{ .Value }}: {{ .Count }}"></a>
|
||||
{{ end }}
|
||||
</span>
|
||||
<span class="cls-total faint">{{ .Total }}</span>
|
||||
</div>
|
||||
{{ end }}
|
||||
{{ with .Sparse }}<p class="muted"><small>Not shown, with fewer than {{ $.MinMarks }} markings each:
|
||||
{{ range $i, $y := . }}{{ if $i }}, {{ end }}<a href="{{ $y.URL }}">{{ $y.Year }}</a> ({{ $y.Total }}){{ end }}.</small></p>{{ end }}
|
||||
<h2>Agency × classification</h2>
|
||||
<div class="table-scroll">
|
||||
<table class="data-table heat-table">
|
||||
<thead>
|
||||
<tr><th>Agency</th>{{ range .Cols }}<th><a href="{{ .URL }}">{{ .Name }}</a></th>{{ end }}</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{{ range .HeatRows }}
|
||||
<tr>
|
||||
<td><a href="{{ .URL }}">{{ .Name }}</a></td>
|
||||
{{ range .Cells }}
|
||||
<td class="heat-cell">{{ if .URL }}<a href="{{ .URL }}">{{ .Count }}</a>{{ else }}<span class="faint">-</span>{{ end }}</td>
|
||||
{{ end }}
|
||||
</tr>
|
||||
{{ end }}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{{ end }}
|
||||
|
||||
{{ template "base.html" . }}
|
||||
38
templates/codewords.html
Normal file
38
templates/codewords.html
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
{{ define "title" }}Acronyms & codewords - Snowden Archive{{ end }}
|
||||
|
||||
{{ define "content" }}
|
||||
<header class="list-header">
|
||||
<p class="crumb"><a href="/explore">← Explore</a></p>
|
||||
<h1>Acronyms & codewords</h1>
|
||||
</header>
|
||||
|
||||
<p class="muted">ALL-CAPS tokens (3+ letters) extracted automatically from titles and
|
||||
transcripts - acronyms, program names, compartments, org designators. This is a
|
||||
text statistic, not curated metadata. A token links to a search for the word in any
|
||||
letter case, across the metadata too, so the search can find more documents than the
|
||||
capitals counted here.</p>
|
||||
|
||||
<div class="table-scroll">
|
||||
<table class="data-table codeword-table">
|
||||
<thead>
|
||||
<tr>{{ range .Heads }}{{ template "sort-th" . }}{{ end }}</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{{ if .RowsHTML }}{{ .RowsHTML }}{{ else }}{{ template "codeword-rows" . }}{{ end }} </tbody>
|
||||
</table>
|
||||
</div>
|
||||
{{ end }}
|
||||
|
||||
{{ template "base.html" . }}
|
||||
|
||||
{{ define "codeword-rows" }} {{ range .Rows }}
|
||||
<tr>
|
||||
<td><a href="{{ .SearchURL }}">{{ .Token }}</a>{{ if .ProgURL }} <a class="faint cw-prog" href="{{ .ProgURL }}" title="Also a curated program - its page tracks metadata mentions, including descriptions">program →</a>{{ end }}</td>
|
||||
<td>{{ .Count }}</td>
|
||||
<td>{{ .Docs }}</td>
|
||||
<td>{{ if .First }}<a href="{{ .FirstURL }}">{{ .First }}</a>{{ else }}<span class="faint">-</span>{{ end }}</td>
|
||||
<td>{{ if .Last }}<a href="{{ .LastURL }}">{{ .Last }}</a>{{ else }}<span class="faint">-</span>{{ end }}</td>
|
||||
<td>{{ if .Gloss }}<a href="{{ .GlossURL }}">defined</a>{{ else }}<span class="faint">-</span>{{ end }}</td>
|
||||
</tr>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
272
templates/doc.html
Normal file
272
templates/doc.html
Normal file
|
|
@ -0,0 +1,272 @@
|
|||
{{ define "title" }}{{ .Doc.Title }} - Snowden Archive{{ end }}
|
||||
|
||||
{{ define "head" }}
|
||||
<meta name="description" content="{{ .Doc.Description }}">
|
||||
<link rel="schema.DC" href="http://purl.org/dc/elements/1.1/">
|
||||
<meta name="DC.title" content="{{ .Doc.Title }}">
|
||||
{{ if .Doc.DocMeta.Creator }}<meta name="DC.creator" content="{{ .Doc.DocMeta.Creator }}">{{ end }}
|
||||
{{ if .Doc.DocMeta.DateCreated }}<meta name="DC.date" content="{{ .Doc.DocMeta.DateCreated }}">{{ end }}
|
||||
{{ if .Doc.Description }}<meta name="DC.description" content="{{ .Doc.Description }}">{{ end }}
|
||||
{{ range .Doc.DocMeta.Programs }}<meta name="DC.subject" content="{{ . }}">{{ end }}
|
||||
{{ range .Doc.Tags }}<meta name="DC.subject" content="{{ . }}">{{ end }}
|
||||
<meta name="DC.type" content="Text">
|
||||
<meta name="DC.format" content="application/pdf">
|
||||
<meta name="DC.identifier" content="{{ .DocAbsURL }}">
|
||||
<meta name="DC.language" content="en">
|
||||
<meta name="DC.publisher" content="Snowden Archive">
|
||||
<meta property="og:type" content="article">
|
||||
<meta property="og:site_name" content="Snowden Archive">
|
||||
<meta property="og:title" content="{{ .Doc.Title }}">
|
||||
{{ if .Doc.Description }}<meta property="og:description" content="{{ .Doc.Description }}">{{ end }}
|
||||
<meta property="og:url" content="{{ .DocAbsURL }}">
|
||||
{{ with .OGImageURL }}<meta property="og:image" content="{{ . }}">{{ end }}
|
||||
<meta name="twitter:card" content="summary">
|
||||
{{ if .APIEnabled }}<link rel="alternate" type="application/json" href="/api/doc/{{ .Doc.ID }}">{{ end }}
|
||||
{{ .Embeds.JSONLDTag }}
|
||||
{{ end }}
|
||||
|
||||
{{ define "content" }}
|
||||
|
||||
{{ $sidebar := .Prefs.InfoSidebar }}
|
||||
<div class="doc-layout{{ if $sidebar }} info-sidebar{{ end }}">
|
||||
{{ if $sidebar }}
|
||||
<input type="checkbox" id="info-collapse" class="info-collapse-checkbox">
|
||||
{{ end }}
|
||||
|
||||
<header class="doc-header">
|
||||
<div class="header-main">
|
||||
<div>
|
||||
<h1>{{ .Doc.Title }}</h1>
|
||||
<a href="/files/{{ .Doc.ID }}/{{ pathEscape .Doc.PDFFile }}" target="_blank" rel="noopener"
|
||||
class="doc-top-btn doc-pdf-link" title="Open original PDF">Open PDF</a>
|
||||
{{ with .Doc.Versions }}<a href="#publications" class="doc-top-btn" title="Other published copies of this document: less redacted, extra pages or a different scan">+{{ len . }} other cop{{ if eq (len .) 1 }}y{{ else }}ies{{ end }}</a>{{ end }}
|
||||
<form method="POST" action="/pin/{{ .Doc.ID }}" class="inline-form">
|
||||
{{ $back := print "/doc/" .Doc.ID }}
|
||||
<input type="hidden" name="back" value="{{ $back }}">
|
||||
<button type="submit" class="doc-top-btn pin-btn">{{ if .Pinned }}★ Unpin{{ else }}☆ Pin{{ end }}</button>
|
||||
</form>
|
||||
{{ if or .SerialPrev .SerialNext }}
|
||||
<nav class="serial-nav" aria-label="SIDtoday issues">
|
||||
{{ with .SerialPrev }}<a href="/doc/{{ .ID }}" class="serial-prev" title="{{ .Title }}">← Previous issue{{ with .DocMeta.DateCreated }} ({{ . }}){{ end }}</a>{{ end }}
|
||||
{{ with .SerialNext }}<a href="/doc/{{ .ID }}" class="serial-next" title="{{ .Title }}">Next issue{{ with .DocMeta.DateCreated }} ({{ . }}){{ end }} →</a>{{ end }}
|
||||
</nav>
|
||||
{{ end }}
|
||||
|
||||
<div class="doc-info">
|
||||
<strong>Creating agency:</strong> {{ with .Doc.DocMeta.Creator }}<a href="/browse/agency/{{ pathEscape . }}{{ $.BrowseQS }}">{{ . }}</a>{{ with agencyName . }} <small class="muted">{{ . }}</small>{{ end }}{{ else }}-{{ end }}<br>
|
||||
<strong>Document date:</strong> {{ with .Doc.DocMeta.DateCreated }}{{ . }}{{ else }}-{{ end }}<br>
|
||||
{{ if or (eq .Doc.Type "FOIA") (eq .Doc.Type "SIDTODAY") }}<strong>Type:</strong> {{ template "type-badge" .Doc.Type }}<br>{{ end }}
|
||||
{{ if .Doc.Pages }}<strong>Pages:</strong> <a href="/doc/{{ .Doc.ID }}/pages">{{ .Doc.Pages }}</a><br>{{ end }}
|
||||
{{ with .Doc.DocMeta.Genre }}<strong>Genre:</strong> <a href="/browse/genre/{{ pathEscape . }}{{ $.BrowseQS }}">{{ . }}</a><br>{{ end }}
|
||||
{{ with .Doc.DocMeta.Banner }}<strong>Banner:</strong> <a href="/browse/banner/{{ pathEscape . }}{{ $.BrowseQS }}" class="doc-banner-link banner">{{ . }}</a><br>{{ end }}
|
||||
<strong>Classification: </strong>
|
||||
{{- facetLinks (dict "Vals" .Doc.DocMeta.Classification "Cat" "classification" "Sep" "space" "Class" "tag classification" "QS" $.BrowseQS) -}}
|
||||
<br>
|
||||
|
||||
<strong>Dissemination: </strong>
|
||||
{{- facetLinks (dict "Vals" .Doc.DocMeta.Dissemination "Cat" "dissemination" "Class" "doc-banner-link dissemination" "QS" $.BrowseQS) -}}
|
||||
</div>
|
||||
|
||||
{{ with .SeriesGroup }}
|
||||
<div class="rule-top series-box">
|
||||
<strong>Series:</strong>
|
||||
<a href="/browse/series/{{ pathEscape .Name }}{{ $.BrowseQS }}">{{ .Name }}</a>
|
||||
<ol class="series-list">
|
||||
{{ range .Docs }}
|
||||
<li{{ if gt .DocMeta.Series.Part 0 }} value="{{ .DocMeta.Series.Part }}"{{ end }}>
|
||||
{{ if eq .ID $.Doc.ID }}<strong aria-current="page">{{ .Title }}</strong>
|
||||
{{ else }}<a href="/doc/{{ .ID }}">{{ .Title }}</a>{{ end }}
|
||||
</li>
|
||||
{{ end }}
|
||||
</ol>
|
||||
</div>
|
||||
{{ end }}
|
||||
|
||||
{{ with .Assemblies }}
|
||||
<div class="rule-top series-box">
|
||||
<strong>Part of {{ if gt (len .) 1 }}assemblies{{ else }}an assembly{{ end }}:</strong>
|
||||
{{ range $i, $a := . }}{{ if $i }}, {{ end }}<a href="/explore/assemblies/{{ $a.ID }}">{{ $a.Title }}</a>{{ end }}
|
||||
</div>
|
||||
{{ end }}
|
||||
|
||||
<div class="rule-top pub-block" id="publications">
|
||||
{{- range $i, $pub := .Doc.Publication }}
|
||||
<div class="pub-entry{{ if $i }} pub-sep{{ end }}">
|
||||
{{- if or $pub.Link $pub.Publisher $pub.DatePublished }}
|
||||
<div class="pub-head">
|
||||
{{- if $pub.Link }}<a href="{{ $pub.Link }}" target="_blank" rel="noopener" class="pub-original">{{ with $pub.Publisher }}{{ . }}{{ else }}Read the original{{ end }} ↗</a>
|
||||
{{- else }}{{ with $pub.Publisher }}<span class="pub-outlet">{{ . }}</span>{{ end }}{{ end -}}
|
||||
{{- with $pub.DatePublished }}<span class="pub-date muted">{{ . }}</span>{{ end -}}
|
||||
</div>
|
||||
{{- end -}}
|
||||
{{- if $pub.Title -}}
|
||||
{{ $ak := articleKey $pub.Link }}
|
||||
<div class="pub-title">{{ if $ak }}<a href="/article/{{ $ak }}"><em>{{ $pub.Title }}</em></a>{{ else }}<em>{{ $pub.Title }}</em>{{ end }}</div>
|
||||
{{- end -}}
|
||||
{{- if or $pub.Journalists $pub.Contributors }}
|
||||
<div class="pub-people muted">
|
||||
{{- range $j, $name := $pub.Journalists -}}
|
||||
{{- if $j }} <span class="doc-dot">·</span> {{ end -}}
|
||||
<a href="/browse/journalist/{{ pathEscape $name }}{{ $.BrowseQS }}" class="meta-link">{{ $name }}</a>
|
||||
{{- end -}}
|
||||
{{- range $j, $name := $pub.Contributors -}}
|
||||
{{- if or $j $pub.Journalists }} <span class="doc-dot">·</span> {{ end -}}
|
||||
<span class="meta-link">{{ $name }}</span>
|
||||
{{- end -}}
|
||||
</div>
|
||||
{{- end -}}
|
||||
{{- if $pub.ReportingOnly }}
|
||||
<div class="pub-reporting-only muted">Reporting partner - published this story, not the document</div>
|
||||
{{- end -}}
|
||||
{{- with $pub.Note }}
|
||||
<div class="pub-note">{{ . }}</div>
|
||||
{{- end -}}
|
||||
{{- $copy := $.Doc.VersionFor $pub.Version -}}
|
||||
{{- $shown := and (not $pub.ReportingOnly) (not $pub.Version) $.Doc.Versions -}}
|
||||
{{- if and (not $pub.ReportingOnly) (or $copy $pub.DocSource $pub.DocURL $shown) }}
|
||||
<div class="pub-copy muted">
|
||||
{{- if $shown }}<span class="pub-shown">the copy shown on this page</span>{{ end -}}
|
||||
{{- with $copy }}<a href="/doc/{{ $.Doc.ID }}/versions/{{ .Slug }}">view this copy →</a>{{ if .ChangedPages }}<span class="changed-pages">changed pages: {{ template "changed-page-links" . }}</span>{{ end }}{{ end -}}
|
||||
{{- with $pub.DocSource }}<a href="{{ . }}" target="_blank" rel="noopener">document page ↗</a>{{ end -}}
|
||||
{{- with $pub.DocURL }}<a href="{{ . }}" target="_blank" rel="noopener">original PDF ↗</a>{{ end -}}
|
||||
</div>
|
||||
{{- end -}}
|
||||
</div>
|
||||
{{- end }}
|
||||
{{- with .Doc.UnlinkedVersions }}
|
||||
<div class="pub-entry pub-sep">
|
||||
<div class="pub-title">Other copies</div>
|
||||
<div class="pub-copy muted">
|
||||
{{- range . }}<span><a href="/doc/{{ $.Doc.ID }}/versions/{{ .Slug }}">{{ .Slug }} →</a>{{ if .ChangedPages }} (changed pages: {{ template "changed-page-links" . }}){{ end }}</span>{{ end -}}
|
||||
</div>
|
||||
</div>
|
||||
{{- end }}
|
||||
</div>
|
||||
|
||||
<div class="rule-top">
|
||||
<p>{{ .Doc.Description }}</p>
|
||||
|
||||
<div class="meta-expand">
|
||||
<input type="checkbox" id="meta-toggle" class="meta-checkbox"{{ if .Prefs.OpenInfo }} checked{{ end }}>
|
||||
<label for="meta-toggle" class="meta-label">
|
||||
<span class="collapsed-text">[+] Show more info</span>
|
||||
<span class="expanded-text">[-] Show less info</span>
|
||||
<span class="sr-only">Toggle extended document metadata</span>
|
||||
</label>
|
||||
|
||||
<div class="meta-content">
|
||||
<div class="meta-grid">
|
||||
<div class="meta-wide long-desc-container">
|
||||
<strong>Long Description:</strong>
|
||||
<div class="long-description-text">
|
||||
{{- with .Doc.LongDescription }}{{ . }}{{ else }}<em>No extended description available.</em>{{ end -}}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div>
|
||||
<strong>Programs:</strong><br>
|
||||
{{- facetLinks (dict "Vals" .Doc.DocMeta.Programs "Cat" "program" "Class" "tag" "AsList" .Prefs.ProgramsAsList "QS" $.BrowseQS) -}}
|
||||
{{- if not (and .Doc.DocMeta.Programs .Prefs.ProgramsAsList) }}<br>{{ end -}}
|
||||
<br>
|
||||
|
||||
<strong>People mentioned:</strong><br>
|
||||
{{- facetLinks (dict "Vals" .Doc.DocMeta.People "Cat" "people" "Class" "meta-link" "QS" $.BrowseQS) -}}
|
||||
<br><br>
|
||||
|
||||
<strong>Countries:</strong><br>
|
||||
{{- facetLinks (dict "Vals" .Doc.DocMeta.Countries "Cat" "country" "Class" "meta-link" "QS" $.BrowseQS) -}}
|
||||
<br><br>
|
||||
|
||||
<strong>Tags:</strong><br>
|
||||
{{- facetLinks (dict "Vals" .Doc.Tags "Cat" "tag" "Class" "meta-link" "AsList" .Prefs.TagsAsList "QS" $.BrowseQS) -}}
|
||||
|
||||
<br><br>
|
||||
<strong>Identifier:</strong><br>
|
||||
<code class="checksum">{{ .Doc.ID }}</code>
|
||||
{{ with .Doc.AkaID }}
|
||||
<br><strong>Also known as:</strong><br>
|
||||
{{ range $i, $aka := . }}{{ if $i }}<br>{{ end }}<code class="checksum">{{ $aka.ID }}</code>{{ with $aka.DocSource }} <a href="{{ . }}" target="_blank" rel="noopener">page ↗</a>{{ end }}{{ with $aka.DocURL }} <a href="{{ . }}" target="_blank" rel="noopener">PDF ↗</a>{{ end }}{{ end }}
|
||||
{{ end }}
|
||||
{{ with .Doc.PDFSha256 }}
|
||||
<br><br>
|
||||
<strong>PDF SHA-256:</strong><br>
|
||||
<code class="checksum">{{ . }}</code>
|
||||
{{ end }}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<details class="note-box note-subtle" id="note-box" hidden data-doc-id="{{ .Doc.ID }}" data-doc-title="{{ .Doc.Title }}">
|
||||
<summary>My notes <span class="note-status faint" id="note-status"></span></summary>
|
||||
<textarea id="note-text" rows="4" placeholder="Private notes about this document - stored only in this browser (localStorage). See all your notes on the Pinned page."></textarea>
|
||||
</details>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
{{ if $sidebar }}
|
||||
<label for="info-collapse" class="info-toggle-label" title="Show/hide document info">
|
||||
<span class="sr-only">Toggle the document info sidebar</span>
|
||||
</label>
|
||||
{{ end }}
|
||||
|
||||
<div class="doc-container">
|
||||
|
||||
{{ template "doc-viewport" . }}
|
||||
</div>
|
||||
</div>{{/* /.doc-layout */}}
|
||||
|
||||
{{ if and .Images (not .Prefs.TextOnly) }}
|
||||
{{ template "lightbox-overlays" . }}
|
||||
|
||||
{{ end }}
|
||||
|
||||
<div class="doc-extra">
|
||||
{{ with .Doc.ArchiveComment }}
|
||||
<div class="archive-note">
|
||||
<strong>Archive note:</strong> {{ . }}
|
||||
</div>
|
||||
{{ end }}
|
||||
|
||||
<span class="Z3988" title="{{ .Embeds.COinS }}"></span>
|
||||
</div>
|
||||
|
||||
{{ if or .PublishedTogether .RelatedDocs }}
|
||||
<div class="related-section">
|
||||
|
||||
{{ if .PublishedTogether }}
|
||||
<details class="collapse-section"{{ if not $.Prefs.CollapsePubTogether }} open{{ end }}>
|
||||
<summary>
|
||||
<span class="collapse-ind" aria-hidden="true"></span>
|
||||
<h2>Published in the Same Report
|
||||
<small>{{ if .PublishedMore }}({{ .PublishedShown }} of {{ len .PublishedTogether }} documents){{ else }}({{ len .PublishedTogether }} documents){{ end }}</small>
|
||||
</h2>
|
||||
</summary>
|
||||
{{ template "doc-cards" .PublishedCards }}
|
||||
{{ with .PublishedMore }}{{ $one := eq (len .) 1 }}
|
||||
{{ range . }}
|
||||
<p class="pub-more"><a class="see-all" href="/article/{{ .Key }}">See all {{ len .DocIDs }} documents published with {{ if $one }}this article{{ else if .Title }}“{{ .Title }}”{{ else }}the article at {{ .Host }}{{ end }} →</a></p>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
</details>
|
||||
|
||||
{{ end }}
|
||||
|
||||
{{ if .RelatedDocs }}
|
||||
<details class="collapse-section"{{ if not $.Prefs.CollapseRelated }} open{{ end }}>
|
||||
<summary>
|
||||
<span class="collapse-ind" aria-hidden="true"></span>
|
||||
<h2>Related Documents</h2>
|
||||
</summary>
|
||||
{{ template "doc-cards" .RelatedCards }}
|
||||
</details>
|
||||
{{ end }}
|
||||
|
||||
</div>
|
||||
{{ end }}
|
||||
|
||||
{{ end }}
|
||||
|
||||
{{ template "base.html" . }}
|
||||
50
templates/explore.html
Normal file
50
templates/explore.html
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
{{ define "title" }}Explore - Snowden Archive{{ end }}
|
||||
{{ define "content" }}
|
||||
|
||||
<h1>Explore the archive</h1>
|
||||
<p class="tab-desc">
|
||||
Ranked views and lists across the whole archive, which the
|
||||
<a href="/browse">/browse</a> doesn't include.
|
||||
</p>
|
||||
|
||||
<ul class="explore-list">
|
||||
{{ range .Metrics }}
|
||||
<li>
|
||||
<a href="/explore/{{ .Slug }}">{{ .Title }}</a>
|
||||
<span class="muted">- {{ .Blurb }}</span>
|
||||
</li>
|
||||
{{ end }}
|
||||
<li>
|
||||
<a href="/explore/portions">Portion markings</a>
|
||||
<span class="muted">- The marking on each classified portion, read out of the OCRs (levels, compartments, and how often).</span>
|
||||
</li>
|
||||
<li>
|
||||
<a href="/explore/classification">Classification patterns</a>
|
||||
<span class="muted">- The marking mix across document years, and an agency x classification heat table.</span>
|
||||
</li>
|
||||
<li>
|
||||
<a href="/explore/outlets">Publishers by year</a>
|
||||
<span class="muted">- When each publisher released documents, and how much (i.e. documents and stories per publisher and year).</span>
|
||||
</li>
|
||||
<li>
|
||||
<a href="/explore/ids">Technical IDs</a>
|
||||
<span class="muted">- Every document id, PDF filename, size and SHA-256 checksum.</span>
|
||||
</li>
|
||||
<li>
|
||||
<a href="/browse/genre">Genres</a>
|
||||
<span class="muted">- How many documents of each kind the archive holds (slides, reports, wikis, manuals, guides, newsletters).</span>
|
||||
</li>
|
||||
{{ if $.HasAssemblies }}
|
||||
<li>
|
||||
<a href="/explore/assemblies">Assemblies</a>
|
||||
<span class="muted">- Sets of pages this archive put together from documents and copies published separately, so they can be read in one place.</span>
|
||||
</li>
|
||||
{{ end }}
|
||||
<li>
|
||||
<a href="/explore/terms">Acronyms & codewords</a>
|
||||
<span class="muted">- Every ALL-CAPS token across the corpus, auto-extracted (e.g. acronyms, program names, compartments, org designators).</span>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
{{ end }}
|
||||
{{ template "base.html" . }}
|
||||
35
templates/explore_table.html
Normal file
35
templates/explore_table.html
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
{{ define "title" }}{{ .Metric.Title }} - Snowden Archive{{ end }}
|
||||
{{ define "content" }}
|
||||
|
||||
<p class="crumb"><a href="/explore">← Explore</a></p>
|
||||
<h1>{{ .Metric.Title }}</h1>
|
||||
<p class="tab-desc">{{ .Metric.Blurb }}</p>
|
||||
<p class="muted">Top {{ .Shown }} of {{ .Total }} documents.</p>
|
||||
|
||||
<div class="table-scroll">
|
||||
<table class="cat-table explore-table">
|
||||
<thead>
|
||||
<tr>{{ range .Heads }}{{ template "sort-th" . }}{{ end }}</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{{ if .RowsHTML }}{{ .RowsHTML }}{{ else }}{{ template "explore-rows" . }}{{ end }} </tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
{{ end }}
|
||||
{{ template "base.html" . }}
|
||||
|
||||
{{ define "explore-rows" }} {{ range .Rows }}
|
||||
<tr>
|
||||
<td class="num-col faint">{{ .Rank }}</td>
|
||||
<td>
|
||||
<a href="/doc/{{ .Doc.ID }}">{{ .Doc.Title }}</a>
|
||||
{{ template "type-badge" .Doc.Type }}
|
||||
</td>
|
||||
<td class="explore-date">{{ with .Doc.DocMeta.DateCreated }}{{ . }}{{ else }}<span class="faint">-</span>{{ end }}</td>
|
||||
{{ range .Vals }}
|
||||
<td class="num-col">{{ . }}</td>
|
||||
{{ end }}
|
||||
</tr>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
36
templates/glossary.html
Normal file
36
templates/glossary.html
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
{{ define "title" }}Glossary - Snowden Archive{{ end }}
|
||||
{{ define "content" }}
|
||||
<h1>Glossary {{ if .Total }}<small class="muted">({{ .Total }} terms)</small>{{ end }}</h1>
|
||||
|
||||
{{ if .Groups }}
|
||||
<nav class="letter-nav" aria-label="Jump to letter">
|
||||
{{ range .Groups }}<a href="#g{{ .Letter }}">{{ .Letter }}</a>{{ end }}
|
||||
</nav>
|
||||
|
||||
{{ if .GroupsHTML }}{{ .GroupsHTML }}{{ else }}{{ template "glossary-groups" . }}{{ end }}{{ else }}
|
||||
<p class="muted">No glossary entries.</p>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
{{ template "base.html" . }}
|
||||
|
||||
{{ define "glossary-groups" }}{{ range .Groups }}
|
||||
<section class="glossary-group" id="g{{ .Letter }}">
|
||||
<h2>{{ .Letter }}</h2>
|
||||
<dl class="glossary-list">
|
||||
{{ range .Entries }}
|
||||
<dt id="term-{{ pathEscape .Term }}">
|
||||
{{ .Term }}
|
||||
{{ if .Aka }}<span class="muted glossary-aka">(a.k.a. {{ range $i, $a := .Aka }}{{ if $i }}, {{ end }}{{ $a }}{{ end }})</span>{{ end }}
|
||||
<a class="glossary-search" href="{{ .SearchURL }}"{{ if .Aka }} title="Searches the term and its aliases"{{ end }}>search →</a>
|
||||
</dt>
|
||||
<dd>
|
||||
{{ .DefinitionHTML }}
|
||||
{{ if .SeeLinks }}<div class="glossary-see muted">See also:
|
||||
{{ range $i, $s := .SeeLinks }}{{ if $i }}, {{ end }}{{ if $s.Term }}<a href="#term-{{ pathEscape $s.Term }}">{{ $s.Name }}</a>{{ else }}{{ $s.Name }}{{ end }}{{ end }}
|
||||
</div>{{ end }}
|
||||
</dd>
|
||||
{{ end }}
|
||||
</dl>
|
||||
</section>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
63
templates/ids.html
Normal file
63
templates/ids.html
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
{{ define "title" }}Technical IDs - Snowden Archive{{ end }}
|
||||
|
||||
{{ define "content" }}
|
||||
<header class="list-header">
|
||||
<p class="crumb"><a href="/explore">← Explore</a></p>
|
||||
<h1>Technical IDs <small class="count-note">{{ .Shown }} document{{ if ne .Shown 1 }}s{{ end }}</small></h1>
|
||||
</header>
|
||||
|
||||
<p class="muted">
|
||||
<a href="/explore/ids.txt">Checksums as plain text</a>
|
||||
<span class="faint">(<code>sha256sum -c</code> in the corpus root)</span>
|
||||
</p>
|
||||
|
||||
<div class="table-scroll">
|
||||
<table class="data-table id-table">
|
||||
<colgroup>
|
||||
<col class="c-id"><col class="c-coll"><col class="c-pages">
|
||||
<col class="c-pdf"><col class="c-size"><col class="c-sha">
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Document</th>
|
||||
<th>Collection</th>
|
||||
<th class="num">Pages</th>
|
||||
<th>PDF</th>
|
||||
<th class="num">Size</th>
|
||||
<th>SHA-256</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{{ if .RowsHTML }}{{ .RowsHTML }}{{ else }}{{ template "id-rows" . }}{{ end }}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{{ end }}
|
||||
|
||||
{{ template "base.html" . }}
|
||||
|
||||
{{ define "id-rows" }}{{ range .Rows }}
|
||||
<tr>
|
||||
<td><a href="/doc/{{ .ID }}">{{ .ID }}</a>
|
||||
{{- range .Aka }}<br><span class="faint">also published as</span> <code>{{ .ID }}</code>{{ end -}}
|
||||
</td>
|
||||
<td>{{ .Collection }}</td>
|
||||
<td class="num">{{ if .Pages }}{{ .Pages }}{{ end }}</td>
|
||||
<td><a href="{{ .PDFURL }}" target="_blank" rel="noopener"><code>{{ .PDFFile }}</code></a></td>
|
||||
<td class="num">{{ .Size }}</td>
|
||||
<td>{{ with .SHA }}<code class="checksum">{{ . }}</code>{{ else }}<span class="faint">-</span>{{ end }}</td>
|
||||
</tr>
|
||||
{{ $id := .ID }}
|
||||
{{ range .Versions }}
|
||||
<tr class="id-version">
|
||||
<td><span class="faint">↳</span> <a href="/doc/{{ $id }}/versions/{{ .Slug }}">versions/{{ .Slug }}</a></td>
|
||||
<td class="faint">copy</td>
|
||||
<td class="num">{{ if .Pages }}{{ .Pages }}{{ end }}</td>
|
||||
<td><a href="{{ .PDFURL }}" target="_blank" rel="noopener"><code>{{ .PDFFile }}</code></a></td>
|
||||
<td class="num">{{ .Size }}</td>
|
||||
<td>{{ if .SHA }}<code class="checksum">{{ .SHA }}</code>{{ else }}<span class="faint">-</span>{{ end }}</td>
|
||||
</tr>
|
||||
{{ end }}
|
||||
{{ else }}
|
||||
<tr><td colspan="6" class="muted">No documents.</td></tr>
|
||||
{{ end }}{{ end }}
|
||||
150
templates/index.html
Normal file
150
templates/index.html
Normal file
|
|
@ -0,0 +1,150 @@
|
|||
{{ define "content" }}
|
||||
<h1>Snowden Archive</h1>
|
||||
|
||||
<p class="muted home-intro">
|
||||
This archive holds {{ .SnowdenTotal }} published Snowden documents. {{ .ReportedTotal }} documents are
|
||||
published with news reporting, and {{ .ArchiveTotal }} documents are the NSA newsletter SIDtoday
|
||||
issues, from The Intercept's bulk release.{{ if .FOIATotal }} This archive also holds {{ .FOIATotal }}
|
||||
FOIA documents related to the Snowden disclosures (FOIA litigation, FISC and ODNI declassifications),
|
||||
which {{ if .IncludeFOIA }}you have included via <a href="/preferences">Preferences</a>{{ else }}are left out by default but can be included via <a href="/preferences">Preferences</a>{{ end }}.{{ end }}
|
||||
</p>
|
||||
|
||||
<form action="/search" method="GET" class="home-search-form" role="search">
|
||||
<input type="text" name="q" class="home-search" placeholder="Search the archive... (Enter to search)" aria-label="Search the archive">
|
||||
</form>
|
||||
|
||||
<div class="home-actions">
|
||||
<a class="home-action" href="/browse">Browse the archive →</a>
|
||||
</div>
|
||||
|
||||
{{ if and .TrackRecent .RecentDocs }}
|
||||
<section class="home-section">
|
||||
<h2>Recently viewed
|
||||
<button type="submit" form="clear-recent-docs" class="link-btn faint">clear</button>
|
||||
</h2>
|
||||
<form method="POST" action="/history/clear" id="clear-recent-docs" class="inline-form">
|
||||
<input type="hidden" name="which" value="docs">
|
||||
<input type="hidden" name="back" value="/">
|
||||
</form>
|
||||
<ul class="stat-list">
|
||||
{{ range .RecentDocs }}
|
||||
<li><a href="/doc/{{ .ID }}">{{ .Title }}</a> <span class="muted">{{ .DocMeta.Creator }}</span></li>
|
||||
{{ end }}
|
||||
</ul>
|
||||
</section>
|
||||
{{ end }}
|
||||
|
||||
<p class="home-stats-note muted">{{ if .WithSIDtoday }}The statistics below include SIDtoday archive documents (<a href="/">exclude them</a>).{{ else }}The statistics below exclude SIDtoday archive documents (<a href="/?sidtoday=1">include them</a>).{{ end }}</p>
|
||||
|
||||
{{ if .DocHist }}
|
||||
<section class="home-section">
|
||||
<h2>When the documents were written</h2>
|
||||
<div class="date-hist">
|
||||
{{ range .DocHist }}
|
||||
<a class="hist-col" href="{{ .URL }}" title="{{ .Count }} documents dated {{ .Year }}">
|
||||
<span class="hist-count">{{ .Count }}</span>
|
||||
<span class="hist-track"><span class="hist-bar" style="height: {{ .Pct }}%"></span></span>
|
||||
<span class="hist-year">{{ .Year }}</span>
|
||||
</a>
|
||||
{{ end }}
|
||||
</div>
|
||||
</section>
|
||||
{{ end }}
|
||||
|
||||
{{ if .PubHist }}
|
||||
<section class="home-section">
|
||||
<h2>When they were published</h2>
|
||||
<div class="date-hist">
|
||||
{{ range .PubHist }}
|
||||
<a class="hist-col" href="{{ .URL }}" title="{{ .Count }} documents first published in {{ .Year }}">
|
||||
<span class="hist-count">{{ .Count }}</span>
|
||||
<span class="hist-track"><span class="hist-bar" style="height: {{ .Pct }}%"></span></span>
|
||||
<span class="hist-year">{{ .Year }}</span>
|
||||
</a>
|
||||
{{ end }}
|
||||
</div>
|
||||
</section>
|
||||
{{ end }}
|
||||
|
||||
{{ if .TopPrograms }}
|
||||
<section class="home-section">
|
||||
<h2>Top programs <a class="see-all" href="/browse/program">all →</a></h2>
|
||||
<div class="pill-list">
|
||||
{{ range .TopPrograms }}
|
||||
<a class="pill" href="{{ .URL }}">{{ .Name }} <span class="pill-count">{{ .Count }}</span></a>
|
||||
{{ end }}
|
||||
</div>
|
||||
</section>
|
||||
{{ end }}
|
||||
|
||||
<div class="home-grid">
|
||||
{{ if .TopAgencies }}
|
||||
<section class="home-section">
|
||||
<h2>Agencies <a class="see-all" href="/browse/agency">all →</a></h2>
|
||||
<ul class="stat-list">
|
||||
{{ range .TopAgencies }}
|
||||
<li><a href="{{ .URL }}">{{ .Name }}</a> <span class="muted">{{ .Count }}</span></li>
|
||||
{{ end }}
|
||||
</ul>
|
||||
</section>
|
||||
{{ end }}
|
||||
|
||||
{{ if .TopCountries }}
|
||||
<section class="home-section">
|
||||
<h2>Countries <a class="see-all" href="/browse/country">all →</a></h2>
|
||||
<ul class="stat-list">
|
||||
{{ range .TopCountries }}
|
||||
<li><a href="{{ .URL }}">{{ .Name }}</a> <span class="muted">{{ .Count }}</span></li>
|
||||
{{ end }}
|
||||
</ul>
|
||||
</section>
|
||||
{{ end }}
|
||||
|
||||
{{ if .TopPeople }}
|
||||
<section class="home-section">
|
||||
<h2>People <a class="see-all" href="/browse/people">all →</a></h2>
|
||||
<ul class="stat-list">
|
||||
{{ range .TopPeople }}
|
||||
<li><a href="{{ .URL }}">{{ .Name }}</a> <span class="muted">{{ .Count }}</span></li>
|
||||
{{ end }}
|
||||
</ul>
|
||||
</section>
|
||||
{{ end }}
|
||||
|
||||
{{ if .TopJournalists }}
|
||||
<section class="home-section">
|
||||
<h2>Journalists <a class="see-all" href="/browse/journalist">all →</a></h2>
|
||||
<ul class="stat-list">
|
||||
{{ range .TopJournalists }}
|
||||
<li><a href="{{ .URL }}">{{ .Name }}</a> <span class="muted">{{ .Count }}</span></li>
|
||||
{{ end }}
|
||||
</ul>
|
||||
</section>
|
||||
{{ end }}
|
||||
|
||||
{{ if .TopPublishers }}
|
||||
<section class="home-section">
|
||||
<h2>Publishers <a class="see-all" href="/browse/publisher">all →</a></h2>
|
||||
<ul class="stat-list">
|
||||
{{ range .TopPublishers }}
|
||||
<li><a href="{{ .URL }}">{{ .Name }}</a> <span class="muted">{{ .Count }}</span></li>
|
||||
{{ end }}
|
||||
</ul>
|
||||
</section>
|
||||
{{ end }}
|
||||
</div>
|
||||
|
||||
{{ if .TopTags }}
|
||||
<section class="home-section">
|
||||
<h2>Top tags <a class="see-all" href="/browse/tag">all →</a></h2>
|
||||
<div class="pill-list">
|
||||
{{ range .TopTags }}
|
||||
<a class="pill" href="{{ .URL }}">{{ .Name }} <span class="pill-count">{{ .Count }}</span></a>
|
||||
{{ end }}
|
||||
</div>
|
||||
</section>
|
||||
{{ end }}
|
||||
|
||||
<p class="explore-link">Digging deeper? <a href="/explore">Explore the archive</a> for ranked views and lists.
|
||||
{{ end }}
|
||||
{{ template "base.html" . }}
|
||||
95
templates/list.html
Normal file
95
templates/list.html
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
{{ define "title" }}{{ with .CatLabel }}{{ . }}{{ else }}Category{{ end }}: {{ .Title }} - Snowden Archive{{ end }}
|
||||
{{ define "content" }}
|
||||
|
||||
<header class="list-header">
|
||||
<h1>{{ with .CatLabel }}{{ . }}{{ else }}Category{{ end }}: {{ .Title }}
|
||||
{{- if eq .ActiveCategory "agency" }}{{ with agencyName .Title }} <small class="muted cat-note">{{ . }}</small>{{ end }}{{ end }}
|
||||
<small class="count-note">{{ .Total }} document{{ if ne .Total 1 }}s{{ end }}</small>
|
||||
</h1>
|
||||
{{ if .SearchLink }}
|
||||
<a href="{{ .SearchLink }}" class="faint search-link">Search documents matching “{{ .Title }}” →</a>
|
||||
{{ end }}
|
||||
{{ with .Blurb }}<p class="muted cat-blurb">{{ . }}</p>{{ end }}
|
||||
{{ with .YearScope }}
|
||||
<p class="muted coll-scope">Showing the documents {{ $.Title }} itself published in <strong>{{ . }}</strong> -
|
||||
<a href="{{ $.YearScopeAllURL }}">all years</a>.</p>
|
||||
{{ end }}
|
||||
</header>
|
||||
|
||||
{{ if .MonthNav }}
|
||||
<nav class="month-nav" aria-label="Months in {{ .MonthNavYear }}">
|
||||
<a href="{{ .MonthNavYearURL }}"{{ if .MonthNavYearActive }} class="active"{{ end }}>All {{ .MonthNavYear }}</a>
|
||||
{{ range .MonthNav }}<a href="{{ .URL }}"{{ if .Active }} class="active"{{ end }}>{{ .Label }} <span class="facet-count">{{ .Count }}</span></a>
|
||||
{{ end }}
|
||||
{{ if .MonthlessCount }}<span class="faint month-nav-note">+{{ .MonthlessCount }} in {{ .MonthNavYear }} with year-only dates</span>{{ end }}
|
||||
</nav>
|
||||
{{ end }}
|
||||
|
||||
{{ with .GlossaryEntry }}
|
||||
<div class="glossary-box">
|
||||
<strong>{{ .Term }}</strong>
|
||||
{{ if .Aka }}<span class="muted glossary-aka">(a.k.a. {{ range $i, $a := .Aka }}{{ if $i }}, {{ end }}{{ $a }}{{ end }})</span>{{ end }}
|
||||
- {{ .DefinitionHTML }}
|
||||
<a href="/glossary#term-{{ pathEscape .Term }}" class="glossary-search">glossary →</a>
|
||||
</div>
|
||||
{{ end }}
|
||||
|
||||
{{ if or .Timeline .PubTimeline .Breakdowns }}
|
||||
<details class="entity-timeline">
|
||||
<summary>Graphs</summary>
|
||||
{{ if .Timeline }}{{ template "timeline-hist" (dict "Bars" .Timeline "Label" "By document date" "Verb" "dated") }}{{ end }}
|
||||
{{ if .PubTimeline }}{{ template "timeline-hist" (dict "Bars" .PubTimeline "Label" "By published date" "Verb" "published") }}{{ end }}
|
||||
{{ range .Breakdowns }}
|
||||
<div class="timeline-block">
|
||||
<h2 class="timeline-label">By {{ .Label }}</h2>
|
||||
<div class="breakdown-bars">
|
||||
{{ range .Values }}
|
||||
<a class="bd-row" href="{{ .URL }}" title="{{ .Count }} documents - open as a search">
|
||||
<span class="bd-name">{{ .Value }}</span>
|
||||
<span class="bd-track"><span class="bd-bar" style="width: {{ .Pct }}%"></span></span>
|
||||
<span class="bd-count">{{ .Count }}</span>
|
||||
</a>
|
||||
{{ end }}
|
||||
</div>
|
||||
</div>
|
||||
{{ end }}
|
||||
</details>
|
||||
{{ end }}
|
||||
|
||||
{{ if .Refinements }}
|
||||
<details class="search-facets">
|
||||
<summary>Refine <span class="faint">(combine with another filter, via search)</span></summary>
|
||||
<div class="facets-body">
|
||||
{{ range .Refinements }}
|
||||
<div class="facet-group">
|
||||
<span class="facet-label">{{ .Label }}</span>
|
||||
<span class="facet-values">{{ range .Values }}<a class="facet-link" href="{{ .URL }}">{{ .Name }}<span class="facet-count">{{ .Count }}</span></a>{{ end }}</span>
|
||||
</div>
|
||||
{{ end }}
|
||||
</div>
|
||||
</details>
|
||||
{{ end }}
|
||||
|
||||
<div class="list-controls">
|
||||
{{ if ne .View "table" }}
|
||||
{{ template "sort-bar" (dict "SortURL" .SortURL "SortBy" .Sort "Order" .Order "ShowRelevance" false "DefaultSort" .DefaultSort) }}
|
||||
{{ end }}
|
||||
{{ template "type-filter" (dict
|
||||
"Action" (print "/browse/" .ActiveCategory "/" (pathEscape .Title))
|
||||
"Params" .Params
|
||||
"AllTypes" .AllTypes
|
||||
"SelectedTypes" .SelectedTypes
|
||||
) }}
|
||||
</div>
|
||||
|
||||
{{ if eq .View "table" }}
|
||||
{{ template "doc-table" . }}
|
||||
{{ else }}
|
||||
<h2 class="sr-only">Documents</h2>
|
||||
{{ template "doc-cards" .Cards }}
|
||||
{{ template "pagination" . }}
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
|
||||
|
||||
{{ template "base.html" . }}
|
||||
54
templates/outlets.html
Normal file
54
templates/outlets.html
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
{{ define "title" }}Publishers by year - Snowden Archive{{ end }}
|
||||
|
||||
{{ define "content" }}
|
||||
<header class="list-header">
|
||||
<p class="crumb"><a href="/explore">← Explore</a></p>
|
||||
<h1>Publishers by year</h1>
|
||||
</header>
|
||||
|
||||
<p class="muted">When each publisher released documents, and how much. The number of
|
||||
distinct stories are after the slash.</p>
|
||||
|
||||
<p class="hist-toggles faint">Showing:
|
||||
{{ if .NewsOnly }}<a href="/explore/outlets">all collections</a> · <strong>reported only</strong>
|
||||
{{ else }}<strong>all collections</strong> · <a href="/explore/outlets?coll=news">reported only</a>{{ end }}
|
||||
</p>
|
||||
|
||||
{{ with .Table }}
|
||||
{{ if .Rows }}
|
||||
<div class="table-scroll">
|
||||
<table class="data-table heat-table outlet-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Outlet</th>
|
||||
{{ range .YearLinks }}<th class="num"><a href="{{ .URL }}">{{ .Name }}</a></th>{{ end }}
|
||||
<th class="num-col">Total</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{{ range .Rows }}
|
||||
<tr>
|
||||
<td><a href="{{ .URL }}">{{ .Name }}</a></td>
|
||||
{{ range .Cells }}
|
||||
<td class="heat-cell">{{ if .Docs }}<a href="{{ .URL }}" title="{{ .Docs }} documents, {{ .Articles }} stories">{{ .Docs }}{{ if .Articles }}<small class="faint">/{{ .Articles }}</small>{{ end }}</a>{{ else }}<span class="faint">-</span>{{ end }}</td>
|
||||
{{ end }}
|
||||
<td class="num-col">{{ .Total }}</td>
|
||||
</tr>
|
||||
{{ end }}
|
||||
</tbody>
|
||||
<tfoot>
|
||||
<tr>
|
||||
<th>All outlets</th>
|
||||
{{ range .Totals }}<th class="num-col">{{ if . }}{{ . }}{{ else }}<span class="faint">-</span>{{ end }}</th>{{ end }}
|
||||
<th></th>
|
||||
</tr>
|
||||
</tfoot>
|
||||
</table>
|
||||
</div>
|
||||
{{ else }}
|
||||
<p class="muted">No dated publications in the corpus yet.</p>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
|
||||
{{ template "base.html" . }}
|
||||
40
templates/pages.html
Normal file
40
templates/pages.html
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
{{ define "title" }}Pages - {{ .Doc.Title }}{{ with .Version }} ({{ .Slug }}){{ end }} - Snowden Archive{{ end }}
|
||||
|
||||
{{ define "content" }}
|
||||
<header class="list-header">
|
||||
<p class="crumb">{{ with .Version }}<a href="/doc/{{ $.Doc.ID }}/versions/{{ .Slug }}">← {{ $.Doc.Title }} (copy: {{ .Slug }})</a>{{ else }}<a href="/doc/{{ .Doc.ID }}">← {{ .Doc.Title }}</a>{{ end }}</p>
|
||||
<h1>All pages <small class="count-note">{{ len .Images }} page{{ if ne (len .Images) 1 }}s{{ end }}</small></h1>
|
||||
{{ with .Version }}{{ if .ChangedPages }}<p class="changed-pages-note">Pages that differ from the main document: {{ template "changed-page-links" . }}</p>{{ end }}{{ end }}
|
||||
</header>
|
||||
|
||||
{{ if and .Images .TextOnly }}
|
||||
<p class="muted textonly-note">Text-only mode is on - page images are not loaded. <a href="/preferences">Preferences</a></p>
|
||||
<ul class="page-link-list">
|
||||
{{ $id := .Doc.ID }}
|
||||
{{ range $i, $img := .Images }}
|
||||
<li><a href="{{ $.ImageBase }}{{ pathEscape $img }}" target="_blank" rel="noopener">Page {{ add $i 1 }}</a>{{ if and $.Version ($.Version.PageChanged (add $i 1)) }} <span class="muted">(differs from the main document)</span>{{ end }}</li>
|
||||
{{ end }}
|
||||
</ul>
|
||||
{{ else if .Images }}
|
||||
|
||||
<ul class="page-sheet">
|
||||
{{ $id := .Doc.ID }}
|
||||
{{ range $i, $img := .Images }}
|
||||
<li{{ if and $.Version ($.Version.PageChanged (add $i 1)) }} class="page-changed"{{ end }}>
|
||||
<a href="#lb-{{ add $i 1 }}" title="Open page {{ add $i 1 }} full screen">
|
||||
<img src="{{ $.ImageBase }}{{ pathEscape $img }}"{{ $.ImageSize $i }}
|
||||
alt="Page {{ add $i 1 }}" loading="lazy">
|
||||
<span class="page-num faint">{{ add $i 1 }}</span>{{ if and $.Version ($.Version.PageChanged (add $i 1)) }}<span class="page-changed-badge">differs</span>{{ end }}
|
||||
</a>
|
||||
</li>
|
||||
{{ end }}
|
||||
</ul>
|
||||
|
||||
{{ template "lightbox-overlays" . }}
|
||||
|
||||
{{ else }}
|
||||
<p class="muted">This document has no page images.</p>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
|
||||
{{ template "base.html" . }}
|
||||
412
templates/partials.html
Normal file
412
templates/partials.html
Normal file
|
|
@ -0,0 +1,412 @@
|
|||
{{ define "cat-boxes" }}
|
||||
{{ $a := .ActiveCategory }}{{ if not $a }}{{ $a = "" }}{{ end }}
|
||||
<nav id="cat-boxes" aria-label="Browse categories">
|
||||
<a href="/browse/program" class="cat-box{{ if eq $a "program" }} cat-box-active{{ end }}"><strong>Programs</strong><span class="count">{{ index .CatCounts "program" }}</span></a>
|
||||
<a href="/browse/journalist" class="cat-box{{ if eq $a "journalist" }} cat-box-active{{ end }}"><strong>Journalists</strong><span class="count">{{ index .CatCounts "journalist" }}</span></a>
|
||||
<a href="/browse/agency" class="cat-box{{ if eq $a "agency" }} cat-box-active{{ end }}"><strong>Agencies</strong><span class="count">{{ index .CatCounts "agency" }}</span></a>
|
||||
<a href="/browse/publisher" class="cat-box{{ if eq $a "publisher" }} cat-box-active{{ end }}"><strong>Publishers</strong><span class="count">{{ index .CatCounts "publisher" }}</span></a>
|
||||
<a href="/browse/country" class="cat-box{{ if eq $a "country" }} cat-box-active{{ end }}"><strong>Countries</strong><span class="count">{{ index .CatCounts "country" }}</span></a>
|
||||
<a href="/browse/people" class="cat-box{{ if eq $a "people" }} cat-box-active{{ end }}"><strong>People</strong><span class="count">{{ index .CatCounts "people" }}</span></a>
|
||||
<a href="/browse/classification" class="cat-box{{ if eq $a "classification" }} cat-box-active{{ end }}"><strong>Classification</strong><span class="count">{{ index .CatCounts "classification" }}</span></a>
|
||||
<a href="/browse/dissemination" class="cat-box{{ if eq $a "dissemination" }} cat-box-active{{ end }}"><strong>Dissemination</strong><span class="count">{{ index .CatCounts "dissemination" }}</span></a>
|
||||
<a href="/browse/published" class="cat-box{{ if eq $a "published" }} cat-box-active{{ end }}"><strong>Pub Year</strong><span class="count">{{ index .CatCounts "published" }}</span></a>
|
||||
<a href="/browse/docdate" class="cat-box{{ if eq $a "docdate" }} cat-box-active{{ end }}"><strong>Doc Date</strong><span class="count">{{ index .CatCounts "docdate" }}</span></a>
|
||||
<a href="/browse/tag" class="cat-box{{ if eq $a "tag" }} cat-box-active{{ end }}"><strong>Tags</strong><span class="count">{{ index .CatCounts "tag" }}</span></a>
|
||||
</nav>
|
||||
{{ end }}
|
||||
|
||||
{{/* pub-publisher is a publication entry's outlet: a link to the story where
|
||||
the entry has one, else to the outlet's page - which a partner that only
|
||||
reported has none of. */}}
|
||||
{{ define "pub-publisher" -}}
|
||||
{{ if .Pub.Link }}<a href="{{ .Pub.Link }}" target="_blank" rel="noopener"{{ if or .LinkClass .On }} class="{{ .LinkClass }}{{ if and .LinkClass .On }} {{ end }}{{ if .On }}active{{ end }}"{{ end }}>{{ .Pub.Publisher }} ↗</a>
|
||||
{{- else if .Pub.ReportingOnly }}{{ .Pub.Publisher }}
|
||||
{{- else }}<a href="/browse/publisher/{{ pathEscape .Pub.Publisher }}{{ with .QS }}{{ . }}{{ end }}"{{ if or .PageClass .On }} class="{{ .PageClass }}{{ if and .PageClass .On }} {{ end }}{{ if .On }}active{{ end }}"{{ end }}>{{ .Pub.Publisher }}</a>{{ end }}
|
||||
{{- end }}
|
||||
|
||||
{{ define "sort-th" }}{{ if .URL }}<th{{ with .Class }} class="{{ . }}"{{ end }}><a href="{{ .URL }}">{{ .Label }} {{ if .Active }}{{ if .Desc }}▼{{ else }}▲{{ end }}{{ else }}<span class="sort-hint">↕</span>{{ end }}</a></th>{{ else }}<th{{ with .Class }} class="{{ . }}"{{ end }}>{{ .Label }}</th>{{ end }}{{ end }}
|
||||
|
||||
{{ define "doc-table" }}
|
||||
<div class="table-scroll">
|
||||
<table class="data-table">
|
||||
<thead>
|
||||
<tr>{{ range .TableHeads }}{{ template "sort-th" . }}{{ end }}</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{{ if .RowsHTML }}{{ .RowsHTML }}{{ else }}{{ range .TableRows }}{{ template "doc-table-row" . }}{{ end }}{{ end }} </tbody>
|
||||
</table>
|
||||
</div>
|
||||
{{ end }}
|
||||
|
||||
{{/* The rows are rendered once for every reader, so a FOIA release's links
|
||||
always name every collection: they must list the release itself. */}}
|
||||
{{ define "doc-table-row" }}{{ $qs := "" }}{{ if eq .Collection "foia" }}{{ $qs = wholeArchiveQS }}{{ end }}
|
||||
<tr>
|
||||
<td>
|
||||
<a href="/doc/{{ .ID }}"><strong>{{ .Title }}</strong></a>
|
||||
[<a href="/files/{{ .ID }}/{{ pathEscape .PDFFile }}" target="_blank" rel="noopener">pdf</a>]
|
||||
</td>
|
||||
<td>
|
||||
{{ if .DocMeta.Creator }}<a href="/browse/agency/{{ pathEscape .DocMeta.Creator }}{{ $qs }}">{{ .DocMeta.Creator }}</a>{{ else }}-{{ end }}
|
||||
</td>
|
||||
<td>
|
||||
{{ with .DocMeta.DateCreated }}{{ . }}{{ else }}-{{ end }}
|
||||
</td>
|
||||
<td>
|
||||
{{- $first := true -}}
|
||||
{{- range .Publication -}}
|
||||
{{- if .DatePublished -}}
|
||||
{{- if not $first -}}, {{ end -}}
|
||||
{{ .DatePublished }}
|
||||
{{- $first = false -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
{{- if $first -}}-{{- end -}}
|
||||
</td>
|
||||
<td>{{ facetLinks (dict "Vals" .DocMeta.Classification "Cat" "classification" "QS" $qs) }}</td>
|
||||
<td>{{ facetLinks (dict "Vals" .DocMeta.Dissemination "Cat" "dissemination" "QS" $qs) }}</td>
|
||||
<td>{{ facetLinks (dict "Vals" .DocMeta.Programs "Cat" "program" "QS" $qs) }}</td>
|
||||
<td>{{ facetLinks (dict "Vals" .DocMeta.Countries "Cat" "country" "QS" $qs) }}</td>
|
||||
<td>{{ facetLinks (dict "Vals" .DocMeta.People "Cat" "people" "QS" $qs) }}</td>
|
||||
<td>{{ facetLinks (dict "Vals" .JournalistsDeduped "Cat" "journalist" "QS" $qs) }}</td>
|
||||
<td>
|
||||
{{- $first = true -}}
|
||||
{{- range .Publication -}}
|
||||
{{- if .Publisher -}}
|
||||
{{- if not $first -}}, {{ end -}}
|
||||
{{ template "pub-publisher" (dict "Pub" . "QS" $qs) }}
|
||||
{{- $first = false -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
{{- if $first -}}-{{- end -}}
|
||||
</td>
|
||||
<td>
|
||||
{{ template "type-badge" .Type }}
|
||||
{{ if and .MarkReported .ReportedIssue }}<span class="issue-reported" title="Individually reported - also in the Reported collection">reported</span>{{ end }}
|
||||
</td>
|
||||
</tr>
|
||||
{{ end }}
|
||||
|
||||
{{ define "doc-cards" }}
|
||||
<ul class="doc-grid">
|
||||
{{ range .Docs }}
|
||||
<li class="doc-card">
|
||||
|
||||
{{ if $.ShowThumbs }}
|
||||
<div class="thumb-container"{{ if and $.CardPagePreview .Images }} data-pp-id="{{ .ID }}" data-pp-imgs="{{ range $i, $im := .Images }}{{ if $i }} {{ end }}{{ pathEscape $im }}{{ end }}"{{ end }}>
|
||||
<a href="/doc/{{ .ID }}" class="thumb" tabindex="-1" aria-hidden="true">
|
||||
{{ if .HasThumb }}
|
||||
<img src="/files/{{ .ID }}/thumb.jpg"
|
||||
{{ if .HasGifThumb }}data-gif="/files/{{ .ID }}/thumb.gif"{{ end }}
|
||||
class="doc-img"
|
||||
alt="" loading="lazy">
|
||||
{{ end }}
|
||||
</a>
|
||||
</div>
|
||||
{{ end }}
|
||||
|
||||
<div class="doc-body">
|
||||
{{ $cat := $.ActiveCategory }}{{ $val := $.ActiveValue }}
|
||||
{{/* A FOIA release's links name every collection for a reader who
|
||||
leaves FOIA out: the pages they open must list the release. */}}
|
||||
{{ $qs := "" }}{{ if eq .Collection "foia" }}{{ $qs = $.FOIAQS }}{{ end }}
|
||||
|
||||
<h3>
|
||||
<a href="/doc/{{ .ID }}" class="doc-title">{{ if .Title }}{{ .Title }}{{ else }}<span class="doc-untitled">(untitled)</span>{{ end }}</a>
|
||||
{{ template "type-badge" .Type }}
|
||||
{{ if and $.MarkReported .ReportedIssue }}<span class="issue-reported" title="Individually reported - also in the Reported collection">reported</span>{{ end }}
|
||||
<a href="/files/{{ .ID }}/{{ pathEscape .PDFFile }}" target="_blank" rel="noopener" class="doc-pdf" title="Open PDF">[pdf]</a>
|
||||
{{ if $.ShowUnpin }}
|
||||
<button type="submit" form="unpin-{{ .ID }}" class="link-btn unpin-btn" title="Remove from pinned">★ unpin</button>
|
||||
{{ end }}
|
||||
</h3>
|
||||
{{ if $.ShowUnpin }}
|
||||
<form method="POST" action="/pin/{{ .ID }}" id="unpin-{{ .ID }}" class="inline-form">
|
||||
<input type="hidden" name="back" value="/pinned">
|
||||
</form>
|
||||
{{ end }}
|
||||
|
||||
<div class="doc-content">
|
||||
|
||||
<div>
|
||||
{{ if .DocMeta.Creator }}
|
||||
<a href="/browse/agency/{{ pathEscape .DocMeta.Creator }}{{ $qs }}"
|
||||
class="doc-creator{{ if and (eq $cat "agency") (eqFold .DocMeta.Creator $val) }} active{{ end }}">{{ .DocMeta.Creator }}</a>
|
||||
{{ end }}
|
||||
{{ if and .DocMeta.Creator .DocMeta.DateCreated }}
|
||||
<span class="doc-dot">·</span>
|
||||
{{ end }}
|
||||
{{ if .DocMeta.DateCreated }}
|
||||
{{ if ge (len .DocMeta.DateCreated) 4 }}
|
||||
{{ $year := slice .DocMeta.DateCreated 0 4 }}
|
||||
<a href="/browse/docdate/{{ $year }}{{ $qs }}"
|
||||
class="doc-created{{ if and (eq $cat "docdate") (eq $year $val) }} active{{ end }}">{{ .DocMeta.DateCreated }}</a>
|
||||
{{ else }}
|
||||
<span class="doc-created">{{ .DocMeta.DateCreated }}</span>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
{{ if .Pages }}
|
||||
{{ if or .DocMeta.Creator .DocMeta.DateCreated }}
|
||||
<span class="doc-dot">·</span>
|
||||
{{ end }}
|
||||
<span class="doc-pages">{{ .Pages }} p.</span>
|
||||
{{ end }}
|
||||
</div>
|
||||
|
||||
{{ if and .Description (not $.HideSummary) }}
|
||||
<p class="card-summary">{{ .Description }}</p>
|
||||
{{ end }}
|
||||
|
||||
{{ if and (or .DocMeta.Classification .DocMeta.Dissemination) (not $.HideClassification) }}
|
||||
<div>
|
||||
<span class="doc-classification">Marking:</span>
|
||||
{{ with .DocMeta.Classification }}{{ facetLinks (dict "Vals" . "Cat" "classification" "Sep" "dot" "Active" (and (eq $cat "classification") $val) "QS" $qs) }}{{ end }}
|
||||
{{ if and .DocMeta.Classification .DocMeta.Dissemination }}<span class="doc-dot">·</span>{{ end }}
|
||||
{{ with .DocMeta.Dissemination }}{{ facetLinks (dict "Vals" . "Cat" "dissemination" "Sep" "dot" "Active" (and (eq $cat "dissemination") $val) "QS" $qs) }}{{ end }}
|
||||
</div>
|
||||
{{ end }}
|
||||
|
||||
{{ if and .DocMeta.Programs (not $.HidePrograms) }}
|
||||
<div>
|
||||
<span class="doc-programs">Programs:</span>
|
||||
{{ facetLinks (dict "Vals" .DocMeta.Programs "Cat" "program" "Sep" "dot" "Active" (and (eq $cat "program") $val) "QS" $qs) }}
|
||||
</div>
|
||||
{{ end }}
|
||||
|
||||
{{ if and .DocMeta.People (not $.HidePeople) }}
|
||||
<div>
|
||||
<span class="doc-people">People:</span>
|
||||
{{ facetLinks (dict "Vals" .DocMeta.People "Cat" "people" "Sep" "dot" "Active" (and (eq $cat "people") $val) "QS" $qs) }}
|
||||
</div>
|
||||
{{ end }}
|
||||
|
||||
<div class="doc-publication">
|
||||
{{ $shown := false }}
|
||||
{{ range $pub := .Publication }}
|
||||
{{ if or $pub.Title $pub.Link $pub.Publisher $pub.DatePublished }}
|
||||
{{ if $shown }}<span class="doc-dot">·</span>{{ end }}{{ $shown = true }}
|
||||
{{ if $pub.Title }}
|
||||
{{ $ak := articleKey $pub.Link }}
|
||||
{{ if $ak }}<a href="/article/{{ $ak }}" class="pub-article-title">“{{ $pub.Title }}”</a>{{ else }}<span class="pub-article-title">“{{ $pub.Title }}”</span>{{ end }}
|
||||
-
|
||||
{{ end }}
|
||||
{{ if or $pub.Link $pub.Publisher }}
|
||||
{{ template "pub-publisher" (dict "Pub" $pub "QS" $qs "LinkClass" "pub-link" "PageClass" "pub-source" "On" (and (eq $cat "publisher") (eqFold $pub.Publisher $val))) }}
|
||||
{{ end }}
|
||||
{{ if $pub.DatePublished }}
|
||||
<span class="doc-published">({{ $pub.DatePublished }})</span>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
|
||||
{{ $journos := .JournalistsDeduped }}
|
||||
{{ if and $journos (not $.HideJournalists) }}
|
||||
<span class="doc-dot">·</span>
|
||||
{{ range $i, $j := $journos }}
|
||||
{{ if $i }}<span class="doc-dot">·</span>{{ end }}
|
||||
<a href="/browse/journalist/{{ pathEscape $j }}{{ $qs }}"
|
||||
class="doc-journalist{{ if and (eq $cat "journalist") (eqFold $j $val) }} active{{ end }}">{{ $j }}</a>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</li>
|
||||
{{ end }}
|
||||
</ul>
|
||||
{{ end }}
|
||||
|
||||
{{ define "type-filter" }}
|
||||
<form method="GET" action="{{ .Action }}" class="type-filter">
|
||||
{{ range $k, $vals := .Params }}
|
||||
{{ if and (ne $k "type") (ne $k "p") (ne $k "action") }}
|
||||
{{ range $vals }}
|
||||
<input type="hidden" name="{{ $k }}" value="{{ . }}">
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
|
||||
<span class="muted">Show:</span>
|
||||
|
||||
{{ range $type, $count := .AllTypes }}
|
||||
<label class="type-filter-option">
|
||||
<input type="checkbox" name="type" value="{{ $type }}"
|
||||
{{ if index $.SelectedTypes $type }}checked{{ end }}>
|
||||
{{ template "type-badge" $type }}
|
||||
{{ if eq $type "REPORTED" }}<span class="type-filter-leak">Reported</span>{{ end }}
|
||||
<span class="faint type-filter-count">({{ $count }})</span>
|
||||
</label>
|
||||
{{ end }}
|
||||
|
||||
<button type="submit" class="ctl">Apply</button>
|
||||
</form>
|
||||
{{ end }}
|
||||
|
||||
|
||||
{{ define "pagination" }}
|
||||
{{ if gt .TotalPages 1 }}
|
||||
{{ $sortParam := "" }}{{ if .Sort }}{{ $sortParam = print "&sort=" .Sort }}{{ end }}
|
||||
{{ $orderParam := "" }}{{ if .Order }}{{ $orderParam = print "&order=" .Order }}{{ end }}
|
||||
<nav class="pagination" aria-label="Pagination">
|
||||
|
||||
{{ if gt .Page 1 }}
|
||||
<a href="{{ .BaseURL }}p={{ add .Page -1 }}{{ $sortParam }}{{ $orderParam }}" class="pag-btn" rel="prev">« Prev</a>
|
||||
{{ end }}
|
||||
|
||||
{{ range pages .Page .TotalPages }}
|
||||
{{ if eq . 0 }}
|
||||
<span class="pag-ellipsis" aria-hidden="true">...</span>
|
||||
{{ else if eq . $.Page }}
|
||||
<span class="pag-btn pag-current" aria-current="page">{{ . }}</span>
|
||||
{{ else }}
|
||||
<a href="{{ $.BaseURL }}p={{ . }}{{ $sortParam }}{{ $orderParam }}" class="pag-btn">{{ . }}</a>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
|
||||
{{ if lt .Page .TotalPages }}
|
||||
<a href="{{ .BaseURL }}p={{ add .Page 1 }}{{ $sortParam }}{{ $orderParam }}" class="pag-btn" rel="next">Next »</a>
|
||||
{{ end }}
|
||||
|
||||
</nav>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
|
||||
{{ define "timeline-hist" }}
|
||||
<div class="timeline-block">
|
||||
<h2 class="timeline-label">{{ .Label }} <span class="faint">({{ len .Bars }} years)</span></h2>
|
||||
<div class="date-hist entity-timeline-hist">
|
||||
{{ range .Bars }}
|
||||
{{ if .URL }}
|
||||
<a class="hist-col" href="{{ .URL }}" title="{{ .Count }} documents {{ $.Verb }} {{ .Year }}">
|
||||
<span class="hist-count">{{ .Count }}</span>
|
||||
<span class="hist-track"><span class="hist-bar" style="height: {{ .Pct }}%"></span></span>
|
||||
<span class="hist-year">{{ .Year }}</span>
|
||||
</a>
|
||||
{{ else }}
|
||||
<span class="hist-col" title="{{ .Count }} documents {{ $.Verb }} {{ .Year }}">
|
||||
<span class="hist-count">{{ .Count }}</span>
|
||||
<span class="hist-track"><span class="hist-bar" style="height: {{ .Pct }}%"></span></span>
|
||||
<span class="hist-year">{{ .Year }}</span>
|
||||
</span>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
</div>
|
||||
</div>
|
||||
{{ end }}
|
||||
|
||||
{{ define "sort-bar" }}
|
||||
{{ $base := .SortURL }}
|
||||
{{ $order := .Order }}
|
||||
{{ $showRelevance := .ShowRelevance }}
|
||||
|
||||
{{ $cur := .SortBy }}
|
||||
{{ if eq $cur "" }}{{ $cur = .DefaultSort }}{{ end }}
|
||||
|
||||
{{ $activeLabel := sortLabel $cur }}
|
||||
{{ $directed := ne $cur "relevance" }}
|
||||
{{ $orderLabel := "↑" }}{{ if eq $order "desc" }}{{ $orderLabel = "↓" }}{{ end }}
|
||||
|
||||
<details class="sort-dropdown">
|
||||
<summary
|
||||
aria-label="Sort documents, currently sorted by {{ $activeLabel }}{{ if $directed }} {{ if eq $order "desc" }}descending{{ else }}ascending{{ end }}{{ end }}">
|
||||
Sort: <strong>{{ $activeLabel }}</strong>
|
||||
{{ if $directed }}<span aria-hidden="true">{{ $orderLabel }}</span>{{ end }}
|
||||
</summary>
|
||||
<div role="navigation" aria-label="Sort options" class="sort-options">
|
||||
{{ range sortOptions }}
|
||||
{{ if or (ne .Key "relevance") $showRelevance }}
|
||||
{{ $active := eq .Key $cur }}
|
||||
{{ $flips := and $active (ne .Key "relevance") }}
|
||||
{{ $href := print $base "sort=" .Key }}
|
||||
{{ if and $flips (ne $order "desc") }}{{ $href = print $href "&order=desc" }}{{ end }}
|
||||
<a href="{{ $href }}" class="{{ if $active }}active{{ end }}" {{ if $active }}aria-current="true"{{ end }}>
|
||||
{{ .Label }}
|
||||
{{ if $flips }}
|
||||
<span aria-hidden="true">{{ $orderLabel }}</span>
|
||||
<span class="sr-only">{{ if eq $order "desc" }}descending, click for ascending{{ else }}ascending, click for descending{{ end }}</span>
|
||||
{{ end }}
|
||||
</a>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
</div>
|
||||
</details>
|
||||
{{ end }}
|
||||
|
||||
{{ define "type-badge" }}
|
||||
{{ if eq . "FOIA" }}
|
||||
<span class="type-badge type-badge--foia">FOIA</span>
|
||||
{{ else if eq . "SIDTODAY" }}
|
||||
<span class="type-badge type-badge--sidtoday" title="This document is part of SIDtoday">SIDtoday</span>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
|
||||
{{ define "lightbox-overlays" }}
|
||||
{{ $lbBase := .ImageBase }}{{ $lbN := len .Images }}
|
||||
{{ range $i, $img := .Images }}
|
||||
<div class="lightbox" id="lb-{{ add $i 1 }}" aria-label="Page {{ add $i 1 }} of {{ $lbN }}">
|
||||
<a class="lb-backdrop" href="#lb-closed" tabindex="-1" aria-hidden="true"></a>
|
||||
<a class="lb-close" href="#lb-closed" title="Close">×</a>
|
||||
{{ if $i }}<a class="lb-prev" href="#lb-{{ $i }}" title="Previous page">‹</a>{{ end }}
|
||||
{{ if lt (add $i 1) $lbN }}<a class="lb-next" href="#lb-{{ add $i 2 }}" title="Next page">›</a>{{ end }}
|
||||
<img src="{{ $lbBase }}{{ pathEscape $img }}"{{ $.ImageSize $i }} alt="Page {{ add $i 1 }} of {{ $lbN }}" loading="lazy" decoding="async">
|
||||
<span class="lb-counter">{{ add $i 1 }} / {{ $lbN }}{{ if and $.Version ($.Version.PageChanged (add $i 1)) }} · differs from the main document{{ end }}</span>
|
||||
</div>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
|
||||
{{ define "changed-page-links" }}
|
||||
{{- $v := . -}}
|
||||
{{- range $k, $p := .ChangedPages -}}
|
||||
{{- if $k }}, {{ end -}}
|
||||
{{- if $p.Pos }}<a href="/doc/{{ $v.DocID }}/versions/{{ $v.Slug }}/pages#lb-{{ $p.Pos }}" title="Open page {{ $p.Page }} of this copy">{{ $p.Page }}</a>{{ else }}{{ $p.Page }}{{ end -}}
|
||||
{{- end -}}
|
||||
{{ end }}
|
||||
|
||||
{{ define "doc-viewport" }}
|
||||
<input type="radio" name="view-control" id="view-images" aria-controls="doc-viewport" {{ if eq .Prefs.DocView "images" }}checked{{ end }}>
|
||||
<input type="radio" name="view-control" id="view-split" aria-controls="doc-viewport" {{ if eq .Prefs.DocView "split" }}checked{{ end }}>
|
||||
<input type="radio" name="view-control" id="view-text" aria-controls="doc-viewport" {{ if eq .Prefs.DocView "text" }}checked{{ end }}>
|
||||
|
||||
<div class="view-picker" role="radiogroup" aria-label="Document display mode">
|
||||
<label for="view-images" class="btn-img">Images Only</label>
|
||||
<label for="view-split" class="btn-split">Split View</label>
|
||||
<label for="view-text" class="btn-txt">Text Only</label>
|
||||
</div>
|
||||
|
||||
<div id="doc-viewport" class="content-wrapper{{ if .Prefs.ScrollDoc }} scroll-panes{{ end }}">
|
||||
<div class="pane images-pane">
|
||||
{{ if gt (len .Images) 1 }}
|
||||
<nav class="page-jump" aria-label="Jump to page">
|
||||
<span class="faint">Page:</span>
|
||||
{{ range $i, $img := .Images }}<a href="#page-{{ add $i 1 }}"{{ if and $.Version ($.Version.PageChanged (add $i 1)) }} class="page-changed-jump" title="Differs from the main document"{{ end }}>{{ add $i 1 }}</a>{{ end }}
|
||||
</nav>
|
||||
{{ end }}
|
||||
<div class="image-list">
|
||||
{{ if .Prefs.TextOnly }}
|
||||
<p class="muted textonly-note">Text-only mode is on - page images are not loaded. <a href="/preferences">Preferences</a></p>
|
||||
<ul class="page-link-list">
|
||||
{{ range $i, $img := .Images }}
|
||||
<li id="page-{{ add $i 1 }}"><a href="{{ $.ImageBase }}{{ pathEscape $img }}" target="_blank" rel="noopener">Page {{ add $i 1 }}</a>{{ if and $.Version ($.Version.PageChanged (add $i 1)) }} <span class="muted">(differs from the main document)</span>{{ end }}</li>
|
||||
{{ end }}
|
||||
</ul>
|
||||
{{ else }}
|
||||
{{ range $i, $img := .Images }}
|
||||
<a href="#lb-{{ add $i 1 }}" id="page-{{ add $i 1 }}" class="page-open{{ if and $.Version ($.Version.PageChanged (add $i 1)) }} page-changed{{ end }}" title="Open page {{ add $i 1 }} full screen">
|
||||
<img src="{{ $.ImageBase }}{{ pathEscape $img }}"{{ $.ImageSize $i }}
|
||||
alt="Page {{ add $i 1 }}"
|
||||
loading="lazy">
|
||||
{{- if and $.Version ($.Version.PageChanged (add $i 1)) }}<span class="page-changed-badge">differs</span>{{ end }}
|
||||
</a>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="pane text-pane">
|
||||
<div class="prose">
|
||||
{{ .Content }}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
{{ end }}
|
||||
46
templates/pinned.html
Normal file
46
templates/pinned.html
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
{{ define "title" }}Pinned - Snowden Archive{{ end }}
|
||||
{{ define "content" }}
|
||||
<header class="list-header">
|
||||
<h1>Pinned documents
|
||||
<small class="count-note">{{ .Total }} pinned</small>
|
||||
{{ if .Total }}<a href="/pinned/export" class="export-link" title="Download your pinned list as JSON">export JSON</a>{{ end }}
|
||||
</h1>
|
||||
</header>
|
||||
<p class="muted">Pins are stored only in this browser's cookies.</p>
|
||||
{{ if .Full }}
|
||||
<p class="muted" role="status"><strong>Your pinned list is full.</strong> A browser cookie holds only so much, so that pin was not added. Unpin a document or a search below to make room.</p>
|
||||
{{ end }}
|
||||
|
||||
{{ if .Cards.Docs }}
|
||||
<h2 class="sr-only">Documents</h2>
|
||||
{{ template "doc-cards" .Cards }}
|
||||
{{ else }}
|
||||
<p class="muted">Nothing pinned yet. Open a document and press <strong>☆ Pin</strong> to start a reading list.</p>
|
||||
{{ end }}
|
||||
|
||||
{{ if .Searches }}
|
||||
<h2 class="pinned-searches-heading">Pinned searches</h2>
|
||||
<ul class="pinned-searches">
|
||||
{{ range .Searches }}
|
||||
<li class="pinned-search-item">
|
||||
<a class="pill" href="{{ .URL }}">{{ .Label }}</a>
|
||||
<form method="POST" action="/pinsearch" class="inline-form">
|
||||
<input type="hidden" name="qs" value="{{ .QS }}">
|
||||
<input type="hidden" name="back" value="/pinned">
|
||||
<button type="submit" class="link-btn unpin-btn" title="Remove this search">★ unpin</button>
|
||||
</form>
|
||||
</li>
|
||||
{{ end }}
|
||||
</ul>
|
||||
{{ end }}
|
||||
|
||||
<section class="note-list-box" id="note-list-box" hidden>
|
||||
<h2>My notes
|
||||
<button type="button" class="export-link" id="note-export" title="Download all notes as JSON">export JSON</button>
|
||||
</h2>
|
||||
<p class="muted">Stored only in this browser (localStorage).</p>
|
||||
<ul class="note-list" id="note-list"></ul>
|
||||
</section>
|
||||
|
||||
{{ end }}
|
||||
{{ template "base.html" . }}
|
||||
82
templates/portions.html
Normal file
82
templates/portions.html
Normal file
|
|
@ -0,0 +1,82 @@
|
|||
{{ define "title" }}Portion markings - Snowden Archive{{ end }}
|
||||
|
||||
{{ define "content" }}
|
||||
<header class="list-header">
|
||||
<p class="crumb"><a href="/explore">← Explore</a></p>
|
||||
<h1>Portion markings</h1>
|
||||
</header>
|
||||
|
||||
<p class="muted">The marking each classified portion of a document carries at the
|
||||
start of its own line (for example "<code>(U//FOUO)</code>", "<code>(TS//SI//NF)</code>").</p>
|
||||
|
||||
<h2>By level</h2>
|
||||
<div class="table-scroll">
|
||||
<table class="cat-table explore-table">
|
||||
<thead><tr><th>Level</th><th></th><th class="num">Markings</th><th class="num">Documents</th></tr></thead>
|
||||
<tbody>
|
||||
{{ range .Stats.Levels }}
|
||||
<tr>
|
||||
<td><a href="{{ .SearchURL }}"><code>{{ .Name }}</code></a></td>
|
||||
<td class="faint">{{ .Label }}</td>
|
||||
<td class="num">{{ .Marks }}</td>
|
||||
<td class="num">{{ .Docs }}</td>
|
||||
</tr>
|
||||
{{ end }}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<h2>Markings</h2>
|
||||
<p class="muted">Every distinct marking, most used first.
|
||||
{{ if gt .Stats.Distinct .RowLimit }}Top {{ .RowLimit }} of {{ .Stats.Distinct }}.{{ end }}</p>
|
||||
<div class="table-scroll">
|
||||
<table class="cat-table explore-table">
|
||||
<thead><tr><th>Marking</th><th class="num">Markings</th><th class="num">Documents</th></tr></thead>
|
||||
<tbody>
|
||||
{{ range .Stats.Markings }}
|
||||
<tr>
|
||||
<td><a href="{{ .SearchURL }}"><code>{{ .Name }}</code></a></td>
|
||||
<td class="num">{{ .Marks }}</td>
|
||||
<td class="num">{{ .Docs }}</td>
|
||||
</tr>
|
||||
{{ end }}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<h2>Control markings</h2>
|
||||
<p class="muted">The compartments and dissemination controls appearing inside
|
||||
those markings.</p>
|
||||
<div class="table-scroll">
|
||||
<table class="cat-table explore-table">
|
||||
<thead><tr><th>Control</th><th class="num">Markings</th><th class="num">Documents</th></tr></thead>
|
||||
<tbody>
|
||||
{{ range .Stats.Controls }}
|
||||
<tr>
|
||||
<td><a href="{{ .SearchURL }}"><code>{{ .Name }}</code></a></td>
|
||||
<td class="num">{{ .Marks }}</td>
|
||||
<td class="num">{{ .Docs }}</td>
|
||||
</tr>
|
||||
{{ end }}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
{{ if .Stats.UnrecogEg }}
|
||||
<h2>Not recognised</h2>
|
||||
<p class="muted">{{ .Stats.Unrecog }} markings open a line with a classification
|
||||
level but carry a part this server has no rule for.</p>
|
||||
<div class="table-scroll">
|
||||
<table class="cat-table explore-table">
|
||||
<thead><tr><th>As the transcript writes it</th><th class="num">Occurrences</th></tr></thead>
|
||||
<tbody>
|
||||
{{ range .Stats.UnrecogEg }}
|
||||
<tr><td><code>({{ .Name }})</code></td><td class="num">{{ .Marks }}</td></tr>
|
||||
{{ end }}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
|
||||
{{ template "base.html" . }}
|
||||
97
templates/preferences.html
Normal file
97
templates/preferences.html
Normal file
|
|
@ -0,0 +1,97 @@
|
|||
{{ define "title" }}Preferences - Snowden Archive{{ end }}
|
||||
{{ define "content" }}
|
||||
<h1>Preferences</h1>
|
||||
<p class="muted">Preferences are saved in browser cookies.</p>
|
||||
|
||||
{{ if .Saved }}<p class="pref-saved">Settings saved.</p>{{ end }}
|
||||
{{ if .Reset }}<p class="pref-saved">Preferences reset to defaults.</p>{{ end }}
|
||||
|
||||
<form action="/preferences/save" method="POST" class="pref-form">
|
||||
<fieldset>
|
||||
<legend>Theme</legend>
|
||||
<label><input type="radio" name="theme" value="auto" {{ if eq .Prefs.Theme "auto" }}checked{{ end }}> Auto (follow browser)</label>
|
||||
<label><input type="radio" name="theme" value="light" {{ if eq .Prefs.Theme "light" }}checked{{ end }}> Light</label>
|
||||
<label><input type="radio" name="theme" value="dark" {{ if eq .Prefs.Theme "dark" }}checked{{ end }}> Dark</label>
|
||||
<label><input type="checkbox" name="text_only" value="1" {{ if .Prefs.TextOnly }}checked{{ end }}> Text-only mode - no thumbnails or page images, links instead</label>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Browse lists</legend>
|
||||
<label>Documents per page:
|
||||
<select name="per_page">
|
||||
<option value="10" {{ if eq .Prefs.PerPage 10 }}selected{{ end }}>10</option>
|
||||
<option value="20" {{ if eq .Prefs.PerPage 20 }}selected{{ end }}>20</option>
|
||||
<option value="50" {{ if eq .Prefs.PerPage 50 }}selected{{ end }}>50</option>
|
||||
<option value="100" {{ if eq .Prefs.PerPage 100 }}selected{{ end }}>100</option>
|
||||
<option value="all" {{ if eq .Prefs.PerPage 0 }}selected{{ end }}>All (up to {{ .MaxCards }} a page)</option>
|
||||
</select>
|
||||
</label>
|
||||
<label><input type="checkbox" name="hide_classification" value="1" {{ if not .Prefs.ShowClassification }}checked{{ end }}> Hide classification and dissemination markings on document cards</label>
|
||||
<label><input type="checkbox" name="hide_journalists" value="1" {{ if not .Prefs.ShowJournalists }}checked{{ end }}> Hide journalist names on document cards</label>
|
||||
<label><input type="checkbox" name="hide_people" value="1" {{ if not .Prefs.ShowPeople }}checked{{ end }}> Hide the people line on document cards</label>
|
||||
<label><input type="checkbox" name="hide_programs" value="1" {{ if not .Prefs.ShowPrograms }}checked{{ end }}> Hide program names on document cards</label>
|
||||
<label><input type="checkbox" name="hide_summary" value="1" {{ if not .Prefs.ShowSummary }}checked{{ end }}> Hide short summaries on document cards</label>
|
||||
<label><input type="checkbox" name="hide_thumbs" value="1" {{ if not .Prefs.ShowThumbs }}checked{{ end }}> Hide thumbnails on search results and document cards</label>
|
||||
<label><input type="checkbox" name="show_coll_cols" value="1" {{ if .Prefs.ShowCollCols }}checked{{ end }}> Always show the per-type count columns on category tables <span class="muted pref-note">(Reported / SIDtoday / FOIA breakdown - otherwise behind a toggle on each table)</span></label>
|
||||
<label><input type="checkbox" name="page_preview" value="1" {{ if .Prefs.CardPagePreview }}checked{{ end }}> Page preview on card-thumbnail hover <span class="muted pref-note">(desktop only - hover a thumbnail to preview the full pages; press h / l or the arrow keys to flip pages.)</span></label>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>FOIA releases</legend>
|
||||
<label><input type="checkbox" name="include_foia" value="1" {{ if .Prefs.IncludeFOIA }}checked{{ end }}> Include FOIA releases <span class="muted pref-note">(official releases from FOIA litigation and FISC and ODNI declassifications, not Snowden documents)</span></label>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Document layout</legend>
|
||||
<label>Default view:
|
||||
<select name="doc_view">
|
||||
<option value="images" {{ if eq .Prefs.DocView "images" }}selected{{ end }}>Images only</option>
|
||||
<option value="split" {{ if eq .Prefs.DocView "split" }}selected{{ end }}>Split view (images + text)</option>
|
||||
<option value="text" {{ if eq .Prefs.DocView "text" }}selected{{ end }}>Text only</option>
|
||||
</select>
|
||||
</label>
|
||||
<label>Document info:
|
||||
<select name="info_position">
|
||||
<option value="above" {{ if not .Prefs.InfoSidebar }}selected{{ end }}>Above the document</option>
|
||||
<option value="beside" {{ if .Prefs.InfoSidebar }}selected{{ end }}>Beside the document (sidebar)</option>
|
||||
</select>
|
||||
<span class="muted pref-note">(desktop only)</span>
|
||||
</label>
|
||||
<label><input type="checkbox" name="scroll_doc" value="1" {{ if .Prefs.ScrollDoc }}checked{{ end }}> Scroll within the document panes <span class="muted pref-note">(cap the image/text area height instead of letting a long document stretch the page; ignored when the info sidebar is on)</span></label>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>Document details</legend>
|
||||
<label><input type="checkbox" name="open_info" value="1" {{ if .Prefs.OpenInfo }}checked{{ end }}> Expand “More info” by default</label>
|
||||
<label><input type="checkbox" name="collapse_pub" value="1" {{ if .Prefs.CollapsePubTogether }}checked{{ end }}> Collapse “Published in the same report” by default</label>
|
||||
<label><input type="checkbox" name="collapse_related" value="1" {{ if .Prefs.CollapseRelated }}checked{{ end }}> Collapse “Related documents” by default</label>
|
||||
<label><input type="checkbox" name="programs_list" value="1" {{ if .Prefs.ProgramsAsList }}checked{{ end }}> Show programs one per line <span class="muted pref-note">(instead of comma-separated)</span></label>
|
||||
<label><input type="checkbox" name="tags_list" value="1" {{ if .Prefs.TagsAsList }}checked{{ end }}> Show tags one per line <span class="muted pref-note">(instead of comma-separated)</span></label>
|
||||
</fieldset>
|
||||
|
||||
<fieldset>
|
||||
<legend>History <span class="muted pref-note">(off by default - nothing is recorded unless you opt in; saved in this browser's cookies; turning this off deletes the history)</span></legend>
|
||||
<label><input type="checkbox" name="track_recent" value="1" {{ if .Prefs.TrackRecent }}checked{{ end }}> Remember recently viewed documents (shown on the home page)</label>
|
||||
<label><input type="checkbox" name="track_searches" value="1" {{ if .Prefs.TrackSearches }}checked{{ end }}> Remember recent searches (shown on the search page)</label>
|
||||
</fieldset>
|
||||
|
||||
<div class="pref-actions">
|
||||
<button type="submit" class="pref-save">Save preferences</button>
|
||||
<button type="submit" name="reset" value="1" class="pref-reset">Reset to defaults</button>
|
||||
</div>
|
||||
</form>
|
||||
|
||||
<details id="custom-css-details" class="custom-css-details" hidden>
|
||||
<summary>Custom CSS <span class="muted pref-note">(for advanced users)</span></summary>
|
||||
<p class="muted pref-note">Stored only in the localStorage and applied on every page. To turn it off, empty the box and save. If a bad rule ever makes the site unusable, open any page with <code>?nocss=1</code> (e.g. <code>/preferences?nocss=1</code>).</p>
|
||||
<textarea id="custom-css" rows="8" spellcheck="false" placeholder="/* e.g. */
|
||||
.prose { font-size: 1.05rem; line-height: 1.8; }
|
||||
.doc-card { border-width: 2px; }"></textarea>
|
||||
<div class="custom-css-actions">
|
||||
<button type="button" id="custom-css-save" class="ctl">Save CSS</button>
|
||||
<span class="custom-css-status faint" id="custom-css-status"></span>
|
||||
</div>
|
||||
</details>
|
||||
|
||||
{{ end }}
|
||||
{{ template "base.html" . }}
|
||||
280
templates/search.html
Normal file
280
templates/search.html
Normal file
|
|
@ -0,0 +1,280 @@
|
|||
{{ define "title" }}{{ with .QueryLabel }}{{ . }} - {{ end }}Search - Snowden Archive{{ end }}
|
||||
{{ define "search-result-item" }}
|
||||
{{ $r := .Result }}{{ $d := $r.Doc }}
|
||||
{{ $qs := "" }}{{ if eq $d.Collection "foia" }}{{ $qs = .FOIAQS }}{{ end }}
|
||||
<li class="doc-card search-result">
|
||||
{{ if .ShowThumbs }}
|
||||
<a href="/doc/{{ $d.ID }}" class="result-thumb" tabindex="-1" aria-hidden="true"{{ if and .CardPagePreview $d.Images }} data-pp-id="{{ $d.ID }}" data-pp-imgs="{{ range $i, $im := $d.Images }}{{ if $i }} {{ end }}{{ pathEscape $im }}{{ end }}"{{ end }}>
|
||||
{{ if $d.HasThumb }}
|
||||
<img src="/files/{{ $d.ID }}/thumb.jpg"{{ if $d.HasGifThumb }} data-gif="/files/{{ $d.ID }}/thumb.gif"{{ end }}
|
||||
class="doc-img" alt="" loading="lazy">
|
||||
{{ end }}
|
||||
</a>
|
||||
{{ end }}
|
||||
<div class="result-body">
|
||||
<h3 class="result-title">
|
||||
<a href="/doc/{{ $d.ID }}" class="result-title-link">{{ $r.Title }}</a>
|
||||
<a href="/files/{{ $d.ID }}/{{ pathEscape $d.PDFFile }}" target="_blank" rel="noopener" class="doc-pdf">[pdf]</a>
|
||||
</h3>
|
||||
<span class="doc-type">{{ template "type-badge" $d.Type }}</span>
|
||||
|
||||
{{ if $d.Publication }}
|
||||
{{/* A year links to its publication-year page only when that page
|
||||
lists the document: the year of its FIRST publication. */}}
|
||||
<div class="result-pubs">
|
||||
{{- $shown := false -}}
|
||||
{{- range $pub := $d.Publication -}}
|
||||
{{- if $pub.Publisher -}}
|
||||
{{- if $shown }} · {{ end }}{{ $shown = true -}}
|
||||
{{ template "pub-publisher" (dict "Pub" $pub "QS" $qs) }}
|
||||
{{- if ge (len $pub.DatePublished) 4 }}{{ $y := slice $pub.DatePublished 0 4 }} {{ if eq $y $d.FirstPubYear }}<a href="/browse/published/{{ $y }}{{ $qs }}" class="meta-link-softer">{{ $y }}</a>{{ else }}<span class="meta-link-softer">{{ $y }}</span>{{ end }}{{ end -}}
|
||||
{{- range $j, $r := $pub.Journalists }}{{ if $j }}, {{ else }} - {{ end }}<a href="/browse/journalist/{{ pathEscape $r }}{{ $qs }}">{{ $r }}</a>{{ end -}}
|
||||
{{- end -}}
|
||||
{{- end }}
|
||||
</div>
|
||||
{{ end }}
|
||||
|
||||
<div class="result-meta">
|
||||
{{ if $d.DocMeta.Creator }}
|
||||
<a href="/browse/agency/{{ pathEscape $d.DocMeta.Creator }}{{ $qs }}" class="meta-link-soft">{{ $d.DocMeta.Creator }}</a>
|
||||
{{ end }}
|
||||
{{ if and $d.DocMeta.Creator $d.DocMeta.DateCreated }}
|
||||
<span class="doc-dot">·</span>
|
||||
{{ end }}
|
||||
{{ if $d.DocMeta.DateCreated }}
|
||||
{{ if ge (len $d.DocMeta.DateCreated) 4 }}
|
||||
<a href="/browse/docdate/{{ slice $d.DocMeta.DateCreated 0 4 }}{{ $qs }}" class="meta-link-softer">{{ $d.DocMeta.DateCreated }}</a>
|
||||
{{ else }}
|
||||
<span class="meta-link-softer">{{ $d.DocMeta.DateCreated }}</span>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
|
||||
{{ if $d.DocMeta.Programs }}
|
||||
<span class="doc-dot">·</span>
|
||||
{{ $progs := $d.DocMeta.Programs }}
|
||||
{{ $shown := $progs }}{{ if gt (len $progs) 8 }}{{ $shown = slice $progs 0 8 }}{{ end }}
|
||||
{{ facetLinks (dict "Vals" $shown "Cat" "program" "Sep" "dot" "Class" "meta-link-soft" "QS" $qs) }}
|
||||
{{ if gt (len $progs) 8 }}<span class="faint">+{{ add (len $progs) -8 }} more</span>{{ end }}
|
||||
{{ end }}
|
||||
</div>
|
||||
|
||||
<div class="result-desc">{{ $r.Desc }}</div>
|
||||
|
||||
{{ range $r.Snippets }}
|
||||
<div class="search-result-snippet">{{ . }}</div>
|
||||
{{ end }}
|
||||
{{ range $r.VersionSnippets }}
|
||||
<div class="search-result-snippet version-snippet"><a href="/doc/{{ $d.ID }}/versions/{{ .Slug }}" class="snippet-source">Copy: {{ .Label }}</a> {{ .HTML }}</div>
|
||||
{{ end }}
|
||||
{{ with $r.MatchesIn }}
|
||||
<div class="result-matches-in muted">Matches in:
|
||||
{{- range $i, $v := . }}{{ if $i }},{{ end }} <a href="/doc/{{ $d.ID }}/versions/{{ $v.Slug }}">{{ $v.Label }}</a>{{ end }}
|
||||
</div>
|
||||
{{ end }}
|
||||
</div>
|
||||
</li>
|
||||
{{ end }}
|
||||
|
||||
{{ define "content" }}
|
||||
<h1 class="search-title">Search</h1>
|
||||
|
||||
<details class="search-ops-hint">
|
||||
<summary>Search syntax help</summary>
|
||||
<p>
|
||||
Ways to write a value, in any row or after any operator:
|
||||
<ul>
|
||||
<li><code>cable tapping</code> finds both words anywhere, in their other forms too (<em>tapping</em> also finds <em>tapped</em>);</li>
|
||||
<li><code>"apping"</code> finds that text anywhere, even inside a word, like <code>grep -i</code> (it finds for example <em>mapping</em> and <em>tapping</em>);</li>
|
||||
<li><code>=tap</code> or <code>="cable tapping"</code> finds exactly those whole words, so <code>=tap</code> skips <em>taps</em> and <em>strap</em>, and on a metadata field it matches one whole value, so <code>agency:=NSA</code> finds documents whose agency is NSA but not <em>NSA, GCHQ</em>;</li>
|
||||
<li><code>*</code> wildcard finds part of a word: <code>tap*</code> finds words starting with <em>tap</em>, <code>*land</code> words ending in <em>land</em> and <code>*keys*</code> words containing <em>keys</em>. A leading <code>*</code> needs at least three letters after it.</li>
|
||||
<li><code>"upstream collection"~10</code> finds the words within 10 words of each other (in any order, common stopwords ignored).</li>
|
||||
<li>A leading <code>-</code> excludes within its row: <code>xkeyscore -tempora</code> finds <em>xkeyscore</em> without <em>tempora</em>, and the row keeps that meaning under OR and NOT.</li>
|
||||
</ul>
|
||||
</p>
|
||||
<p>
|
||||
The simple search box (on home page, on 404 page, and via URL <code>?q=</code>) understands field operators:
|
||||
<ul>
|
||||
<li><code>program:XKEYSCORE country:Germany cable</code> or <code>journalist:="Poitras, Laura"</code>.</li>
|
||||
<li>A leading <code>-</code> negates (<code>-tag:=junk</code>, <code>-draft</code>);</li>
|
||||
<li><code>date:2013</code>, <code>date:2013..2015</code>, and <code>published:2014-06</code> filter by date.</li>
|
||||
<li>Operators: text, title, content, description, agency, journalist, publisher, classification, program, dissemination, banner, genre, series, country, people, tag, type.</li>
|
||||
<li>Bare words search all fields.</li>
|
||||
</ul>
|
||||
</p>
|
||||
</details>
|
||||
|
||||
<form method="GET" action="/search" class="search-form search-panel">
|
||||
<input type="hidden" name="do_search" value="1">
|
||||
<button type="submit" name="action" value="search" aria-hidden="true" tabindex="-1" class="sr-only">Search</button>
|
||||
|
||||
<div class="query-rows">
|
||||
{{ range $i, $val := .Queries }}
|
||||
<div class="query-row">
|
||||
<div class="qr-conj">
|
||||
{{ if eq $i 0 }}
|
||||
<span class="faint">Find</span>
|
||||
<input type="hidden" name="fqc" value="and">
|
||||
{{ else }}
|
||||
<select name="fqc" aria-label="How row {{ add $i 1 }} combines with the rows above">
|
||||
<option value="and" {{ if eq (index $.Conjs $i) "and" }}selected{{ end }}>and</option>
|
||||
<option value="or" {{ if eq (index $.Conjs $i) "or" }}selected{{ end }}>or</option>
|
||||
<option value="not" {{ if eq (index $.Conjs $i) "not" }}selected{{ end }}>and not</option>
|
||||
</select>
|
||||
{{ end }}
|
||||
</div>
|
||||
<input type="text" name="fqv" value="{{ $val }}" class="qr-term" aria-label="Search terms, row {{ add $i 1 }}"
|
||||
placeholder='word, "text", =exact or *part*'>
|
||||
<div class="qr-field">
|
||||
<span class="faint qr-in">in</span>
|
||||
{{ $fi := index $.Fields $i }}
|
||||
<select name="fqf" aria-label="Field to search, row {{ add $i 1 }}">
|
||||
{{ range $.FieldOptions }}
|
||||
<option value="{{ .Code }}" {{ if eq $fi .Code }}selected{{ end }}>{{ .Label }}</option>
|
||||
{{ end }}
|
||||
</select>
|
||||
</div>
|
||||
{{ if ne $i 0 }}
|
||||
<button type="submit" name="action" value="remove-{{ $i }}"
|
||||
class="remove-row-btn" title="Remove this row">×</button>
|
||||
{{ else }}
|
||||
<span class="remove-row-spacer"></span>
|
||||
{{ end }}
|
||||
</div>
|
||||
{{ end }}
|
||||
|
||||
<div class="query-row query-row-add">
|
||||
<div class="qr-conj"></div>
|
||||
{{ if lt (len .Queries) .ClauseLimit }}
|
||||
<button type="submit" name="action" value="add" class="add-row-btn">+ Add row</button>
|
||||
{{ else }}
|
||||
<span class="faint">{{ .ClauseLimit }} rows is the most a search can have</span>
|
||||
{{ end }}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{{ $hasDates := or .DateDocStart .DateDocEnd .DatePubStart .DatePubEnd }}
|
||||
{{ $hasFilters := or $hasDates .CollFiltered }}
|
||||
<details class="date-filters"{{ if $hasFilters }} open{{ end }}>
|
||||
<summary>Filters{{ if $hasFilters }} <span class="faint">(active)</span>{{ end }}</summary>
|
||||
<div class="coll-filter-row">
|
||||
<span class="faint df-label">Collections</span>
|
||||
<label class="coll-check"><input type="checkbox" name="coll" value="news"{{ if index .SelColl "news" }} checked{{ end }}> Reported</label>
|
||||
<label class="coll-check"><input type="checkbox" name="coll" value="sidtoday-archive"{{ if index .SelColl "sidtoday-archive" }} checked{{ end }}> SIDtoday archive</label>
|
||||
<label class="coll-check"><input type="checkbox" name="coll" value="foia"{{ if index .SelColl "foia" }} checked{{ end }}> FOIA</label>
|
||||
</div>
|
||||
<div class="date-filter-rows">
|
||||
<div class="date-filter-row">
|
||||
<span class="faint df-label">Doc date</span>
|
||||
<input type="text" name="date_doc_start" value="{{ .DateDocStart }}" aria-label="Document date from" placeholder="yyyy-mm-dd">
|
||||
<span class="faint">to</span>
|
||||
<input type="text" name="date_doc_end" value="{{ .DateDocEnd }}" aria-label="Document date to" placeholder="yyyy-mm-dd">
|
||||
</div>
|
||||
<div class="date-filter-row">
|
||||
<span class="faint df-label">Published</span>
|
||||
<input type="text" name="date_pub_start" value="{{ .DatePubStart }}" aria-label="Published from" placeholder="yyyy-mm-dd">
|
||||
<span class="faint">to</span>
|
||||
<input type="text" name="date_pub_end" value="{{ .DatePubEnd }}" aria-label="Published to" placeholder="yyyy-mm-dd">
|
||||
</div>
|
||||
</div>
|
||||
</details>
|
||||
|
||||
<div class="search-actions">
|
||||
<button type="submit" name="action" value="search" class="search-btn">Search</button>
|
||||
<a href="/search" class="faint reset-link">Reset</a>
|
||||
</div>
|
||||
</form>
|
||||
|
||||
{{ if .RecentSearches }}
|
||||
<details class="recent-searches">
|
||||
<summary>Recent searches <span class="faint">({{ len .RecentSearches }})</span></summary>
|
||||
<div class="recent-list">
|
||||
{{ range .RecentSearches }}<a class="pill" href="{{ .URL }}">{{ .Label }}</a>{{ end }}
|
||||
<form method="POST" action="/history/clear" class="inline-form">
|
||||
<input type="hidden" name="which" value="searches">
|
||||
<input type="hidden" name="back" value="/search">
|
||||
<button type="submit" class="link-btn faint">clear</button>
|
||||
</form>
|
||||
</div>
|
||||
</details>
|
||||
{{ end }}
|
||||
|
||||
<hr class="hr-soft">
|
||||
|
||||
{{ if .OverClauses }}
|
||||
<p class="muted"><strong>Search not run:</strong> it has {{ .OverClauses }} parts - rows, or the
|
||||
<code>field:</code>, “quoted” and <code>-excluded</code> terms of a typed query -
|
||||
and a search can have at most {{ .ClauseLimit }}. The form above keeps the first {{ .ClauseLimit }};
|
||||
the rest, including any exclusions at the end, were left out. Check the rows and search again.</p>
|
||||
{{ else if .OverWords }}
|
||||
<p class="muted"><strong>Search not run:</strong> it has {{ .OverWords }} words, counting the words of
|
||||
“quoted” phrases and <code>-excluded</code> terms, and a search can have at most
|
||||
{{ .WordLimit }}. Shorten it and search again.</p>
|
||||
{{ end }}
|
||||
|
||||
{{ if .Searched }}
|
||||
{{ if gt (len .SearchTypeCounts) 1 }}{{ template "type-filter" (dict "Action" "/search" "Params" .FormParams "AllTypes" .SearchTypeCounts "SelectedTypes" .SelTypeSet) }}{{ end }}
|
||||
<div class="results-bar">
|
||||
<h2 class="results-heading">{{ .Total }} result{{ if ne .Total 1 }}s{{ end }}</h2>
|
||||
{{ template "sort-bar" (dict "SortURL" .SearchBaseURL "SortBy" .Sort "Order" .Order "ShowRelevance" true "DefaultSort" "relevance") }}
|
||||
{{ if .Savable }}
|
||||
<form method="POST" action="/pinsearch" class="inline-form">
|
||||
<input type="hidden" name="qs" value="{{ .CanonicalQS }}">
|
||||
<input type="hidden" name="back" value="{{ .SearchBaseURL }}">
|
||||
<button type="submit" class="export-link pin-search-btn" title="{{ if .SearchPinned }}Remove this search from your pinned list{{ else }}Pin this search to your pinned list{{ end }}">{{ if .SearchPinned }}★ unpin search{{ else }}☆ pin search{{ end }}</button>
|
||||
</form>
|
||||
{{ end }}
|
||||
</div>
|
||||
|
||||
{{ if gt (len .DateHist) 1 }}
|
||||
<div class="date-hist" aria-label="Matches per document year">
|
||||
{{ range .DateHist }}
|
||||
{{ if .URL }}
|
||||
<a class="hist-col" href="{{ .URL }}" title="{{ .Count }} matches dated {{ .Year }}">
|
||||
<span class="hist-count">{{ .Count }}</span>
|
||||
<span class="hist-track"><span class="hist-bar" style="height: {{ .Pct }}%"></span></span>
|
||||
<span class="hist-year">{{ .Year }}</span>
|
||||
</a>
|
||||
{{ else }}
|
||||
<span class="hist-col hist-col-undated" title="{{ .Count }} matches with no document date">
|
||||
<span class="hist-count">{{ .Count }}</span>
|
||||
<span class="hist-track"><span class="hist-bar" style="height: {{ .Pct }}%"></span></span>
|
||||
<span class="hist-year">{{ .Year }}</span>
|
||||
</span>
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
</div>
|
||||
{{ end }}
|
||||
|
||||
{{ if .Results }}
|
||||
{{ if .Facets }}
|
||||
<details class="search-facets">
|
||||
<summary>Refine results</summary>
|
||||
<div class="facets-body">
|
||||
{{ range .Facets }}
|
||||
<div class="facet-group">
|
||||
<span class="facet-label">{{ .Label }}</span>
|
||||
<span class="facet-values">{{ range .Values }}<a class="facet-link" href="{{ .URL }}">{{ .Name }}<span class="facet-count">{{ .Count }}</span></a>{{ end }}</span>
|
||||
</div>
|
||||
{{ end }}
|
||||
</div>
|
||||
</details>
|
||||
{{ end }}
|
||||
<ul class="search-results{{ if .ShowThumbs }} has-thumbs{{ end }}">
|
||||
{{ $showThumbs := .ShowThumbs }}
|
||||
{{ range .Results }}{{ template "search-result-item" (dict "Result" . "ShowThumbs" $showThumbs "CardPagePreview" $.CardPagePreview "FOIAQS" $.FOIAQS) }}{{ end }}
|
||||
</ul>
|
||||
{{ if .SnippetsCapped }}
|
||||
<p class="muted">Text snippets are shown for the first {{ .SnippetLimit }} results on this page -
|
||||
choose a page size in <a href="/preferences">preferences</a>, or narrow the search, to see
|
||||
snippets for the rest.</p>
|
||||
{{ end }}
|
||||
{{ template "pagination" (dict "Page" .Page "TotalPages" .TotalPages "BaseURL" .SearchBaseURL "Sort" .Sort "Order" .Order) }}
|
||||
{{ else }}
|
||||
<p class="muted">No matches found.</p>
|
||||
{{ end }}
|
||||
|
||||
{{ end }}
|
||||
|
||||
{{ end }}
|
||||
|
||||
{{ template "base.html" . }}
|
||||
59
templates/version.html
Normal file
59
templates/version.html
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
{{ define "title" }}{{ .Doc.Title }} ({{ .Version.Slug }}) - Snowden Archive{{ end }}
|
||||
|
||||
{{ define "content" }}
|
||||
<header class="list-header">
|
||||
<p class="crumb"><a href="/doc/{{ .Doc.ID }}">← {{ .Doc.Title }}</a></p>
|
||||
<h1>{{ .Doc.Title }} <small class="count-note">copy: {{ .Version.Slug }}</small></h1>
|
||||
<p class="muted">Another published copy of this document that differs from the one on the
|
||||
<a href="/doc/{{ .Doc.ID }}">document page</a> - for example less redacted, with extra pages, or a different scan.</p>
|
||||
</header>
|
||||
|
||||
<div class="version-actions">
|
||||
<a href="{{ .PDFURL }}" target="_blank" rel="noopener" class="doc-top-btn doc-pdf-link">Download this copy (PDF)</a>
|
||||
{{ with .Version.Images }}<a href="/doc/{{ $.Doc.ID }}/versions/{{ $.Version.Slug }}/pages" class="doc-top-btn">All pages ({{ len . }})</a>{{ end }}
|
||||
</div>
|
||||
{{ if .Version.ChangedPages }}
|
||||
<p class="changed-pages-note">Pages that differ from the main document: {{ template "changed-page-links" .Version }}</p>
|
||||
{{ end }}
|
||||
|
||||
<div class="rule-top pub-block">
|
||||
{{- range $i, $pub := .Publications }}
|
||||
<div class="pub-entry{{ if $i }} pub-sep{{ end }}">
|
||||
<div class="pub-title">
|
||||
{{- with $pub.Publisher }}{{ . }}{{ else }}Unknown publisher{{ end -}}
|
||||
{{- with $pub.DatePublished }} <span class="muted">{{ . }}</span>{{ end -}}
|
||||
</div>
|
||||
{{- if $pub.Title }}
|
||||
<div class="pub-people"><em>{{ $pub.Title }}</em>{{ with $pub.Link }} <a href="{{ . }}" target="_blank" rel="noopener" class="pub-original">article ↗</a>{{ end }}</div>
|
||||
{{- else if $pub.Link }}
|
||||
<div class="pub-people"><a href="{{ $pub.Link }}" target="_blank" rel="noopener" class="pub-original">article ↗</a></div>
|
||||
{{- end }}
|
||||
{{- with $pub.Note }}
|
||||
<div class="pub-note">{{ . }}</div>
|
||||
{{- end }}
|
||||
{{- if or $pub.DocSource $pub.DocURL }}
|
||||
<div class="pub-copy muted">
|
||||
{{- with $pub.DocSource }}<a href="{{ . }}" target="_blank" rel="noopener">document page ↗</a>{{ end -}}
|
||||
{{- with $pub.DocURL }}<a href="{{ . }}" target="_blank" rel="noopener">original PDF ↗</a>{{ end -}}
|
||||
</div>
|
||||
{{- end }}
|
||||
</div>
|
||||
{{- else }}
|
||||
<p class="muted">No publication record points at this copy.</p>
|
||||
{{- end }}
|
||||
</div>
|
||||
|
||||
{{ if or .Images .Version.HTMLContent }}
|
||||
<div class="doc-container">
|
||||
{{ template "doc-viewport" . }}
|
||||
</div>
|
||||
{{ else }}
|
||||
<p class="muted rule-top">This copy has no transcript and no page images - only the PDF above.</p>
|
||||
{{ end }}
|
||||
|
||||
{{ if and .Images (not .Prefs.TextOnly) }}
|
||||
{{ template "lightbox-overlays" . }}
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
|
||||
{{ template "base.html" . }}
|
||||
122
tests/api_test.go
Normal file
122
tests/api_test.go
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestAPIPageSize(t *testing.T) {
|
||||
for raw, want := range map[string]int{
|
||||
"": apiPerPageDefault,
|
||||
"25": 25,
|
||||
"500": 500,
|
||||
"1000": apiMaxPerPage,
|
||||
"99999999": apiMaxPerPage,
|
||||
"0": apiPerPageDefault, // not a page size; fall back
|
||||
"-5": apiPerPageDefault,
|
||||
"lots": apiPerPageDefault,
|
||||
"all": 0,
|
||||
} {
|
||||
v := url.Values{}
|
||||
if raw != "" {
|
||||
v.Set("per_page", raw)
|
||||
}
|
||||
if got := apiPageSize(v); got != want {
|
||||
t.Errorf("apiPageSize(per_page=%q) = %d, want %d", raw, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPIRespPerPage(t *testing.T) {
|
||||
for _, c := range []struct{ perPage, total, want int }{
|
||||
{50, 3106, 50},
|
||||
{0, 3106, 3106},
|
||||
{0, 0, 1},
|
||||
} {
|
||||
if got := apiRespPerPage(c.perPage, c.total); got != c.want {
|
||||
t.Errorf("apiRespPerPage(%d, %d) = %d, want %d", c.perPage, c.total, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPIVersionsShape(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
writeTestFile(t, filepath.Join(root, "doc-a", "metadata.json"), `{
|
||||
"title": "A", "type": "REPORTED",
|
||||
"publication": [{"publisher": "Le Monde", "version": "entier"}],
|
||||
"versions": {"entier": {"changed_pages": [1]}}
|
||||
}`)
|
||||
writeTestFile(t, filepath.Join(root, "doc-a", "a.pdf"), "%PDF-1.4")
|
||||
writeTestFile(t, filepath.Join(root, "doc-a", "versions", "entier", "entier.pdf"), "%PDF-1.4")
|
||||
withCorpus(t, root)
|
||||
|
||||
type versionsObj struct {
|
||||
Versions map[string]VersionInfo `json:"versions"`
|
||||
}
|
||||
var detail struct {
|
||||
versionsObj
|
||||
VersionCopies []apiVersion `json:"version_copies"`
|
||||
}
|
||||
decode := func(path string, v interface{}) {
|
||||
t.Helper()
|
||||
w := get(t, path)
|
||||
if w.Code != 200 {
|
||||
t.Fatalf("%s: %d", path, w.Code)
|
||||
}
|
||||
if err := json.Unmarshal(w.Body.Bytes(), v); err != nil {
|
||||
t.Fatalf("%s: %v", path, err)
|
||||
}
|
||||
}
|
||||
decode("/api/doc/doc-a", &detail)
|
||||
if len(detail.Versions["entier"].ChangedPages) != 1 {
|
||||
t.Errorf("/api/doc versions = %+v, want the metadata object", detail.Versions)
|
||||
}
|
||||
if len(detail.VersionCopies) != 1 || detail.VersionCopies[0].Slug != "entier" {
|
||||
t.Errorf("/api/doc version_copies = %+v, want the entier copy", detail.VersionCopies)
|
||||
}
|
||||
var list struct{ Docs []versionsObj }
|
||||
decode("/api/docs", &list)
|
||||
var search struct{ Results []versionsObj }
|
||||
decode("/api/search?q=A", &search)
|
||||
for name, docs := range map[string][]versionsObj{"/api/docs": list.Docs, "/api/search": search.Results} {
|
||||
if len(docs) != 1 || len(docs[0].Versions["entier"].ChangedPages) != 1 {
|
||||
t.Errorf("%s versions = %+v, want the metadata object", name, docs)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPIFacetsRevalidates(t *testing.T) {
|
||||
w := get(t, "/api/facets")
|
||||
tag := w.Header().Get("ETag")
|
||||
if w.Code != 200 || tag == "" || w.Header().Get("Access-Control-Allow-Origin") != "*" {
|
||||
t.Fatalf("/api/facets: %d, ETag %q, CORS %q", w.Code, tag, w.Header().Get("Access-Control-Allow-Origin"))
|
||||
}
|
||||
mux, _ := buildFixture(t)
|
||||
r := httptest.NewRequest("GET", "/api/facets", nil)
|
||||
r.Header.Set("If-None-Match", tag)
|
||||
if w := record(mux, r); w.Code != http.StatusNotModified || w.Body.Len() != 0 {
|
||||
t.Errorf("revalidation: %d with %d bytes, want 304 and none", w.Code, w.Body.Len())
|
||||
}
|
||||
}
|
||||
|
||||
// /api/docs takes coll= as /api/search and the pages do.
|
||||
func TestAPIDocsCollections(t *testing.T) {
|
||||
total := func(query string) int {
|
||||
var body struct{ Total int }
|
||||
if err := json.Unmarshal(get(t, "/api/docs?"+query).Body.Bytes(), &body); err != nil {
|
||||
t.Fatalf("%s: %v", query, err)
|
||||
}
|
||||
return body.Total
|
||||
}
|
||||
all, news, foia := total(""), total("coll=news"), total("coll=foia")
|
||||
if news == 0 || foia == 0 || news+foia >= all {
|
||||
t.Fatalf("fixture totals: %d documents, %d news, %d foia", all, news, foia)
|
||||
}
|
||||
if got := total("coll=news&coll=foia"); got != news+foia {
|
||||
t.Errorf("coll=news&coll=foia lists %d, want %d", got, news+foia)
|
||||
}
|
||||
}
|
||||
211
tests/assemblies_test.go
Normal file
211
tests/assemblies_test.go
Normal file
|
|
@ -0,0 +1,211 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func writeAssemblyDoc(t *testing.T, root, id string, meta map[string]interface{}, pages []string, versions map[string][]string) {
|
||||
t.Helper()
|
||||
dir := filepath.Join(root, id)
|
||||
b, err := json.Marshal(meta)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writeFixtureFile(t, filepath.Join(dir, "metadata.json"), string(b))
|
||||
writeFixtureFile(t, filepath.Join(dir, id+".pdf"), "%PDF-1.4\n")
|
||||
for _, p := range pages {
|
||||
writeFixtureFile(t, filepath.Join(dir, "images", p), fixtureJPEG)
|
||||
}
|
||||
for slug, imgs := range versions {
|
||||
v := filepath.Join(dir, "versions", slug)
|
||||
writeFixtureFile(t, filepath.Join(v, slug+".pdf"), "%PDF-1.4\n")
|
||||
for _, p := range imgs {
|
||||
writeFixtureFile(t, filepath.Join(v, "images", p), fixtureJPEG)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func assemblyCorpus(t *testing.T, manifest string) string {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
writeAssemblyDoc(t, root, "deck", map[string]interface{}{
|
||||
"title": "The Deck",
|
||||
"publication": []map[string]interface{}{
|
||||
{"publisher": "Outlet A"},
|
||||
{"publisher": "Partner", "reporting_only": true},
|
||||
{"publisher": "Outlet B", "version": "fuller"},
|
||||
},
|
||||
}, []string{"page-1.jpg", "page-2.jpg", "page-3.jpg"}, map[string][]string{"fuller": {"page-1.jpg", "page-2.jpg"}})
|
||||
writeAssemblyDoc(t, root, "single", map[string]interface{}{
|
||||
"title": "One Slide",
|
||||
"publication": []map[string]interface{}{{"publisher": "Outlet C"}},
|
||||
}, []string{"page-1.jpg"}, nil)
|
||||
writeAssemblyDoc(t, root, "bystander", map[string]interface{}{"title": "Bystander"}, []string{"page-1.jpg"}, nil)
|
||||
if manifest != "" {
|
||||
writeFixtureFile(t, filepath.Join(root, "assemblies.json"), manifest)
|
||||
}
|
||||
return root
|
||||
}
|
||||
|
||||
const assemblyManifest = `[{
|
||||
"id": "set", "title": "A Set", "note": "Order is a guess.",
|
||||
"slides": [
|
||||
{"doc": "deck", "pages": "1-2", "alt": [{"doc": "deck", "version": "fuller", "pages": "1-2"}]},
|
||||
{"doc": "deck", "version": "fuller", "pages": "2", "note": "Only here."},
|
||||
{"doc": "single", "pages": "1"}
|
||||
]
|
||||
}]`
|
||||
|
||||
func TestParsePageList(t *testing.T) {
|
||||
for in, want := range map[string][]int{
|
||||
"8": {8}, "1-3": {1, 2, 3}, "1,3,5-6": {1, 3, 5, 6}, " 2 , 4 ": {2, 4}, "9,1": {9, 1},
|
||||
} {
|
||||
got, err := parsePageList(in)
|
||||
if err != nil || !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("parsePageList(%q) = %v, %v; want %v", in, got, err, want)
|
||||
}
|
||||
}
|
||||
for _, in := range []string{"", "a", "0", "3-1", "1-", "1,,2"} {
|
||||
if got, err := parsePageList(in); err == nil {
|
||||
t.Errorf("parsePageList(%q) = %v, want an error", in, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssemblyResolves(t *testing.T) {
|
||||
withCorpus(t, assemblyCorpus(t, assemblyManifest))
|
||||
a := corpus().assemblyIndex["set"]
|
||||
if a == nil {
|
||||
t.Fatal("assembly not loaded")
|
||||
}
|
||||
if len(a.Slides) != 4 {
|
||||
t.Fatalf("%d slides, want 4", len(a.Slides))
|
||||
}
|
||||
for i, s := range a.Slides {
|
||||
if s.Pos != i+1 {
|
||||
t.Errorf("slide %d has Pos %d", i+1, s.Pos)
|
||||
}
|
||||
}
|
||||
first := a.Slides[0]
|
||||
if first.Image != "/files/deck/images/page-1.jpg" || first.Source != "Outlet A" {
|
||||
t.Errorf("first slide %+v", first)
|
||||
}
|
||||
if len(first.Alts) != 1 || first.Alts[0].Image != "/files/deck/versions/fuller/images/page-1.jpg" || first.Alts[0].Source != "Outlet B" {
|
||||
t.Errorf("first slide alts %+v", first.Alts)
|
||||
}
|
||||
if a.Slides[2].Note != "Only here." || a.Slides[2].Path != "/doc/deck/versions/fuller" {
|
||||
t.Errorf("third slide %+v", a.Slides[2])
|
||||
}
|
||||
if fmt.Sprint(a.DocList) != "[deck single]" {
|
||||
t.Errorf("DocList %v", a.DocList)
|
||||
}
|
||||
if a.Copies != 3 {
|
||||
t.Errorf("Copies %d, want 3", a.Copies)
|
||||
}
|
||||
if len(corpus().assembliesByDoc["single"]) != 1 || len(corpus().assembliesByDoc["bystander"]) != 0 {
|
||||
t.Errorf("assembliesByDoc %v", corpus().assembliesByDoc)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssemblyPages(t *testing.T) {
|
||||
withCorpus(t, assemblyCorpus(t, assemblyManifest))
|
||||
h := withBaseHeaders(newMux())
|
||||
body := func(path string, code int) string {
|
||||
t.Helper()
|
||||
w := record(h, httptest_NewGet(path))
|
||||
if w.Code != code {
|
||||
t.Fatalf("%s: %d, want %d", path, w.Code, code)
|
||||
}
|
||||
return w.Body.String()
|
||||
}
|
||||
list := body("/explore/assemblies", 200)
|
||||
if !strings.Contains(list, `href="/explore/assemblies/set"`) || !strings.Contains(list, "4 slides from 3 copies") {
|
||||
t.Errorf("/explore/assemblies:\n%s", list)
|
||||
}
|
||||
page := body("/explore/assemblies/set", 200)
|
||||
for _, want := range []string{
|
||||
"<title>A Set - Snowden Archive</title>", "from 3 separately published copies.", "Order is a guess.", "not a document in its own right",
|
||||
`id="lb-4"`, `href="#lb-a1-1"`, `id="lb-a1-1"`, "Other copy: Outlet B", "Back to slide 1", "(copy: fuller)", "Outlet C", "· Only here.",
|
||||
} {
|
||||
if !strings.Contains(page, want) {
|
||||
t.Errorf("/explore/assemblies/set lacks %q", want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(page, `id="lb-5"`) || strings.Contains(page, "Partner") {
|
||||
t.Errorf("/explore/assemblies/set has a fifth overlay or a reporting-only credit")
|
||||
}
|
||||
if !strings.Contains(body("/doc/single", 200), `href="/explore/assemblies/set"`) {
|
||||
t.Errorf("/doc/single has no assembly link")
|
||||
}
|
||||
if strings.Contains(body("/doc/bystander", 200), "/explore/assemblies/") {
|
||||
t.Errorf("/doc/bystander links an assembly")
|
||||
}
|
||||
// Assemblies live under /explore only: listed there, not on /browse.
|
||||
if !strings.Contains(body("/explore", 200), `href="/explore/assemblies"`) {
|
||||
t.Errorf("/explore has no assemblies link")
|
||||
}
|
||||
if strings.Contains(body("/browse", 200), "assemblies") {
|
||||
t.Errorf("/browse still links assemblies")
|
||||
}
|
||||
body("/explore/assemblies/nope", 404)
|
||||
if !slicesContains(corpus().sitemapPaths, "/explore/assemblies/set") {
|
||||
t.Errorf("sitemap lacks the assembly page")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssemblyProblemsWarn(t *testing.T) {
|
||||
cases := map[string]struct {
|
||||
manifest string
|
||||
slides int
|
||||
loaded bool
|
||||
}{
|
||||
"missing doc": {`[{"id":"x","title":"X","slides":[{"doc":"ghost","pages":"1"},{"doc":"single","pages":"1"}]}]`, 1, true},
|
||||
"missing page": {`[{"id":"x","title":"X","slides":[{"doc":"deck","pages":"3-4"}]}]`, 1, true},
|
||||
"bad version": {`[{"id":"x","title":"X","slides":[{"doc":"deck","version":"no","pages":"1"},{"doc":"single","pages":"1"}]}]`, 1, true},
|
||||
"bad pages": {`[{"id":"x","title":"X","slides":[{"doc":"deck","pages":"two"},{"doc":"single","pages":"1"}]}]`, 1, true},
|
||||
"alt mismatch": {`[{"id":"x","title":"X","slides":[{"doc":"deck","pages":"1-2","alt":[{"doc":"single","pages":"1"}]}]}]`, 2, true},
|
||||
"nothing left": {`[{"id":"x","title":"X","slides":[{"doc":"ghost","pages":"1"}]}]`, 0, false},
|
||||
"unsafe id": {`[{"id":"a/b","title":"X","slides":[{"doc":"single","pages":"1"}]}]`, 0, false},
|
||||
"malformed": {`[{`, 0, false},
|
||||
}
|
||||
for name, c := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
withCorpus(t, assemblyCorpus(t, c.manifest))
|
||||
if loadWarnings == 0 {
|
||||
t.Errorf("no warning")
|
||||
}
|
||||
a := corpus().assemblyIndex["x"]
|
||||
if (a != nil) != c.loaded {
|
||||
t.Fatalf("loaded = %v, want %v", a != nil, c.loaded)
|
||||
}
|
||||
if a != nil && len(a.Slides) != c.slides {
|
||||
t.Errorf("%d slides, want %d", len(a.Slides), c.slides)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestNoAssembliesFile(t *testing.T) {
|
||||
withCorpus(t, assemblyCorpus(t, ""))
|
||||
if len(corpus().assemblies) != 0 {
|
||||
t.Fatalf("assemblies %v", corpus().assemblies)
|
||||
}
|
||||
h := withBaseHeaders(newMux())
|
||||
if w := record(h, httptest_NewGet("/explore/assemblies")); w.Code != 200 {
|
||||
t.Errorf("/explore/assemblies: %d", w.Code)
|
||||
}
|
||||
if w := record(h, httptest_NewGet("/explore")); strings.Contains(w.Body.String(), `href="/explore/assemblies"`) {
|
||||
t.Errorf("/explore links assemblies with none loaded")
|
||||
}
|
||||
if slicesContains(corpus().sitemapPaths, "/explore/assemblies") {
|
||||
t.Errorf("sitemap lists assemblies with none loaded")
|
||||
}
|
||||
}
|
||||
|
||||
func slicesContains(s []string, v string) bool { return slices.Contains(s, v) }
|
||||
312
tests/bench_test.go
Normal file
312
tests/bench_test.go
Normal file
|
|
@ -0,0 +1,312 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"runtime"
|
||||
"sort"
|
||||
"strconv"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Page benchmarks against a REAL corpus - the fixture is four documents, so
|
||||
// it cannot show what 3,000 do. Point $SNOWCHIVE_BENCH_ROOT at an archive
|
||||
// root (the directory holding data/ and project/) and run:
|
||||
// tests/run.sh -run XXX -bench . -benchtime 40x
|
||||
//
|
||||
// What they are for: every page but /search is built from data indexed at
|
||||
// load, so a request should cost what its OUTPUT costs (template execution)
|
||||
// and nothing more. A page whose time or allocations grow with the CORPUS
|
||||
// rather than with the page has picked up a scan.
|
||||
var (
|
||||
benchOnce sync.Once
|
||||
benchMux http.Handler
|
||||
)
|
||||
|
||||
func benchCorpus(tb testing.TB) http.Handler {
|
||||
root := os.Getenv("SNOWCHIVE_BENCH_ROOT")
|
||||
if root == "" {
|
||||
tb.Skip("set SNOWCHIVE_BENCH_ROOT to an archive root to run the page benchmarks")
|
||||
}
|
||||
benchOnce.Do(func() {
|
||||
setRoot(root)
|
||||
devMode = false
|
||||
apiEnabled = true
|
||||
initAssets()
|
||||
if err := parseTemplates(); err != nil {
|
||||
tb.Fatal(err)
|
||||
}
|
||||
loadData()
|
||||
benchMux = withBaseHeaders(newMux())
|
||||
})
|
||||
return benchMux
|
||||
}
|
||||
|
||||
func benchPage(b *testing.B, path string) {
|
||||
h := benchCorpus(b)
|
||||
b.ReportAllocs()
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
r := httptest.NewRequest("GET", path, nil)
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, r)
|
||||
if w.Code != 200 {
|
||||
b.Fatalf("%s: %d", path, w.Code)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func BenchmarkPageHome(b *testing.B) { benchPage(b, "/") }
|
||||
func BenchmarkPageBrowse(b *testing.B) { benchPage(b, "/browse") }
|
||||
func BenchmarkPageArchiveTab(b *testing.B) { benchPage(b, "/browse?collection=sidtoday-archive") }
|
||||
func BenchmarkPageCategoryList(b *testing.B) { benchPage(b, "/browse/program") }
|
||||
func BenchmarkPageCategoryValue(b *testing.B) { benchPage(b, "/browse/agency/NSA") }
|
||||
func BenchmarkPageDoc(b *testing.B) {
|
||||
h := benchCorpus(b)
|
||||
if len(corpus().archive.Docs) == 0 {
|
||||
b.Skip("empty corpus")
|
||||
}
|
||||
benchPageWith(b, h, "/doc/"+corpus().archive.Docs[0].ID)
|
||||
}
|
||||
func BenchmarkPageExploreIDs(b *testing.B) { benchPage(b, "/explore/ids") }
|
||||
func BenchmarkPageGlossary(b *testing.B) { benchPage(b, "/glossary") }
|
||||
func BenchmarkPageSitemap(b *testing.B) { benchPage(b, "/sitemap.xml") }
|
||||
func BenchmarkPageFacetsAPI(b *testing.B) { benchPage(b, "/api/facets") }
|
||||
func BenchmarkPageSearch(b *testing.B) { benchPage(b, "/search?q=nsa&do_search=1") }
|
||||
|
||||
func benchPageWith(b *testing.B, h http.Handler, path string) {
|
||||
b.ReportAllocs()
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
r := httptest.NewRequest("GET", path, nil)
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, r)
|
||||
if w.Code != 200 {
|
||||
b.Fatalf("%s: %d", path, w.Code)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func BenchmarkPageCategoryTag(b *testing.B) { benchPage(b, "/browse/tag") }
|
||||
func BenchmarkPageCategoryPeople(b *testing.B) { benchPage(b, "/browse/people") }
|
||||
func BenchmarkPageCategoryTagSorted(b *testing.B) { benchPage(b, "/browse/tag?sort=count") }
|
||||
func BenchmarkPageExploreMetric(b *testing.B) { benchPage(b, "/explore/pages") }
|
||||
func BenchmarkPageCodewords(b *testing.B) { benchPage(b, "/explore/terms") }
|
||||
func BenchmarkPageOutlets(b *testing.B) { benchPage(b, "/explore/outlets") }
|
||||
func BenchmarkPageClassPatterns(b *testing.B) { benchPage(b, "/explore/classification") }
|
||||
func BenchmarkPageArticles(b *testing.B) { benchPage(b, "/articles") }
|
||||
func BenchmarkPageChangelog(b *testing.B) { benchPage(b, "/changelog") }
|
||||
func BenchmarkPageAbout(b *testing.B) { benchPage(b, "/about") }
|
||||
func BenchmarkPageAPIDocs(b *testing.B) { benchPage(b, "/api/docs") }
|
||||
func BenchmarkPageExploreIndex(b *testing.B) { benchPage(b, "/explore") }
|
||||
func BenchmarkPageContactSheet(b *testing.B) {
|
||||
h := benchCorpus(b)
|
||||
id := ""
|
||||
for i := range corpus().archive.Docs {
|
||||
if len(corpus().archive.Docs[i].Images) > 40 {
|
||||
id = corpus().archive.Docs[i].ID
|
||||
break
|
||||
}
|
||||
}
|
||||
if id == "" {
|
||||
b.Skip("no document with many pages")
|
||||
}
|
||||
benchPageWith(b, h, "/doc/"+id+"/pages")
|
||||
}
|
||||
|
||||
// Pathological documents: the biggest thing in the corpus is what breaks
|
||||
// first. What IS the biggest thing?
|
||||
func TestProbeExtremes(t *testing.T) {
|
||||
if os.Getenv("SNOWCHIVE_BENCH_ROOT") == "" {
|
||||
t.Skip("needs a real corpus")
|
||||
}
|
||||
benchCorpus(t)
|
||||
var maxImgs, maxText, maxPubs, maxVersions *Document
|
||||
for i := range corpus().archive.Docs {
|
||||
d := &corpus().archive.Docs[i]
|
||||
if maxImgs == nil || len(d.Images) > len(maxImgs.Images) {
|
||||
maxImgs = d
|
||||
}
|
||||
if maxText == nil || len(d.FullText) > len(maxText.FullText) {
|
||||
maxText = d
|
||||
}
|
||||
if maxPubs == nil || len(d.Publication) > len(maxPubs.Publication) {
|
||||
maxPubs = d
|
||||
}
|
||||
if maxVersions == nil || len(d.Versions) > len(maxVersions.Versions) {
|
||||
maxVersions = d
|
||||
}
|
||||
}
|
||||
h := benchCorpus(t)
|
||||
timeIt := func(p string) time.Duration {
|
||||
start := time.Now()
|
||||
r := httptest.NewRequest("GET", p, nil)
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, r)
|
||||
return time.Since(start)
|
||||
}
|
||||
fmt.Printf("most pages: %d %s doc %s, contact sheet %s\n", len(maxImgs.Images), trunc30(maxImgs.ID),
|
||||
timeIt("/doc/"+maxImgs.ID).Round(time.Millisecond), timeIt("/doc/"+maxImgs.ID+"/pages").Round(time.Millisecond))
|
||||
fmt.Printf("most text: %d KB %s doc page %s\n", len(maxText.FullText)/1024, trunc30(maxText.ID),
|
||||
timeIt("/doc/"+maxText.ID).Round(time.Millisecond))
|
||||
fmt.Printf("most pubs: %d %s\n", len(maxPubs.Publication), trunc30(maxPubs.ID))
|
||||
fmt.Printf("most copies: %d %s\n", len(maxVersions.Versions), trunc30(maxVersions.ID))
|
||||
}
|
||||
|
||||
// Distribution, not the median: which document pages are slow, and why?
|
||||
func TestProbeDocPageDistribution(t *testing.T) {
|
||||
if os.Getenv("SNOWCHIVE_BENCH_ROOT") == "" {
|
||||
t.Skip("needs a real corpus")
|
||||
}
|
||||
h := benchCorpus(t)
|
||||
totalImgs := 0
|
||||
for i := range corpus().archive.Docs {
|
||||
totalImgs += len(corpus().archive.Docs[i].Images)
|
||||
}
|
||||
var ds []time.Duration
|
||||
for i := 0; i < len(corpus().archive.Docs); i += 7 { // a sample, every 7th
|
||||
d := &corpus().archive.Docs[i]
|
||||
start := time.Now()
|
||||
r := httptest.NewRequest("GET", "/doc/"+d.ID, nil)
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, r)
|
||||
ds = append(ds, time.Since(start))
|
||||
}
|
||||
sort.Slice(ds, func(i, j int) bool { return ds[i] < ds[j] })
|
||||
p := func(q float64) time.Duration { return ds[int(float64(len(ds)-1)*q)] }
|
||||
fmt.Printf("%d page images in the corpus (%d docs sampled)\n", totalImgs, len(ds))
|
||||
fmt.Printf("doc page: p50 %s p90 %s p99 %s max %s\n",
|
||||
p(0.5).Round(time.Microsecond), p(0.9).Round(time.Microsecond),
|
||||
p(0.99).Round(time.Millisecond), ds[len(ds)-1].Round(time.Millisecond))
|
||||
}
|
||||
|
||||
func TestProbeWorstDocSplit(t *testing.T) {
|
||||
if os.Getenv("SNOWCHIVE_BENCH_ROOT") == "" {
|
||||
t.Skip("needs a real corpus")
|
||||
}
|
||||
h := benchCorpus(t)
|
||||
var worst *Document
|
||||
for i := range corpus().archive.Docs {
|
||||
if worst == nil || len(corpus().archive.Docs[i].Images) > len(worst.Images) {
|
||||
worst = &corpus().archive.Docs[i]
|
||||
}
|
||||
}
|
||||
run := func(path string, cookies ...*http.Cookie) (time.Duration, int) {
|
||||
var best time.Duration
|
||||
var size int
|
||||
for i := 0; i < 5; i++ {
|
||||
r := httptest.NewRequest("GET", path, nil)
|
||||
for _, c := range cookies {
|
||||
r.AddCookie(c)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
start := time.Now()
|
||||
h.ServeHTTP(w, r)
|
||||
d := time.Since(start)
|
||||
if i == 0 || d < best {
|
||||
best = d
|
||||
}
|
||||
size = w.Body.Len()
|
||||
}
|
||||
return best, size
|
||||
}
|
||||
full, fullSize := run("/doc/" + worst.ID)
|
||||
textOnly, textSize := run("/doc/"+worst.ID, &http.Cookie{Name: "pref_text_only", Value: "1"})
|
||||
sheet, sheetSize := run("/doc/" + worst.ID + "/pages")
|
||||
fmt.Printf("worst doc: %d images, %d KB transcript\n", len(worst.Images), len(worst.FullText)/1024)
|
||||
fmt.Printf(" full page %6s %d KB\n", full.Round(time.Microsecond), fullSize/1024)
|
||||
fmt.Printf(" text-only %6s %d KB (no image grid, no lightbox)\n", textOnly.Round(time.Microsecond), textSize/1024)
|
||||
fmt.Printf(" contact sheet %6s %d KB\n", sheet.Round(time.Microsecond), sheetSize/1024)
|
||||
}
|
||||
|
||||
func trunc30(s string) string {
|
||||
if len(s) > 30 {
|
||||
return s[:30]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// TestSoak hammers the server for a while and watches its memory. The claim
|
||||
// it checks: nothing accumulates between requests - no cache, no growing
|
||||
// map, no goroutine left behind. Run it against a real corpus:
|
||||
//
|
||||
// SNOWCHIVE_BENCH_ROOT=/path/to/archive tests/run.sh -run TestSoak -timeout 30m
|
||||
//
|
||||
// SNOWCHIVE_SOAK_REQUESTS sets the count (default 20,000).
|
||||
func TestSoak(t *testing.T) {
|
||||
h := benchCorpus(t)
|
||||
n := 20000
|
||||
if v := os.Getenv("SNOWCHIVE_SOAK_REQUESTS"); v != "" {
|
||||
if parsed, err := strconv.Atoi(v); err == nil && parsed > 0 {
|
||||
n = parsed
|
||||
}
|
||||
}
|
||||
paths := []string{"/", "/browse", "/browse?collection=foia", "/browse/program",
|
||||
"/browse/tag?sort=count", "/browse/agency/NSA", "/search?q=nsa&do_search=1",
|
||||
"/search?q=%22upstream+collection%22&do_search=1", "/explore/ids", "/glossary",
|
||||
"/articles", "/api/facets", "/api/search?q=prism", "/sitemap.xml"}
|
||||
cookies := []*http.Cookie{
|
||||
{Name: "pref_theme", Value: "dark"},
|
||||
{Name: "pref_per_page", Value: "100"},
|
||||
{Name: "pref_text_only", Value: "1"},
|
||||
}
|
||||
|
||||
read := func() (heap, sys uint64, goroutines int) {
|
||||
runtime.GC()
|
||||
var m runtime.MemStats
|
||||
runtime.ReadMemStats(&m)
|
||||
return m.HeapAlloc, m.Sys, runtime.NumGoroutine()
|
||||
}
|
||||
// Warm up first: the pre-rendered bodies and the first template
|
||||
// executions allocate once and stay.
|
||||
for i := 0; i < 200; i++ {
|
||||
r := httptest.NewRequest("GET", paths[i%len(paths)], nil)
|
||||
h.ServeHTTP(httptest.NewRecorder(), r)
|
||||
}
|
||||
heap0, sys0, gor0 := read()
|
||||
|
||||
start := time.Now()
|
||||
var wg sync.WaitGroup
|
||||
const workers = 8
|
||||
for w := 0; w < workers; w++ {
|
||||
wg.Add(1)
|
||||
go func(id int) {
|
||||
defer wg.Done()
|
||||
for i := id; i < n; i += workers {
|
||||
r := httptest.NewRequest("GET", paths[i%len(paths)], nil)
|
||||
if i%4 == 0 {
|
||||
r.AddCookie(cookies[i%len(cookies)])
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
h.ServeHTTP(rec, r)
|
||||
if rec.Code != 200 {
|
||||
t.Errorf("%s: %d", paths[i%len(paths)], rec.Code)
|
||||
return
|
||||
}
|
||||
}
|
||||
}(w)
|
||||
}
|
||||
wg.Wait()
|
||||
elapsed := time.Since(start)
|
||||
heap1, sys1, gor1 := read()
|
||||
|
||||
mb := func(b uint64) float64 { return float64(b) / (1 << 20) }
|
||||
fmt.Printf("%d requests in %s (%.0f req/s, %d workers)\n",
|
||||
n, elapsed.Round(time.Millisecond), float64(n)/elapsed.Seconds(), workers)
|
||||
fmt.Printf("heap %.1f MB -> %.1f MB (%+.1f)\n", mb(heap0), mb(heap1), mb(heap1)-mb(heap0))
|
||||
fmt.Printf("sys %.1f MB -> %.1f MB (%+.1f)\n", mb(sys0), mb(sys1), mb(sys1)-mb(sys0))
|
||||
fmt.Printf("goroutines %d -> %d\n", gor0, gor1)
|
||||
|
||||
if gor1 > gor0+2 {
|
||||
t.Errorf("goroutines grew from %d to %d", gor0, gor1)
|
||||
}
|
||||
// The heap after a GC should be the corpus and the pre-rendered pages,
|
||||
// not a function of how many requests have been served.
|
||||
if grown := mb(heap1) - mb(heap0); grown > 32 {
|
||||
t.Errorf("heap grew by %.1f MB over %d requests - something is accumulating", grown, n)
|
||||
}
|
||||
}
|
||||
78
tests/bugs_test.go
Normal file
78
tests/bugs_test.go
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// A hyphenated term must match its words in sequence, not just the first one.
|
||||
func TestHyphenatedTermMatchesInSequence(t *testing.T) {
|
||||
hit := Document{ID: "hit", FullText: "A cyber attack on the power grid."}
|
||||
miss := Document{ID: "miss", FullText: "Cyber policy, and an attack helicopter."}
|
||||
for _, d := range []*Document{&hit, &miss} {
|
||||
setBodySearchText(d)
|
||||
d.StemmedText, d.SearchText = d.StemmedFull, d.FoldedFull
|
||||
}
|
||||
|
||||
for _, field := range []string{"FT", "TE"} {
|
||||
if !compileQuery([]string{"cyber-attack"}, []string{field}, []string{"and"}).matches(&hit) {
|
||||
t.Errorf("%s: cyber-attack should match text containing the words in sequence", field)
|
||||
}
|
||||
if compileQuery([]string{"cyber-attack"}, []string{field}, []string{"and"}).matches(&miss) {
|
||||
t.Errorf("%s: cyber-attack matched a document where the words are apart", field)
|
||||
}
|
||||
}
|
||||
// The words on their own still match both, as before.
|
||||
if !compileQuery([]string{"cyber"}, []string{"FT"}, []string{"and"}).matches(&miss) {
|
||||
t.Error("a plain word should still match")
|
||||
}
|
||||
}
|
||||
|
||||
// A value page's publication-year bars count every year a document was
|
||||
// published in, because each bar links to a search that matches any of them.
|
||||
func TestPubYearsOf(t *testing.T) {
|
||||
var d Document
|
||||
if err := json.Unmarshal([]byte(`{"publication": [
|
||||
{"publisher": "The Guardian", "date_published": "2013-06-06"},
|
||||
{"publisher": "Le Monde", "date_published": "2014-10-21"},
|
||||
{"publisher": "Der Spiegel", "date_published": "2013-12-30"},
|
||||
{"publisher": "No date"},
|
||||
{"publisher": "Bad date", "date_published": "201"}
|
||||
]}`), &d); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := fmt.Sprint(pubYearsOf(&d)); got != "[2013 2014]" {
|
||||
t.Errorf("pubYearsOf = %s, want [2013 2014]", got)
|
||||
}
|
||||
if got := fmt.Sprint(pubYearsOf(&Document{})); got != "[]" {
|
||||
t.Errorf("pubYearsOf(no publications) = %s, want []", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A saved search's label must name the type filter, or two saved searches
|
||||
// differing only by Reported/SIDtoday are indistinguishable.
|
||||
func TestSearchLabelIncludesTypeFilter(t *testing.T) {
|
||||
entry := func(v url.Values) string { return url.QueryEscape(v.Encode()) }
|
||||
base := url.Values{"fqv": {"xkeyscore"}, "fqf": {"TE"}, "fqc": {"and"}}
|
||||
typed := url.Values{"fqv": {"xkeyscore"}, "fqf": {"TE"}, "fqc": {"and"}, "type": {"REPORTED"}}
|
||||
both := url.Values{"fqv": {"xkeyscore"}, "fqf": {"TE"}, "fqc": {"and"},
|
||||
"type": {"SIDTODAY"}, "coll": {"news"}}
|
||||
|
||||
got := corpus().parseSearchEntries([]string{entry(typed), entry(base), entry(both)})
|
||||
if len(got) != 3 {
|
||||
t.Fatalf("parsed %d entries, want 3", len(got))
|
||||
}
|
||||
if !strings.Contains(got[0].Label, "REPORTED") {
|
||||
t.Errorf("label %q should name the type filter", got[0].Label)
|
||||
}
|
||||
if got[0].Label == got[1].Label {
|
||||
t.Errorf("searches differing only by type share the label %q", got[0].Label)
|
||||
}
|
||||
// Collection and type share one bracket rather than two.
|
||||
if !strings.Contains(got[2].Label, "(news, SIDTODAY)") {
|
||||
t.Errorf("label %q should list collection and type together", got[2].Label)
|
||||
}
|
||||
}
|
||||
538
tests/consistency_test.go
Normal file
538
tests/consistency_test.go
Normal file
|
|
@ -0,0 +1,538 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"html"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"regexp"
|
||||
"slices"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// A corpus built to exercise the places where one fact used to be computed
|
||||
// twice: search rows and their negations, the page and the API, counts and
|
||||
// the pages they link to. Each document's words are chosen so the expected
|
||||
// result sets can be written down by hand.
|
||||
//
|
||||
// n1 news alpha gamma ZEBRA 2010 The Intercept 2014 + the intercept 2016
|
||||
// n2 news alpha ZEBRA 2011 The Intercept 2014 + Der Spiegel 2014
|
||||
// n3 news beta gamma 2012 Der Spiegel 2015
|
||||
// n4 news beta 2012 Der Spiegel 2015
|
||||
// f1 foia alpha ZEBRA 2010 (court filing: "(c)" list labels)
|
||||
// s1 sidtoday ZEBRA 2011
|
||||
// a1 news "aka_id": null
|
||||
// d1 news date_created "ODNI", date_published "15 May 2015"
|
||||
func writeConsistencyCorpus(t *testing.T) string {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
doc := func(dir, id string, meta map[string]interface{}, text string) {
|
||||
b, err := json.Marshal(meta)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
d := filepath.Join(root, dir, id)
|
||||
writeFixtureFile(t, filepath.Join(d, "metadata.json"), string(b))
|
||||
writeFixtureFile(t, filepath.Join(d, id+".pdf"), "%PDF-1.4\n")
|
||||
if text != "" {
|
||||
writeFixtureFile(t, filepath.Join(d, id+".md"), text)
|
||||
}
|
||||
}
|
||||
pub := func(publisher, date, link string, reporters ...string) map[string]interface{} {
|
||||
return map[string]interface{}{"publisher": publisher, "date_published": date,
|
||||
"link": link, "reporters": reporters}
|
||||
}
|
||||
meta := func(title, typ, created string, programs []string, pubs ...map[string]interface{}) map[string]interface{} {
|
||||
return map[string]interface{}{"title": title, "type": typ,
|
||||
"document": map[string]interface{}{"creator": "NSA", "date_created": created, "programs": programs},
|
||||
"publication": pubs}
|
||||
}
|
||||
zebra := []string{"ZEBRA"}
|
||||
doc("", "n1", meta("Report one", "REPORTED", "2010-03-01", zebra,
|
||||
pub("The Intercept", "2014-01-01", "https://example.org/a", "Glenn Greenwald"),
|
||||
pub("the intercept", "2016-01-01", "https://example.org/b", "glenn greenwald")),
|
||||
"alpha gamma\n\n(S//SI) a secret line\n(S/SI) the same marking, spelled by OCR\n(U) plain\n")
|
||||
doc("", "n2", meta("Report two", "REPORTED", "2011-03-01", zebra,
|
||||
pub("The Intercept", "2014-02-01", "https://example.org/c"),
|
||||
pub("Der Spiegel", "2014-02-02", "https://example.org/f")),
|
||||
"alpha\n\n(TS//SI//NF) top line\n")
|
||||
doc("", "n3", meta("Report three", "REPORTED", "2012-03-01", nil,
|
||||
pub("Der Spiegel", "2015-01-01", "https://example.org/d")), "beta gamma\n")
|
||||
doc("", "n4", meta("Report four", "REPORTED", "2012-04-01", nil,
|
||||
pub("Der Spiegel", "2015-01-01", "https://example.org/d")), "beta\n")
|
||||
doc("FOIA", "f1", meta("Court filing", "FOIA", "2010-05-01", zebra),
|
||||
"alpha\n\n(c) The Court shall issue an order.\n(s) another subsection\n")
|
||||
doc("SIDtoday-archive", "s1", meta("Issue", "SIDTODAY", "2011-05-01", zebra), "(S) issue line\n")
|
||||
a1 := meta("Null aka", "REPORTED", "", nil)
|
||||
a1["aka_id"] = nil
|
||||
doc("", "a1", a1, "")
|
||||
// A date of no shape the archive reads: served, undated.
|
||||
doc("", "d1", meta("Odd date", "REPORTED", "ODNI", nil, pub("Der Spiegel", "15 May 2015", "https://example.org/e")), "")
|
||||
writeFixtureFile(t, filepath.Join(root, "glossary.json"), `[]`)
|
||||
return root
|
||||
}
|
||||
|
||||
// matchingIDs runs the search engine and returns the matching ids, sorted.
|
||||
func matchingIDs(queries, fields, conjs []string) []string {
|
||||
q, f, c := normalizeClauses(queries, fields, conjs)
|
||||
var ids []string
|
||||
scored, _ := corpus().executeSearch(searchParams{Queries: q, Fields: f, Conjs: c})
|
||||
for _, s := range scored {
|
||||
ids = append(ids, s.Doc.ID)
|
||||
}
|
||||
sort.Strings(ids)
|
||||
return ids
|
||||
}
|
||||
|
||||
// A row is one unit: its -terms are exclusions within it, whatever joins
|
||||
// it to the rows before. And an empty row starts nothing - a blank first
|
||||
// row with "OR x" under it is a search for x.
|
||||
func TestSearchRowsAndNegations(t *testing.T) {
|
||||
withCorpus(t, writeConsistencyCorpus(t))
|
||||
te2 := []string{"TE", "TE"}
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
queries, conjs []string
|
||||
want []string
|
||||
}{
|
||||
{"inline negation", []string{"alpha -gamma"}, []string{"and"}, []string{"f1", "n2"}},
|
||||
// alpha AND NOT (beta AND NOT gamma): no alpha document is lost
|
||||
// for mentioning gamma.
|
||||
{"negation inside NOT", []string{"alpha", "beta -gamma"}, []string{"and", "not"}, []string{"f1", "n1", "n2"}},
|
||||
// alpha OR (beta AND NOT gamma): the exclusion stays inside its row.
|
||||
{"negation inside OR", []string{"alpha", "beta -gamma"}, []string{"and", "or"}, []string{"f1", "n1", "n2", "n4"}},
|
||||
{"blank first row, OR", []string{"", "alpha"}, []string{"and", "or"}, []string{"f1", "n1", "n2"}},
|
||||
{"spaces first row, OR", []string{" ", "alpha"}, []string{"and", "or"}, []string{"f1", "n1", "n2"}},
|
||||
{"blank first row, OR nothing", []string{"", "zzzzunmatched"}, []string{"and", "or"}, nil},
|
||||
{"blank first row, NOT", []string{"", "gamma"}, []string{"and", "not"}, []string{"a1", "d1", "f1", "n2", "n4", "s1"}},
|
||||
{"only exclusions", []string{"-gamma -alpha"}, []string{"and"}, []string{"a1", "d1", "n4", "s1"}},
|
||||
} {
|
||||
if got := matchingIDs(c.queries, te2[:len(c.queries)], c.conjs); !reflect.DeepEqual(got, c.want) {
|
||||
t.Errorf("%s: %q %v matched %v, want %v", c.name, c.queries, c.conjs, got, c.want)
|
||||
}
|
||||
}
|
||||
// The scorer and the highlighter read the same rows: an excluded term
|
||||
// is neither rewarded nor marked.
|
||||
if got := positiveTerms([]string{"alpha -gamma", "", "beta"}, []string{"and", "and", "not"}); !reflect.DeepEqual(got, []string{"alpha"}) {
|
||||
t.Errorf("positiveTerms = %q, want [alpha]", got)
|
||||
}
|
||||
if term, _ := snippetAnchor([]string{"-gamma", "alpha -beta"}, te2, []string{"and", "and"}); term != "alpha" {
|
||||
t.Errorf("snippetAnchor = %q, want alpha", term)
|
||||
}
|
||||
}
|
||||
|
||||
// "+ Add row" and "remove" re-run the search: on the same rows, meaning the
|
||||
// same thing, as the Search button would.
|
||||
func TestRowEditsSearchTheSameRows(t *testing.T) {
|
||||
withCorpus(t, writeConsistencyCorpus(t))
|
||||
rows := "do_search=1&fqv=alpha+-gamma&fqf=TE&fqc=and"
|
||||
want := resultTotal(t, "/search?action=search&"+rows, foiaOn)
|
||||
if want != 2 {
|
||||
t.Fatalf("search for alpha -gamma found %d, want 2", want)
|
||||
}
|
||||
for _, action := range []string{"add", "remove-1"} {
|
||||
path := "/search?action=" + action + "&" + rows + "&fqv=beta&fqf=TE&fqc=or"
|
||||
if action == "add" {
|
||||
path = "/search?action=add&" + rows
|
||||
}
|
||||
if got := resultTotal(t, path, foiaOn); got != want {
|
||||
t.Errorf("action=%s: %d results, the same rows searched find %d", action, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The type filter resubmits the search the page shows, not the request: a
|
||||
// removed row stays removed, and the chosen sort is kept.
|
||||
func TestTypeFilterFormCarriesTheEffectiveSearch(t *testing.T) {
|
||||
withCorpus(t, writeConsistencyCorpus(t))
|
||||
body := get(t, "/search?do_search=1&action=remove-1&sort=title"+
|
||||
"&fqv=alpha&fqf=TE&fqc=and&fqv=removedrow&fqf=TE&fqc=or", foiaOn).Body.String()
|
||||
if !strings.Contains(body, `class="type-filter"`) {
|
||||
t.Fatal("no type filter on a result set holding two types")
|
||||
}
|
||||
if strings.Contains(body, "removedrow") {
|
||||
t.Error("the removed row is still in the page - the type filter would bring it back")
|
||||
}
|
||||
if !strings.Contains(body, `name="sort" value="title"`) {
|
||||
t.Error("the type filter drops the chosen sort")
|
||||
}
|
||||
}
|
||||
|
||||
// /api/search runs the search page's pipeline, type filter included. The
|
||||
// API has no preference: it matches the page of a reader who includes FOIA
|
||||
// releases.
|
||||
func TestAPISearchMatchesThePage(t *testing.T) {
|
||||
withCorpus(t, writeConsistencyCorpus(t))
|
||||
for qs, want := range map[string]int{
|
||||
"q=alpha": 3,
|
||||
"q=alpha&type=REPORTED": 2,
|
||||
"q=alpha&type=FOIA&type=REPORTED": 3,
|
||||
"q=alpha&type=reported": 2,
|
||||
"q=zebra&coll=news": 2,
|
||||
"q=zebra&coll=foia&type=FOIA": 1,
|
||||
"do_search=1&fqv=alpha&fqf=TE&fqc=and&fqv=beta+-gamma&fqf=TE&fqc=or": 4,
|
||||
} {
|
||||
page := resultTotal(t, "/search?"+qs, foiaOn)
|
||||
var api struct{ Total int }
|
||||
if err := json.Unmarshal(get(t, "/api/search?"+qs).Body.Bytes(), &api); err != nil {
|
||||
t.Fatalf("%s: %v", qs, err)
|
||||
}
|
||||
if page != want || api.Total != want {
|
||||
t.Errorf("%s: the page finds %d, the API %d, want %d", qs, page, api.Total, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Every count in this corpus is the number its page or search returns.
|
||||
func TestConsistencyCorpusCounts(t *testing.T) {
|
||||
withCorpus(t, writeConsistencyCorpus(t))
|
||||
|
||||
// Facets, with journalist and publisher spellings that differ by case
|
||||
// across one document's publications.
|
||||
for ci := range categories {
|
||||
c := &categories[ci]
|
||||
for val, want := range c.Facet(corpus()) {
|
||||
if got := len(corpus().categoryMatches(c.Slug, val)); got != want {
|
||||
t.Errorf("%s=%q: facet says %d, the page lists %d", c.Slug, val, want, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
if js := corpus().archive.Facets["journalist"]; len(js) != 1 || js["Glenn Greenwald"] != 1 {
|
||||
t.Errorf("journalists = %v, want Glenn Greenwald once", js)
|
||||
}
|
||||
if n := corpus().archive.Facets["publisher"]["The Intercept"]; n != 2 {
|
||||
t.Errorf("The Intercept: %d documents, want 2 (%v)", n, corpus().archive.Facets["publisher"])
|
||||
}
|
||||
|
||||
// The value-page timelines and refine links, scoped to a collection:
|
||||
// f1 (FOIA) shares 2010 with n1 and s1 (SIDtoday) shares 2011 with n2,
|
||||
// so an unscoped bar link finds more than its bar counts.
|
||||
for _, p := range []string{"/browse/program/ZEBRA?coll=news", "/browse/program/ZEBRA"} {
|
||||
if n := followLinks(t, p, reBarLink, "timeline bar"); n < 2 {
|
||||
t.Errorf("%s: %d timeline bars checked, want the doc and pub timelines", p, n)
|
||||
}
|
||||
}
|
||||
|
||||
// /explore/outlets: rows, cells and the per-year footer.
|
||||
for coll, table := range corpus().outletTables {
|
||||
for _, row := range table.Rows {
|
||||
if got := resultTotal(t, row.URL); got != row.Total {
|
||||
t.Errorf("outlets[%q] %s: total %d, its page lists %d", coll, row.Name, row.Total, got)
|
||||
}
|
||||
for _, cell := range row.Cells {
|
||||
if cell.URL != "" && resultTotal(t, cell.URL) != cell.Docs {
|
||||
t.Errorf("outlets[%q] %s: a cell says %d, its search finds %d", coll, row.Name, cell.Docs, resultTotal(t, cell.URL))
|
||||
}
|
||||
}
|
||||
}
|
||||
// The footer adds up its column: a document several outlets
|
||||
// published counts once for each of them.
|
||||
for i, y := range table.Years {
|
||||
sum := 0
|
||||
for _, row := range table.Rows {
|
||||
sum += row.Cells[i].Docs
|
||||
}
|
||||
if table.Totals[i] != sum {
|
||||
t.Errorf("outlets[%q] %s: footer %d, its column adds up to %d", coll, y, table.Totals[i], sum)
|
||||
}
|
||||
}
|
||||
}
|
||||
cell := func(outlet, year string) (docs, footer int) {
|
||||
table := corpus().outletTables[""]
|
||||
for i, y := range table.Years {
|
||||
if y != year {
|
||||
continue
|
||||
}
|
||||
footer = table.Totals[i]
|
||||
for _, row := range table.Rows {
|
||||
if row.Name == outlet {
|
||||
docs = row.Cells[i].Docs
|
||||
}
|
||||
}
|
||||
}
|
||||
return docs, footer
|
||||
}
|
||||
// n2: The Intercept and Der Spiegel both published it in 2014, so it
|
||||
// counts for both outlets and twice in the year's footer (n1 is the
|
||||
// third: The Intercept, 2014).
|
||||
if spiegel, footer := cell("Der Spiegel", "2014"); spiegel != 1 || footer != 3 {
|
||||
t.Errorf("2014: Der Spiegel cell %d, footer %d - want 1 and 3 (n2 once per outlet)", spiegel, footer)
|
||||
}
|
||||
if intercept, _ := cell("The Intercept", "2014"); intercept != 2 {
|
||||
t.Errorf("2014: The Intercept cell %d, want 2", intercept)
|
||||
}
|
||||
intercept := 0
|
||||
for _, row := range corpus().outletTables[""].Rows {
|
||||
if row.Name == "The Intercept" {
|
||||
intercept = row.Total
|
||||
}
|
||||
}
|
||||
if intercept != 2 {
|
||||
t.Errorf("The Intercept's outlet total = %d, want 2 documents (n1 counts once for its two years)", intercept)
|
||||
}
|
||||
|
||||
// /explore/portions: lower-case list labels are not markings, and every
|
||||
// row's document count is what its search finds.
|
||||
for _, l := range corpus().portionStats.Levels {
|
||||
if l.Name == "C" {
|
||||
t.Errorf("a CONFIDENTIAL row from the court filing's (c) labels: %+v", l)
|
||||
}
|
||||
}
|
||||
rows := append(append(append([]portionRow(nil), corpus().portionStats.Levels...), corpus().portionStats.Markings...), corpus().portionStats.Controls...)
|
||||
if len(rows) < 6 {
|
||||
t.Fatalf("only %d portion rows", len(rows))
|
||||
}
|
||||
for _, r := range rows {
|
||||
if got := resultTotal(t, string(r.SearchURL)); got != r.Docs {
|
||||
t.Errorf("portion row %s: %d documents, its search finds %d (%s)", r.Name, r.Docs, got, r.SearchURL)
|
||||
}
|
||||
}
|
||||
if got := corpus().docIndex["n1"].PortionMarkings; !reflect.DeepEqual(got, []string{"S//SI", "U"}) {
|
||||
t.Errorf("n1 markings = %v, want [S//SI U]", got)
|
||||
}
|
||||
|
||||
// A malformed date is no date: no "ODNI" year that no year page opens.
|
||||
if d := corpus().docIndex["d1"]; d == nil || d.DocMeta.DateCreated != "" || d.Publication[0].DatePublished != "" {
|
||||
t.Errorf("d1's malformed dates were kept: %+v", d)
|
||||
}
|
||||
if _, ok := corpus().archive.Facets["docdate"]["ODNI"]; ok {
|
||||
t.Error(`"ODNI" counted as a document year`)
|
||||
}
|
||||
|
||||
// aka_id: null is no akas, not a reason to drop the document.
|
||||
if corpus().docIndex["a1"] == nil {
|
||||
t.Error(`a document with "aka_id": null was not loaded`)
|
||||
}
|
||||
}
|
||||
|
||||
// A corpus for the counts that turn on WHICH publication entry carries a
|
||||
// date or a byline, and on the reader's collection scope. Every document
|
||||
// says "alpha".
|
||||
//
|
||||
// p1 news 2010-03 Outlet X 2013 (Able), Outlet Y 2014 (Baker)
|
||||
// p2 news 2010-06 Outlet X 2014 (Able) people: Clerk
|
||||
// p3 news 2011-01 Outlet X 2014 (Baker)
|
||||
// p4 news 2012-03 Outlet Y 2014
|
||||
// p5 news 2012-09 Outlet Y 2014
|
||||
// p6 news 2013-02 Outlet Y 2014, Partner Z 2014 (its own story; reporting only)
|
||||
// f1 foia 2009-07 Outlet Y 2014 (a story of its own) people: Docket
|
||||
func writeEntryCorpus(t *testing.T) string {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
pub := func(publisher, date string, reporters ...string) map[string]interface{} {
|
||||
return map[string]interface{}{"publisher": publisher, "date_published": date, "reporters": reporters}
|
||||
}
|
||||
doc := func(dir, id, typ, created string, people []string, pubs ...map[string]interface{}) {
|
||||
b, err := json.Marshal(map[string]interface{}{"title": "Document " + id, "type": typ,
|
||||
"document": map[string]interface{}{"creator": "NSA", "date_created": created, "people": people},
|
||||
"publication": pubs})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
d := filepath.Join(root, dir, id)
|
||||
writeFixtureFile(t, filepath.Join(d, "metadata.json"), string(b))
|
||||
writeFixtureFile(t, filepath.Join(d, id+".pdf"), "%PDF-1.4\n")
|
||||
writeFixtureFile(t, filepath.Join(d, id+".md"), "alpha\n")
|
||||
}
|
||||
doc("", "p1", "REPORTED", "2010-03-01", nil,
|
||||
pub("Outlet X", "2013-05-01", "Able, Ann"), pub("Outlet Y", "2014-02-01", "Baker, Bob"))
|
||||
doc("", "p2", "REPORTED", "2010-06-01", []string{"Clerk, Carol"}, pub("Outlet X", "2014-03-01", "Able, Ann"))
|
||||
doc("", "p3", "REPORTED", "2011-01-01", nil, pub("Outlet X", "2014-04-01", "Baker, Bob"))
|
||||
doc("", "p4", "REPORTED", "2012-03-01", nil, pub("Outlet Y", "2014-05-01"))
|
||||
doc("", "p5", "REPORTED", "2012-09-01", nil, pub("Outlet Y", "2014-06-01"))
|
||||
story := func(p map[string]interface{}, title, link string) map[string]interface{} {
|
||||
p["title"], p["link"] = title, link
|
||||
return p
|
||||
}
|
||||
partner := story(pub("Partner Z", "2014-07-02"), "Partner story", "https://example.org/partner")
|
||||
partner["reporting_only"] = true
|
||||
doc("", "p6", "REPORTED", "2013-02-01", nil, pub("Outlet Y", "2014-07-01"), partner)
|
||||
doc("FOIA", "f1", "FOIA", "2009-07-01", []string{"Docket, Dan"},
|
||||
story(pub("Outlet Y", "2014-08-01"), "FOIA story", "https://example.org/foia-story"))
|
||||
writeFixtureFile(t, filepath.Join(root, "glossary.json"), `[]`)
|
||||
return root
|
||||
}
|
||||
|
||||
// Counts that turn on a publication entry's date or byline, or on the
|
||||
// reader's scope, are still the numbers their links return.
|
||||
func TestCountsTiedToEntriesAndScope(t *testing.T) {
|
||||
withCorpus(t, writeEntryCorpus(t))
|
||||
|
||||
// A publication-year page files a document under its first publication,
|
||||
// which no search asks for: p1, first published in 2013, would join the
|
||||
// 2010 bar's search for 2014. So those bars carry no link. The
|
||||
// document-year page's publication bars do, and they are exact.
|
||||
body := get(t, "/browse/published/2014").Body.String()
|
||||
if !strings.Contains(body, "By document date") || strings.Contains(body, `<a class="hist-col"`) {
|
||||
t.Error("/browse/published/2014: want the document-date bars, and no link on them")
|
||||
}
|
||||
if n := followLinks(t, "/browse/docdate/2010", reBarLink, "publication bar"); n != 2 {
|
||||
t.Errorf("/browse/docdate/2010: %d publication bars checked, want 2013 and 2014", n)
|
||||
}
|
||||
|
||||
// Under a publication year a journalist counts for the documents that
|
||||
// byline them on an entry of that year, because that is what the refine
|
||||
// link's search matches: Able's byline on p1 is Outlet X's of 2013, not
|
||||
// Outlet Y's of 2014.
|
||||
for _, p := range []string{
|
||||
"/browse/publisher/Outlet%20Y?year=2014",
|
||||
"/search?q=alpha+published%3A2014",
|
||||
// An OR row unties the dates, and the bylines with them.
|
||||
"/search?do_search=1&fqv=alpha&fqf=TE&fqc=and&fqv=beta&fqf=TE&fqc=or&date_pub_start=2014&date_pub_end=2014",
|
||||
} {
|
||||
if n := followLinks(t, p, reFacetLink, "refine link"); n == 0 {
|
||||
t.Errorf("%s: no refine links to check", p)
|
||||
}
|
||||
}
|
||||
|
||||
// A year bar narrows the search it stands under: in a range that starts
|
||||
// in June 2012, the 2012 bar counts p5 and not p4 of March.
|
||||
if n := followLinks(t, "/search?q=alpha+date%3A2012-06..2013-06", reBarLink, "year bar"); n != 2 {
|
||||
t.Errorf("%d year bars checked, want 2012 and 2013", n)
|
||||
}
|
||||
|
||||
// Docket is named only in the FOIA release and 2009 is only its year:
|
||||
// neither is on the default reader's lists, nor in the boxes' counts.
|
||||
for _, reader := range [][]*http.Cookie{nil, {foiaOn}} {
|
||||
if n := checkCategoryBoxes(t, reader...); n < 10 {
|
||||
t.Errorf("%d category boxes checked", n)
|
||||
}
|
||||
}
|
||||
|
||||
// The home page's publication chart counts first publications, as the
|
||||
// year pages its bars link file them: p1 counts for 2013 and not 2014.
|
||||
body = get(t, "/").Body.String()
|
||||
for year, want := range map[string]string{"2013": "1", "2014": "5"} {
|
||||
if !strings.Contains(body, `href="/browse/published/`+year+`?coll=news" title="`+want+` documents first published in `+year+`"`) {
|
||||
t.Errorf("home: no bar of %s documents first published in %s linking to that year's page", want, year)
|
||||
}
|
||||
}
|
||||
|
||||
// In a search result a publication year links to its year page only
|
||||
// when that page lists the document: p1 is filed under 2013, its first.
|
||||
body = get(t, "/search?q=title%3Ap1").Body.String()
|
||||
if !strings.Contains(body, `href="/browse/published/2013"`) || strings.Contains(body, `href="/browse/published/2014"`) {
|
||||
t.Error("p1's result should link 2013, its first publication year, and not 2014")
|
||||
}
|
||||
|
||||
// A partner that only reported has no publisher page for its story's
|
||||
// byline to link.
|
||||
body = get(t, "/article/"+articleKeyFor("https://example.org/partner")).Body.String()
|
||||
if !strings.Contains(body, "Partner Z") || strings.Contains(body, "/browse/publisher/Partner") {
|
||||
t.Error("the partner's article should name Partner Z without linking a publisher page")
|
||||
}
|
||||
|
||||
// The outlets table of one collection counts the stories that have a
|
||||
// document in it: Outlet Y's 2014 story published only the FOIA release.
|
||||
stories := func(coll string) int {
|
||||
table := corpus().outletTables[coll]
|
||||
for _, row := range table.Rows {
|
||||
if row.Name == "Outlet Y" {
|
||||
return row.Cells[slices.Index(table.Years, "2014")].Articles
|
||||
}
|
||||
}
|
||||
return -1
|
||||
}
|
||||
if all, news := stories(""), stories(collectionNews); all != 1 || news != 0 {
|
||||
t.Errorf("Outlet Y's 2014 stories: %d over every collection and %d reported, want 1 and 0", all, news)
|
||||
}
|
||||
}
|
||||
|
||||
var reCategoryBox = regexp.MustCompile(`<a href="/browse/([a-z]+)" class="cat-box[^"]*"><strong>[^<]*</strong><span class="count">(\d+)</span></a>`)
|
||||
|
||||
// checkCategoryBoxes holds the number on each category box in the browse
|
||||
// nav to the rows of the list its click opens, for the reader with these
|
||||
// cookies.
|
||||
func checkCategoryBoxes(t *testing.T, cookies ...*http.Cookie) int {
|
||||
t.Helper()
|
||||
boxes := reCategoryBox.FindAllStringSubmatch(get(t, "/browse", cookies...).Body.String(), -1)
|
||||
for _, m := range boxes {
|
||||
slug, shown := m[1], mustAtoi(t, m[2])
|
||||
list := get(t, "/browse/"+slug, cookies...).Body.String()
|
||||
if rows := strings.Count(list, `href="/browse/`+slug+`/`); rows != shown {
|
||||
t.Errorf("the %s box says %d, its list has %d rows (%d cookies)", slug, shown, rows, len(cookies))
|
||||
}
|
||||
}
|
||||
return len(boxes)
|
||||
}
|
||||
|
||||
// -check reads the load counters, so the glossary's problems must still be
|
||||
// in them when it does.
|
||||
func TestGlossaryWarningsReachCheck(t *testing.T) {
|
||||
root := writeConsistencyCorpus(t)
|
||||
withCorpus(t, root)
|
||||
clean := loadWarnings
|
||||
writeFixtureFile(t, filepath.Join(root, "glossary.json"), `{not json`)
|
||||
loadData()
|
||||
if loadWarnings != clean+1 {
|
||||
t.Errorf("malformed glossary.json: %d warnings, want %d", loadWarnings, clean+1)
|
||||
}
|
||||
}
|
||||
|
||||
// A 404 under /files must not carry the files' public cache policy.
|
||||
func TestMissingFileIsNotCached(t *testing.T) {
|
||||
w := get(t, "/files/prism-slides/no-such-file.pdf")
|
||||
if w.Code != http.StatusNotFound {
|
||||
t.Fatalf("status %d, want 404", w.Code)
|
||||
}
|
||||
if cc := w.Header().Get("Cache-Control"); strings.Contains(cc, "public") {
|
||||
t.Errorf("404 carries Cache-Control %q", cc)
|
||||
}
|
||||
}
|
||||
|
||||
// -dev serves the on-disk assets so edits show without a restart: no
|
||||
// startup ETag may pin the old content.
|
||||
func TestDevModeStaticRevalidatesEdits(t *testing.T) {
|
||||
buildFixture(t)
|
||||
devMode = true
|
||||
initAssets()
|
||||
t.Cleanup(func() {
|
||||
devMode = false
|
||||
initAssets()
|
||||
})
|
||||
if staticHashes != nil {
|
||||
t.Errorf("dev mode built %d static hashes", len(staticHashes))
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
handleStatic(w, httptest.NewRequest("GET", "/static/style.css", nil))
|
||||
if w.Code != http.StatusOK || w.Header().Get("ETag") != "" || w.Header().Get("Last-Modified") == "" {
|
||||
t.Errorf("dev static: status %d, ETag %q, Last-Modified %q - want 200, none, the file's",
|
||||
w.Code, w.Header().Get("ETag"), w.Header().Get("Last-Modified"))
|
||||
}
|
||||
}
|
||||
|
||||
// What the search page offers to pin is what the pinned page can list.
|
||||
func TestPinnableSearchesAreListed(t *testing.T) {
|
||||
h, _ := buildFixture(t)
|
||||
if label, ok := corpus().searchLabel(mustQuery(t, "/search?do_search=1&fqv=&fqf=TE&fqc=and&date_doc_start=2013&date_doc_end=2013")); !ok || label != "all documents · (doc 2013)" {
|
||||
t.Errorf("date-only search label = %q, %v", label, ok)
|
||||
}
|
||||
if strings.Contains(get(t, "/search?do_search=1&fqv=&fqf=TE&fqc=and").Body.String(), "pin-search-btn") {
|
||||
t.Error("a search with no terms and no filters offers a pin")
|
||||
}
|
||||
if !strings.Contains(get(t, "/search?q=date:2013").Body.String(), "pin-search-btn") {
|
||||
t.Error("a date-only search offers no pin")
|
||||
}
|
||||
|
||||
pin := func(qs string) *httptest.ResponseRecorder {
|
||||
r := httptest.NewRequest("POST", "/pinsearch", strings.NewReader(url.Values{"qs": {qs}, "back": {"/pinned"}}.Encode()))
|
||||
r.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
return record(h, r)
|
||||
}
|
||||
if w := pin("do_search=1&fqc=and&fqf=TE&fqv="); w.Code != http.StatusBadRequest {
|
||||
t.Errorf("pinning an empty search: status %d, want 400", w.Code)
|
||||
}
|
||||
w := pin("date_doc_end=2013&date_doc_start=2013&do_search=1&fqc=and&fqf=TE&fqv=")
|
||||
if w.Code != http.StatusSeeOther {
|
||||
t.Fatalf("pinning a date-only search: status %d", w.Code)
|
||||
}
|
||||
body := get(t, "/pinned", w.Result().Cookies()...).Body.String()
|
||||
if !strings.Contains(html.UnescapeString(body), "all documents · (doc 2013)") {
|
||||
t.Error("the pinned date-only search is not listed on /pinned")
|
||||
}
|
||||
}
|
||||
199
tests/corpus_test.go
Normal file
199
tests/corpus_test.go
Normal file
|
|
@ -0,0 +1,199 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"html"
|
||||
"net/http"
|
||||
"os"
|
||||
"reflect"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Checks against a REAL corpus. Point $SNOWCHIVE_BENCH_DATA at one, as for
|
||||
// the page benchmarks, and run: go test -run TestCorpus -v
|
||||
//
|
||||
// The fixtures are a handful of documents, each built to show one thing. A
|
||||
// count and its link are computed by different code, and whether the two
|
||||
// agree turns on what the documents happen to hold: a second publication in
|
||||
// a later year, a byline on only one of them, a value found only in FOIA
|
||||
// releases. The disagreements these tests walk the archive for were all
|
||||
// found on the full corpus and on no fixture.
|
||||
func TestCorpus(t *testing.T) {
|
||||
root := os.Getenv("SNOWCHIVE_BENCH_DATA")
|
||||
if root == "" {
|
||||
t.Skip("set SNOWCHIVE_BENCH_DATA to a corpus directory to check a real corpus")
|
||||
}
|
||||
withCorpus(t, root)
|
||||
t.Run("counts", corpusCounts)
|
||||
t.Run("FOIA links", corpusFOIALinks)
|
||||
t.Run("inflections", corpusInflections)
|
||||
t.Run("snippets", corpusSnippets)
|
||||
t.Run("index", corpusIndex)
|
||||
}
|
||||
|
||||
// corpusCounts follows the counted links of the loaded corpus, as a reader
|
||||
// who leaves FOIA releases out and as one who includes them.
|
||||
func corpusCounts(t *testing.T) {
|
||||
// The pages whose bars and refine links are followed: the year pages of
|
||||
// both date categories, the three commonest values of every other
|
||||
// category, each cell of the outlets table (a publisher's page for one
|
||||
// publication year) and searches under each kind of date range.
|
||||
var pages []string
|
||||
for ci := range categories {
|
||||
c := &categories[ci]
|
||||
n := 3
|
||||
if c.isDate() {
|
||||
n = -1
|
||||
}
|
||||
for _, v := range rankCounts(c.Facet(corpus()), n) {
|
||||
pages = append(pages, categoryValueURL(c.Slug, v))
|
||||
}
|
||||
}
|
||||
for _, row := range corpus().outletTables[""].Rows {
|
||||
for _, cell := range row.Cells {
|
||||
if cell.URL != "" {
|
||||
pages = append(pages, cell.URL)
|
||||
}
|
||||
}
|
||||
}
|
||||
pages = append(pages,
|
||||
"/search?q=nsa",
|
||||
"/search?q=nsa+date%3A2012-06..2013-06",
|
||||
"/search?q=nsa+published%3A2016",
|
||||
"/search?q=published%3A2014",
|
||||
"/search?do_search=1&fqv=nsa&fqf=TE&fqc=and&date_pub_start=2015-01-01&date_pub_end=2016-06-30",
|
||||
"/search?do_search=1&fqv=nsa&fqf=TE&fqc=and&fqv=gchq&fqf=TE&fqc=or&date_pub_start=2016&date_pub_end=2016",
|
||||
)
|
||||
|
||||
for name, reader := range map[string][]*http.Cookie{"FOIA left out": nil, "FOIA included": {foiaOn}} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
bars, refines := 0, 0
|
||||
for _, p := range pages {
|
||||
bars += followLinks(t, p, reBarLink, "bar", reader...)
|
||||
refines += followLinks(t, p, reFacetLink, "refine link", reader...)
|
||||
}
|
||||
if bars == 0 || refines == 0 {
|
||||
t.Errorf("nothing exercised: %d bars, %d refine links", bars, refines)
|
||||
}
|
||||
boxes := checkCategoryBoxes(t, reader...)
|
||||
t.Logf("%d pages: %d bars, %d refine links and %d category boxes checked", len(pages), bars, refines, boxes)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// corpusFOIALinks opens the page of every tenth FOIA release as a reader
|
||||
// who leaves FOIA out, and follows each category value it links: the page
|
||||
// behind the link has to list the release.
|
||||
func corpusFOIALinks(t *testing.T) {
|
||||
reValueLink := regexp.MustCompile(`href="(/browse/[a-z]+/[^"]+)"`)
|
||||
links := 0
|
||||
for i, d := range corpus().collectionDocs[collectionFOIA] {
|
||||
if i%10 != 0 {
|
||||
continue
|
||||
}
|
||||
// The release's own metadata: the cards under it are other documents.
|
||||
page, _, _ := strings.Cut(get(t, docPath(d.ID)).Body.String(), `class="related-section"`)
|
||||
for _, m := range reValueLink.FindAllStringSubmatch(page, -1) {
|
||||
link := html.UnescapeString(m[1])
|
||||
links++
|
||||
if !strings.Contains(get(t, withQuery(link, "view=table")).Body.String(), `href="`+docPath(d.ID)+`"`) {
|
||||
t.Errorf("%s links %s, which does not list it", d.ID, link)
|
||||
}
|
||||
}
|
||||
}
|
||||
if links == 0 {
|
||||
t.Error("no FOIA release with a value link to follow")
|
||||
}
|
||||
t.Logf("%d value links followed", links)
|
||||
}
|
||||
|
||||
// A word in -ies or -ied stems as the word in -y it is a form of, whatever
|
||||
// rules that word goes through - checked over every such pair the corpus
|
||||
// holds, since a suffix rule that reads one of the two forms and not the
|
||||
// other splits them silently ("capability" from "capabilities").
|
||||
func corpusInflections(t *testing.T) {
|
||||
vocab := map[string]bool{}
|
||||
for i := range corpus().archive.Docs {
|
||||
for _, w := range splitWords(corpus().archive.Docs[i].SearchText) {
|
||||
vocab[w] = true
|
||||
}
|
||||
}
|
||||
pairs := 0
|
||||
for w := range vocab {
|
||||
if len(w) < 5 || !(strings.HasSuffix(w, "ies") || strings.HasSuffix(w, "ied")) {
|
||||
continue
|
||||
}
|
||||
base := w[:len(w)-3] + "y"
|
||||
if !vocab[base] {
|
||||
continue
|
||||
}
|
||||
pairs++
|
||||
if stem(w) != stem(base) {
|
||||
t.Errorf("stem(%q) = %q, stem(%q) = %q; want equal", w, stem(w), base, stem(base))
|
||||
}
|
||||
}
|
||||
if pairs == 0 {
|
||||
t.Error("the corpus holds no -ies/-ied word next to its -y form")
|
||||
}
|
||||
t.Logf("%d pairs checked in a vocabulary of %d words", pairs, len(vocab))
|
||||
}
|
||||
|
||||
// corpusSnippets holds every document's snippets off the fold kept at load
|
||||
// to the ones folding its transcript again gives - the versions' lines sit
|
||||
// behind that fold in the same string - and a page of "all" cards to its
|
||||
// bound.
|
||||
func corpusSnippets(t *testing.T) {
|
||||
for _, q := range []string{"nsa", "cable tapping", `"upstream collection"`, "tap*", "the"} {
|
||||
for i := range corpus().archive.Docs {
|
||||
d := &corpus().archive.Docs[i]
|
||||
if got, want := d.bodySnippets(q, 3), getSnippets(d.FullText, q, 3); !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("%s, %q: %q from the loaded fold, %q from the text", d.ID, q, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
all := &http.Cookie{Name: "pref_per_page", Value: "all"}
|
||||
for _, c := range categories {
|
||||
for _, v := range rankCounts(c.Facet(corpus()), 1) {
|
||||
if c.Facet(corpus())[v] <= maxCardsPerPage {
|
||||
continue
|
||||
}
|
||||
body := get(t, categoryValueURL(c.Slug, v), all, foiaOn).Body.String()
|
||||
if n := strings.Count(body, `class="doc-card"`); n != maxCardsPerPage {
|
||||
t.Errorf("%s=%s: %d cards on a page of all, want %d", c.Slug, v, n, maxCardsPerPage)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// corpusIndex holds the shortcut a one-word all-fields clause takes - it
|
||||
// asks the document's entry in the index instead of walking its text - to
|
||||
// the walk, for a spread of the vocabulary over every document.
|
||||
func corpusIndex(t *testing.T) {
|
||||
var vocab []string
|
||||
for w := range corpus().tfidf.DocFreq {
|
||||
vocab = append(vocab, w)
|
||||
}
|
||||
slices.Sort(vocab)
|
||||
checked := 0
|
||||
for i := 0; i < len(vocab); i += 131 {
|
||||
w := vocab[i]
|
||||
docs := 0
|
||||
for di := range corpus().archive.Docs {
|
||||
d := &corpus().archive.Docs[di]
|
||||
_, indexed := d.terms[w]
|
||||
if indexed != containsTokens(d.StemmedText, w) {
|
||||
t.Fatalf("%q in %s: the index says %v, the all-fields stream the opposite", w, d.ID, indexed)
|
||||
}
|
||||
if indexed {
|
||||
docs++
|
||||
}
|
||||
}
|
||||
if docs != corpus().tfidf.DocFreq[w] {
|
||||
t.Errorf("%q: in %d documents' entries, with a document frequency of %d", w, docs, corpus().tfidf.DocFreq[w])
|
||||
}
|
||||
checked++
|
||||
}
|
||||
t.Logf("%d terms checked over %d documents", checked, len(corpus().archive.Docs))
|
||||
}
|
||||
123
tests/data_test.go
Normal file
123
tests/data_test.go
Normal file
|
|
@ -0,0 +1,123 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestAkaIDShape(t *testing.T) {
|
||||
for _, s := range []string{
|
||||
`{"title": "no aka_id key"}`,
|
||||
`{"aka_id": []}`,
|
||||
// null is "no akas": not a reason to drop the document.
|
||||
`{"aka_id": null}`,
|
||||
`{"aka_id": [{"id": "a"}, {"id": "b", "doc_source": "https://e.org/d", "doc_url": "https://e.org/b.pdf"}]}`,
|
||||
} {
|
||||
var d Document
|
||||
if err := json.Unmarshal([]byte(s), &d); err != nil {
|
||||
t.Errorf("%s: unexpected error: %v", s, err)
|
||||
}
|
||||
}
|
||||
|
||||
for _, s := range []string{`{"aka_id": "a"}`, `{"aka_id": {"id": "a"}}`, `{"aka_id": ["a", "b"]}`, `{"aka_id": [{"id": "a"}, 3]}`} {
|
||||
var d Document
|
||||
if err := json.Unmarshal([]byte(s), &d); err == nil {
|
||||
t.Errorf("%s: decoded without an error", s)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeAka(t *testing.T) {
|
||||
d := Document{ID: "x", AkaID: []AkaEntry{
|
||||
{ID: " first ", DocSource: " https://e.org/p "},
|
||||
{ID: " "},
|
||||
{ID: "FIRST"},
|
||||
{ID: "second", DocURL: "https://e.org/s.pdf"},
|
||||
}}
|
||||
got := normalizeAka(&d)
|
||||
if len(got) != 2 || got[0].ID != "first" || got[0].DocSource != "https://e.org/p" || got[1].ID != "second" {
|
||||
t.Errorf("normalizeAka = %+v, want first (trimmed) and second, without the blank id and the repeat", got)
|
||||
}
|
||||
}
|
||||
|
||||
func writeTestFile(t *testing.T, path, content string) {
|
||||
t.Helper()
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadVersions(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
v := filepath.Join(dir, "versions")
|
||||
writeTestFile(t, filepath.Join(v, "lemonde-2013", "copy.pdf"), "%PDF-1.4")
|
||||
writeTestFile(t, filepath.Join(v, "lemonde-2013", "copy.md"), "# Slide 12\n\nOnly in Le Monde's copy.")
|
||||
writeTestFile(t, filepath.Join(v, "lemonde-2013", "images", "page-10.jpg"), "")
|
||||
writeTestFile(t, filepath.Join(v, "lemonde-2013", "images", "page-2.jpg"), "")
|
||||
writeTestFile(t, filepath.Join(v, "guardian", "copy.pdf"), "%PDF-1.4") // no transcript or images: still a version
|
||||
writeTestFile(t, filepath.Join(v, "no-pdf", "copy.md"), "text only") // skipped: no PDF
|
||||
writeTestFile(t, filepath.Join(v, "bad name", "copy.pdf"), "%PDF-1.4") // skipped: name
|
||||
writeTestFile(t, filepath.Join(v, "loose.pdf"), "%PDF-1.4") // skipped: not a folder
|
||||
|
||||
var d Document
|
||||
if err := json.Unmarshal([]byte(`{"publication": [
|
||||
{"publisher": "Le Monde", "version": "lemonde-2013"},
|
||||
{"publisher": "The Guardian"},
|
||||
{"publisher": "Dagbladet", "version": " lemonde-2013 "},
|
||||
{"publisher": "Nobody", "version": "missing"}
|
||||
]}`), &d); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
d.ID, d.Dir = "prism", dir
|
||||
corpus().loadVersions(&d)
|
||||
|
||||
var slugs []string
|
||||
for _, v := range d.Versions {
|
||||
slugs = append(slugs, v.Slug)
|
||||
}
|
||||
if got := strings.Join(slugs, ","); got != "guardian,lemonde-2013" {
|
||||
t.Fatalf("versions %q, want guardian,lemonde-2013", got)
|
||||
}
|
||||
|
||||
lm := d.VersionFor("lemonde-2013")
|
||||
if lm.PDFFile != "copy.pdf" || !strings.Contains(lm.FullText, "Only in Le Monde") || lm.HTMLContent == "" {
|
||||
t.Errorf("lemonde-2013 not fully loaded: pdf %q, text %q, html %q", lm.PDFFile, lm.FullText, lm.HTMLContent)
|
||||
}
|
||||
if got := strings.Join(lm.Images, ","); got != "page-2.jpg,page-10.jpg" {
|
||||
t.Errorf("images %q, want page-2.jpg,page-10.jpg (page order)", got)
|
||||
}
|
||||
if got := fmt.Sprint(lm.PubIndex); got != "[0 2]" {
|
||||
t.Errorf("PubIndex %s, want [0 2]", got)
|
||||
}
|
||||
if got := d.Publication[2].Version; got != "lemonde-2013" {
|
||||
t.Errorf("publication version not trimmed: %q", got)
|
||||
}
|
||||
if u := d.UnlinkedVersions(); len(u) != 1 || u[0].Slug != "guardian" {
|
||||
t.Errorf("unlinked versions %+v, want only guardian", u)
|
||||
}
|
||||
if d.VersionFor("missing") != nil || d.VersionFor("") != nil {
|
||||
t.Error("VersionFor returned a version that doesn't exist")
|
||||
}
|
||||
}
|
||||
|
||||
// TestBodyFormsAreTheCombinedTails: loadData slices FoldedFull and
|
||||
// StemmedFull out of the ends of SearchText and StemmedText to save memory;
|
||||
// they must still be exactly what setBodySearchText builds.
|
||||
func TestBodyFormsAreTheCombinedTails(t *testing.T) {
|
||||
buildFixture(t)
|
||||
for i := range corpus().archive.Docs {
|
||||
d := &corpus().archive.Docs[i]
|
||||
c := *d
|
||||
setBodySearchText(&c)
|
||||
if c.FoldedFull != d.FoldedFull || c.StemmedFull != d.StemmedFull {
|
||||
t.Errorf("%s: body forms differ from a fresh build", d.ID)
|
||||
}
|
||||
}
|
||||
}
|
||||
295
tests/fixture_test.go
Normal file
295
tests/fixture_test.go
Normal file
|
|
@ -0,0 +1,295 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"image"
|
||||
"image/jpeg"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// A small corpus on disk plus a running mux: the harness every handler test
|
||||
// uses. The fixture covers the shapes the handlers branch on - one document
|
||||
// per collection, a SIDtoday issue with a series, another published copy in
|
||||
// versions/, a joint investigation with a reporting-only partner, and a
|
||||
// directory with no PDF (which the loader must skip).
|
||||
//
|
||||
// Built once for the package: loadData writes package globals, so tests
|
||||
// share one corpus rather than racing to replace it.
|
||||
|
||||
var (
|
||||
fixtureOnce sync.Once
|
||||
fixtureMux http.Handler
|
||||
fixtureDir string
|
||||
)
|
||||
|
||||
type fixtureDoc struct {
|
||||
id, collection string
|
||||
meta map[string]interface{}
|
||||
text string
|
||||
pages []string
|
||||
versions map[string][]string // slug -> page images
|
||||
noPDF bool
|
||||
}
|
||||
|
||||
// fixtureJPEG is a real 4x3 JPEG, so the loader can read each page image's size.
|
||||
var fixtureJPEG = func() string {
|
||||
var b bytes.Buffer
|
||||
if err := jpeg.Encode(&b, image.NewGray(image.Rect(0, 0, 4, 3)), nil); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
return b.String()
|
||||
}()
|
||||
|
||||
func writeFixtureFile(t *testing.T, path, content string) {
|
||||
t.Helper()
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestMain removes the shared fixture corpus once every test has run.
|
||||
func TestMain(m *testing.M) {
|
||||
code := m.Run()
|
||||
if fixtureDir != "" {
|
||||
os.RemoveAll(fixtureDir)
|
||||
}
|
||||
os.Exit(code)
|
||||
}
|
||||
|
||||
func buildFixture(t *testing.T) (http.Handler, string) {
|
||||
t.Helper()
|
||||
fixtureOnce.Do(func() {
|
||||
root, err := os.MkdirTemp("", "snowchive-fixture")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, d := range fixtureDocs() {
|
||||
dir := root
|
||||
switch d.collection {
|
||||
case collectionFOIA:
|
||||
dir = filepath.Join(root, "FOIA")
|
||||
case collectionArchive:
|
||||
dir = filepath.Join(root, "SIDtoday-archive")
|
||||
}
|
||||
dir = filepath.Join(dir, d.id)
|
||||
b, err := json.MarshalIndent(d.meta, "", " ")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writeFixtureFile(t, filepath.Join(dir, "metadata.json"), string(b)+"\n")
|
||||
if !d.noPDF {
|
||||
writeFixtureFile(t, filepath.Join(dir, d.id+".pdf"), "%PDF-1.4\nfixture\n")
|
||||
}
|
||||
if d.text != "" {
|
||||
writeFixtureFile(t, filepath.Join(dir, d.id+".md"), d.text)
|
||||
}
|
||||
for _, p := range d.pages {
|
||||
writeFixtureFile(t, filepath.Join(dir, "images", p), fixtureJPEG)
|
||||
}
|
||||
for slug, imgs := range d.versions {
|
||||
vdir := filepath.Join(dir, "versions", slug)
|
||||
writeFixtureFile(t, filepath.Join(vdir, slug+".pdf"), "%PDF-1.4\n")
|
||||
writeFixtureFile(t, filepath.Join(vdir, slug+".md"),
|
||||
"# Copy\n\nA line only this copy has: unredacted FAIRVIEW detail.\n")
|
||||
for _, p := range imgs {
|
||||
writeFixtureFile(t, filepath.Join(vdir, "images", p), fixtureJPEG)
|
||||
}
|
||||
}
|
||||
}
|
||||
writeFixtureFile(t, filepath.Join(root, "glossary.json"), `[
|
||||
{"term": "XKEYSCORE", "definition": "A search system.", "aka": ["XKS"]}
|
||||
]`)
|
||||
|
||||
dataDir, projectDir = root, root
|
||||
devMode = false
|
||||
apiEnabled = true
|
||||
initAssets()
|
||||
if err := parseTemplates(); err != nil {
|
||||
t.Fatalf("templates: %v", err)
|
||||
}
|
||||
loadData()
|
||||
fixtureDir = root
|
||||
fixtureMux = withBaseHeaders(newMux())
|
||||
})
|
||||
if fixtureMux == nil {
|
||||
t.Fatal("fixture not built")
|
||||
}
|
||||
return fixtureMux, fixtureDir
|
||||
}
|
||||
|
||||
func fixtureDocs() []fixtureDoc {
|
||||
return []fixtureDoc{
|
||||
{
|
||||
id: "prism-slides",
|
||||
collection: collectionNews,
|
||||
text: "# PRISM\n\nCollection directly from the servers. XKEYSCORE ingests it.\n" +
|
||||
"Upstream collection feeds the corpus, and analysts were collecting metadata\n" +
|
||||
"from RT-RG dashboards. Müller-Maguhn reviewed the São Paulo cable tapping.\n",
|
||||
pages: []string{"page-1.jpg", "page-2.jpg"},
|
||||
versions: map[string][]string{"lemonde": {"page-1.jpg"}},
|
||||
meta: map[string]interface{}{
|
||||
"title": "PRISM Collection Slides",
|
||||
"type": "REPORTED",
|
||||
"description": "Slides describing PRISM collection.",
|
||||
"tags": []string{"PRISM", "collection", "cable tapping"},
|
||||
"pdf_sha256": strings.Repeat("a", 64),
|
||||
"aka_id": []map[string]string{{"id": "prism-slides-guardian"}},
|
||||
"document": map[string]interface{}{
|
||||
"creator": "NSA", "date_created": "2013-04-01",
|
||||
"programs": []string{"PRISM", "XKEYSCORE"},
|
||||
"classification": []string{"TOP SECRET"}, "dissemination": []string{"NOFORN"},
|
||||
"countries": []string{"United States", "Brazil"},
|
||||
"people": []string{"Snowden, Edward", "Müller-Maguhn, Andy"},
|
||||
"banner": "TOP SECRET//SI//NOFORN", "genre": "slides",
|
||||
},
|
||||
"publication": []map[string]interface{}{
|
||||
{"publisher": "The Guardian", "title": "NSA Prism program",
|
||||
"link": "https://example.org/prism", "date_published": "2013-06-06",
|
||||
"reporters": []string{"Greenwald, Glenn"}},
|
||||
{"publisher": "Le Monde", "link": "https://example.org/prism-fr",
|
||||
"date_published": "2013-06-08", "version": "lemonde"},
|
||||
{"publisher": "NRK", "link": "https://example.org/prism",
|
||||
"date_published": "2013-06-06", "reporting_only": true},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
// Shares the Guardian link with prism-slides, which is what
|
||||
// makes the "published in the same report" section exist.
|
||||
id: "prism-companion", collection: collectionNews,
|
||||
text: "# Companion\n\nA second document from the same story.\n",
|
||||
pages: []string{"page-1.jpg"},
|
||||
meta: map[string]interface{}{
|
||||
"title": "PRISM Companion Slide", "type": "REPORTED",
|
||||
"description": "A second slide from the same story.",
|
||||
"tags": []string{"PRISM"},
|
||||
"document": map[string]interface{}{
|
||||
"creator": "NSA", "date_created": "2013-04-02",
|
||||
"programs": []string{"PRISM"},
|
||||
},
|
||||
"publication": []map[string]interface{}{
|
||||
{"publisher": "The Guardian", "title": "NSA Prism program",
|
||||
"link": "https://example.org/prism", "date_published": "2013-06-06",
|
||||
"reporters": []string{"Greenwald, Glenn"}},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
id: "2003-05-29_SIDToday_-_Fixture_Issue",
|
||||
collection: collectionArchive,
|
||||
text: "# SIDtoday\n\nThe analyst cockpit and other matters. Upstream tapping was\n" +
|
||||
"discussed at length by the directorate, and the collection of that traffic\n" +
|
||||
"began only the following year.\n",
|
||||
pages: []string{"page-1.jpg"},
|
||||
meta: map[string]interface{}{
|
||||
"title": "A Day in the Life of a Fixture", "type": "SIDTODAY",
|
||||
"description": "An issue of SIDtoday.",
|
||||
"tags": []string{"analysis"},
|
||||
"document": map[string]interface{}{
|
||||
"creator": "NSA", "date_created": "2003-05-29",
|
||||
"classification": []string{"UNCLASSIFIED"}, "distribution": []string{},
|
||||
"series": map[string]interface{}{"name": "(U) A Day in the Life of...", "part": 1},
|
||||
},
|
||||
"publication": []map[string]interface{}{
|
||||
{"publisher": "The Intercept", "link": "https://example.org/sidtoday/1",
|
||||
"date_published": "2016-05-16"},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
id: "FISC_-_fixture_order",
|
||||
collection: collectionFOIA,
|
||||
text: "# Order\n\nThe Court finds the application satisfies the statute.\n",
|
||||
meta: map[string]interface{}{
|
||||
"title": "Fixture FISC Order", "type": "FOIA",
|
||||
"description": "A court order released under FOIA.",
|
||||
"document": map[string]interface{}{
|
||||
"creator": "FISC", "date_created": "2013-10-11",
|
||||
"classification": []string{"SECRET"},
|
||||
},
|
||||
"publication": []map[string]interface{}{{"publisher": "ODNI", "link": "https://example.org/foia/1"}},
|
||||
},
|
||||
},
|
||||
{
|
||||
id: "undated-fixture",
|
||||
collection: collectionNews,
|
||||
text: "# Undated\n\nNo document date at all.\n",
|
||||
meta: map[string]interface{}{
|
||||
"title": "Undated Fixture", "type": "REPORTED",
|
||||
"description": "A document with no date.",
|
||||
"document": map[string]interface{}{"creator": "GCHQ"},
|
||||
"publication": []map[string]interface{}{{"publisher": "The Intercept", "link": "https://example.org/undated"}},
|
||||
},
|
||||
},
|
||||
{
|
||||
id: "broken-no-pdf",
|
||||
collection: collectionNews,
|
||||
noPDF: true,
|
||||
meta: map[string]interface{}{"title": "Broken", "type": "REPORTED"},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// get runs one request through the real routing table and header wrapper.
|
||||
func get(t *testing.T, path string, cookies ...*http.Cookie) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
mux, _ := buildFixture(t)
|
||||
r := httptest.NewRequest("GET", path, nil)
|
||||
for _, c := range cookies {
|
||||
r.AddCookie(c)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
mux.ServeHTTP(w, r)
|
||||
return w
|
||||
}
|
||||
|
||||
// newPost builds a form POST for the state-changing endpoints.
|
||||
func newPost(path string) *http.Request {
|
||||
r := httptest.NewRequest("POST", path, strings.NewReader("back=/browse"))
|
||||
r.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
return r
|
||||
}
|
||||
|
||||
func record(h http.Handler, r *http.Request) *httptest.ResponseRecorder {
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, r)
|
||||
return w
|
||||
}
|
||||
|
||||
// withCorpus loads another corpus for one test and puts the fixture back
|
||||
// afterwards. Only possible because loadData is idempotent - every build*
|
||||
// starts from empty (TestLoadDataIsIdempotent).
|
||||
func withCorpus(t *testing.T, root string) {
|
||||
t.Helper()
|
||||
withDirs(t, root, root)
|
||||
}
|
||||
|
||||
// withDirs is withCorpus with the corpus and the project files apart, as
|
||||
// -root lays them out.
|
||||
func withDirs(t *testing.T, data, project string) {
|
||||
t.Helper()
|
||||
buildFixture(t) // make sure templates and the fixture path exist first
|
||||
prev := dataDir
|
||||
t.Cleanup(func() {
|
||||
dataDir, projectDir = prev, prev
|
||||
loadData()
|
||||
})
|
||||
dataDir, projectDir = data, project
|
||||
loadData()
|
||||
}
|
||||
|
||||
// httptest_NewGet is a plain GET request for the tests that drive a mux
|
||||
// directly rather than through get().
|
||||
func httptest_NewGet(path string) *http.Request {
|
||||
return httptest.NewRequest("GET", path, nil)
|
||||
}
|
||||
167
tests/flags_test.go
Normal file
167
tests/flags_test.go
Normal file
|
|
@ -0,0 +1,167 @@
|
|||
package main
|
||||
|
||||
// flags_test.go - what each command-line flag actually changes. The flags
|
||||
// are parsed in main(), which no test can call, so these exercise the state
|
||||
// the flags set and the behaviour that depends on it: the routes an
|
||||
// instance serves, the URLs it writes, the warnings it prints.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// -no-api: every /api/ path answers a JSON 403, and the UI stops offering
|
||||
// the API. The kill switch is one registration, so a future endpoint
|
||||
// inherits it - that is the property worth testing.
|
||||
func TestFlagNoAPI(t *testing.T) {
|
||||
buildFixture(t)
|
||||
prev := apiEnabled
|
||||
t.Cleanup(func() { apiEnabled = prev; fixtureMux = withBaseHeaders(newMux()) })
|
||||
|
||||
apiEnabled = false
|
||||
mux := withBaseHeaders(newMux())
|
||||
for _, p := range []string{"/api/docs", "/api/doc/prism-slides", "/api/search?q=x",
|
||||
"/api/facets", "/api/something-added-next-year"} {
|
||||
w := record(mux, httptest_NewGet(p))
|
||||
if w.Code != http.StatusForbidden {
|
||||
t.Errorf("%s with -no-api: status %d, want 403", p, w.Code)
|
||||
}
|
||||
var body map[string]string
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil || body["error"] == "" {
|
||||
t.Errorf("%s: body %q is not a JSON error", p, w.Body.String())
|
||||
}
|
||||
if ct := w.Header().Get("Content-Type"); !strings.HasPrefix(ct, "application/json") {
|
||||
t.Errorf("%s: Content-Type %q", p, ct)
|
||||
}
|
||||
}
|
||||
// The visitor's own export is NOT part of the API and stays available.
|
||||
if w := record(mux, httptest_NewGet("/pinned/export")); w.Code != 200 {
|
||||
t.Errorf("/pinned/export with -no-api: %d", w.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// -base: the absolute URLs in robots.txt and the sitemap follow
|
||||
// the canonical host instead of the request's Host header.
|
||||
func TestFlagBase(t *testing.T) {
|
||||
buildFixture(t)
|
||||
prev, prevMap := canonicalBase, corpus().canonicalSitemap
|
||||
t.Cleanup(func() {
|
||||
canonicalBase, corpus().canonicalSitemap = prev, prevMap
|
||||
corpus().buildCanonicalSitemap()
|
||||
})
|
||||
canonicalBase = "https://archive.example.org"
|
||||
corpus().buildCanonicalSitemap()
|
||||
|
||||
for _, c := range []struct{ path, want string }{
|
||||
{"/robots.txt", "Sitemap: https://archive.example.org/sitemap.xml"},
|
||||
{"/sitemap.xml", "<loc>https://archive.example.org/doc/prism-slides</loc>"},
|
||||
} {
|
||||
if body := get(t, c.path).Body.String(); !strings.Contains(body, c.want) {
|
||||
t.Errorf("%s does not carry %q", c.path, c.want)
|
||||
}
|
||||
}
|
||||
// A hostile Host header cannot reach a response once -base is set.
|
||||
r := httptest_NewGet("/robots.txt")
|
||||
r.Host = "evil.example.net"
|
||||
if body := record(fixtureMux, r).Body.String(); strings.Contains(body, "evil") {
|
||||
t.Errorf("the Host header leaked into robots.txt: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
// -base is validated, not trusted: anything but scheme://host is refused,
|
||||
// so the flag cannot produce URLs that do not resolve.
|
||||
func TestFlagBaseValidation(t *testing.T) {
|
||||
for _, ok := range []string{"https://a.org", "http://a.org", "https://a.org:8443", "https://a.org/"} {
|
||||
if _, err := parseCanonicalBase(ok); err != nil {
|
||||
t.Errorf("%s rejected: %v", ok, err)
|
||||
}
|
||||
}
|
||||
for _, bad := range []string{"", "a.org", "ftp://a.org", "https://", "https://a.org/path",
|
||||
"https://a.org?q=1", "https://a.org#f", "https://user@a.org", "://x"} {
|
||||
if _, err := parseCanonicalBase(bad); err == nil {
|
||||
t.Errorf("%q accepted", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// -check counts what it reports: the fixture holds one document the loader
|
||||
// refuses (no PDF) and one directory that looks like a document.
|
||||
func TestFlagCheckCounts(t *testing.T) {
|
||||
buildFixture(t)
|
||||
if loadErrors == 0 {
|
||||
t.Error("the fixture's PDF-less document should be an error")
|
||||
}
|
||||
if loadErrors+loadWarnings == 0 {
|
||||
t.Error("-check would exit 0 on a corpus with problems")
|
||||
}
|
||||
}
|
||||
|
||||
// -listen warns for anything but loopback, because binding elsewhere puts
|
||||
// the archive in front of the proxy instead of behind it.
|
||||
func TestFlagListenWarning(t *testing.T) {
|
||||
before := loadWarnings
|
||||
for _, quiet := range []string{"127.0.0.1:8080", "localhost:8080", "[::1]:8080", "127.0.0.53:80"} {
|
||||
warnNonLoopback(quiet)
|
||||
if loadWarnings != before {
|
||||
t.Errorf("%s warned", quiet)
|
||||
before = loadWarnings
|
||||
}
|
||||
}
|
||||
for _, loud := range []string{"0.0.0.0:8080", ":8080", "192.168.1.10:8080", "[::]:8080"} {
|
||||
warnNonLoopback(loud)
|
||||
if loadWarnings == before {
|
||||
t.Errorf("%s did not warn", loud)
|
||||
}
|
||||
before = loadWarnings
|
||||
}
|
||||
}
|
||||
|
||||
// -dev: templates come from disk and nothing is pre-rendered, so every page
|
||||
// must still render - the fallback path the pre-rendered blocks rely on.
|
||||
func TestFlagDevMode(t *testing.T) {
|
||||
buildFixture(t)
|
||||
prevDev, prevPre, prevTmpl := devMode, corpus().prerendered, tmplSet
|
||||
t.Cleanup(func() {
|
||||
devMode, corpus().prerendered, tmplSet = prevDev, prevPre, prevTmpl
|
||||
initAssets()
|
||||
})
|
||||
devMode = true
|
||||
initAssets() // templates/ and static/ from the working directory
|
||||
corpus().prerenderPages()
|
||||
if len(corpus().prerendered) != 0 {
|
||||
t.Error("dev mode pre-rendered pages; it must render them live")
|
||||
}
|
||||
for _, p := range []string{"/", "/browse", "/browse/program", "/explore/ids", "/glossary",
|
||||
"/articles", "/explore/terms", "/doc/prism-slides"} {
|
||||
if w := get(t, p); w.Code != 200 {
|
||||
t.Errorf("%s in dev mode: %d", p, w.Code)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// -root: the documents are read from its data/ and the files about the
|
||||
// archive from its project/.
|
||||
func TestRootDir(t *testing.T) {
|
||||
buildFixture(t)
|
||||
prevData, prevProject := dataDir, projectDir
|
||||
t.Cleanup(func() {
|
||||
dataDir, projectDir = prevData, prevProject
|
||||
loadData()
|
||||
})
|
||||
root := t.TempDir()
|
||||
writeFixtureFile(t, filepath.Join(root, "data", "doc-in-root", "metadata.json"), `{"title": "In the root"}`)
|
||||
writeFixtureFile(t, filepath.Join(root, "data", "doc-in-root", "doc-in-root.pdf"), "%PDF-1.4\n")
|
||||
writeFixtureFile(t, filepath.Join(root, "project", "glossary.json"), `[{"term": "INPROJECT", "definition": "Read from project/."}]`)
|
||||
setRoot(root)
|
||||
loadData()
|
||||
|
||||
if corpus().docIndex["doc-in-root"] == nil {
|
||||
t.Error("the document in data/ was not loaded")
|
||||
}
|
||||
if corpus().glossaryLookup("INPROJECT") == nil {
|
||||
t.Error("glossary.json in project/ was not loaded")
|
||||
}
|
||||
}
|
||||
272
tests/foia_test.go
Normal file
272
tests/foia_test.go
Normal file
|
|
@ -0,0 +1,272 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"html"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// foiaOn is the cookie of a reader who opted in to FOIA releases.
|
||||
var foiaOn = &http.Cookie{Name: "pref_include_foia", Value: "1"}
|
||||
|
||||
// In the consistency corpus "alpha" is in n1, n2 and the FOIA release f1.
|
||||
func TestSearchLeavesOutFOIAByDefault(t *testing.T) {
|
||||
withCorpus(t, writeConsistencyCorpus(t))
|
||||
|
||||
if got := resultTotal(t, "/search?q=alpha"); got != 2 {
|
||||
t.Errorf("q=alpha, no preference: %d results, want 2 (FOIA left out)", got)
|
||||
}
|
||||
body := get(t, "/search?q=alpha").Body.String()
|
||||
if strings.Contains(body, "left out") {
|
||||
t.Error("the page counts the FOIA releases it left out")
|
||||
}
|
||||
// The page's own links name the collections it searched, so a copied
|
||||
// link shows the same results to a reader with the other preference.
|
||||
if !strings.Contains(body, "coll=news&coll=sidtoday-archive") {
|
||||
t.Error("the page's links do not carry the default collections")
|
||||
}
|
||||
if got := resultTotal(t, "/search?q=alpha&coll=news&coll=sidtoday-archive&coll=foia"); got != 3 {
|
||||
t.Errorf("q=alpha in every collection: %d results, want 3", got)
|
||||
}
|
||||
|
||||
if got := resultTotal(t, "/search?q=alpha", foiaOn); got != 3 {
|
||||
t.Errorf("q=alpha with FOIA on: %d results, want 3", got)
|
||||
}
|
||||
if strings.Contains(get(t, "/search?q=alpha", foiaOn).Body.String(), "left out") {
|
||||
t.Error("FOIA on, yet the page says something was left out")
|
||||
}
|
||||
// An explicit collection choice wins over the preference.
|
||||
if got := resultTotal(t, "/search?q=alpha&coll=foia"); got != 1 {
|
||||
t.Errorf("q=alpha&coll=foia, no preference: %d results, want 1", got)
|
||||
}
|
||||
if got := resultTotal(t, "/search?q=alpha&coll=news", foiaOn); got != 2 {
|
||||
t.Errorf("q=alpha&coll=news with FOIA on: %d results, want 2", got)
|
||||
}
|
||||
}
|
||||
|
||||
// ZEBRA is on n1, n2, f1 (FOIA) and s1 (SIDtoday).
|
||||
func TestCategoryPagesLeaveOutFOIAByDefault(t *testing.T) {
|
||||
withCorpus(t, writeConsistencyCorpus(t))
|
||||
|
||||
if got := resultTotal(t, "/browse/program/ZEBRA"); got != 3 {
|
||||
t.Errorf("ZEBRA, no preference: %d documents, want 3", got)
|
||||
}
|
||||
if strings.Contains(get(t, "/browse/program/ZEBRA").Body.String(), "left out") {
|
||||
t.Error("the page still carries a FOIA notice")
|
||||
}
|
||||
if got := resultTotal(t, "/browse/program/ZEBRA", foiaOn); got != 4 {
|
||||
t.Errorf("ZEBRA with FOIA on: %d documents, want 4", got)
|
||||
}
|
||||
if strings.Contains(get(t, "/browse/program/ZEBRA", foiaOn).Body.String(), "left out") {
|
||||
t.Error("FOIA on, yet the page says FOIA releases are left out")
|
||||
}
|
||||
}
|
||||
|
||||
// Every total in a category table is the number its row's link lists, for
|
||||
// either preference.
|
||||
func TestCategoryTotalsMatchTheirPages(t *testing.T) {
|
||||
withCorpus(t, writeConsistencyCorpus(t))
|
||||
reRow := regexp.MustCompile(`(?s)<a href="([^"]+)">\s*([^<]+?)\s*</a>.*?</td>\s*<td class="num-col">(\d+)</td>`)
|
||||
for _, cookies := range [][]*http.Cookie{nil, {foiaOn}} {
|
||||
for _, slug := range []string{"program", "agency", "docdate", "published", "publisher"} {
|
||||
body := get(t, "/browse/"+slug, cookies...).Body.String()
|
||||
_, body, _ = strings.Cut(body, "<tbody>")
|
||||
body, _, _ = strings.Cut(body, "</tbody>")
|
||||
rows := reRow.FindAllStringSubmatch(body, -1)
|
||||
if len(rows) == 0 {
|
||||
t.Fatalf("/browse/%s: no rows", slug)
|
||||
}
|
||||
for _, row := range rows {
|
||||
want, _ := strconv.Atoi(row[3])
|
||||
if strings.Contains(row[1], "coll=") {
|
||||
t.Errorf("/browse/%s %s: row links with a collection scope: %s", slug, row[2], row[1])
|
||||
}
|
||||
if got := resultTotal(t, html.UnescapeString(row[1]), cookies...); got != want {
|
||||
t.Errorf("/browse/%s %s (FOIA on: %v): total %d, its page lists %d", slug, row[2], cookies != nil, want, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
off, on := get(t, "/browse/program").Body.String(), get(t, "/browse/program", foiaOn).Body.String()
|
||||
if strings.Contains(off, "Totals leave out") || strings.Contains(off, "sort=foia") {
|
||||
t.Error("FOIA off: the table still has a FOIA notice or column")
|
||||
}
|
||||
if !strings.Contains(on, "sort=foia") {
|
||||
t.Error("FOIA on: the table has no FOIA column")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRandomAndRelatedLeaveOutFOIA(t *testing.T) {
|
||||
withCorpus(t, writeConsistencyCorpus(t))
|
||||
|
||||
sawFOIA := false
|
||||
for i := 0; i < 300; i++ {
|
||||
if d := corpus().randomDoc(false); d == nil || d.Collection == collectionFOIA {
|
||||
t.Fatalf("randomDoc(false) = %v", d)
|
||||
}
|
||||
if corpus().randomDoc(true).Collection == collectionFOIA {
|
||||
sawFOIA = true
|
||||
}
|
||||
}
|
||||
if !sawFOIA {
|
||||
t.Error("randomDoc(true) never picked the FOIA release in 300 tries")
|
||||
}
|
||||
|
||||
corpus().archive.RelatedDocs["n2"] = []string{"f1", "n1"}
|
||||
corpus().archive.RelatedDocs["f1"] = []string{"n1"}
|
||||
body := get(t, "/doc/n2").Body.String()
|
||||
if strings.Contains(body, `href="/doc/f1"`) || !strings.Contains(body, `href="/doc/n1"`) {
|
||||
t.Error("/doc/n2, no preference: want n1 related and f1 left out")
|
||||
}
|
||||
if !strings.Contains(get(t, "/doc/n2", foiaOn).Body.String(), `href="/doc/f1"`) {
|
||||
t.Error("/doc/n2 with FOIA on: f1 is not related")
|
||||
}
|
||||
if !strings.Contains(get(t, "/doc/f1").Body.String(), `href="/doc/n1"`) {
|
||||
t.Error("/doc/f1: its related documents are gone")
|
||||
}
|
||||
}
|
||||
|
||||
// The home page counts the Snowden documents apart from the FOIA releases and
|
||||
// says how the preference treats them. The corpus: n1-n4, a1, d1 (news), s1
|
||||
// (SIDtoday archive), f1 (FOIA).
|
||||
func TestHomeStatesTheFOIAScope(t *testing.T) {
|
||||
withCorpus(t, writeConsistencyCorpus(t))
|
||||
body := get(t, "/").Body.String()
|
||||
for _, want := range []string{"holds 7 published Snowden documents. 6 documents are", "1 documents are the NSA newsletter",
|
||||
"also holds 1", "are left out by default but can be included via"} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("home page lacks %q", want)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(get(t, "/", foiaOn).Body.String(), "you have included via") {
|
||||
t.Error("FOIA on: the home page does not say so")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFOIAPreferenceIsSaved(t *testing.T) {
|
||||
withCorpus(t, writeConsistencyCorpus(t))
|
||||
if !strings.Contains(get(t, "/preferences").Body.String(), `name="include_foia"`) {
|
||||
t.Error("/preferences has no FOIA checkbox")
|
||||
}
|
||||
if readPrefs(httptest_NewGet("/")).IncludeFOIA {
|
||||
t.Error("FOIA releases are included by default")
|
||||
}
|
||||
}
|
||||
|
||||
// The form shows the scope the search ran with: FOIA left out by default
|
||||
// is an unchecked box, so submitting the form keeps it out.
|
||||
func TestSearchFormKeepsTheDefaultScope(t *testing.T) {
|
||||
withCorpus(t, writeConsistencyCorpus(t))
|
||||
|
||||
checked := regexp.MustCompile(`value="foia" checked`)
|
||||
if checked.MatchString(get(t, "/search?q=alpha").Body.String()) {
|
||||
t.Error("FOIA left out, yet its box is checked")
|
||||
}
|
||||
if !checked.MatchString(get(t, "/search?q=alpha", foiaOn).Body.String()) {
|
||||
t.Error("FOIA on, yet its box is not checked")
|
||||
}
|
||||
if got := resultTotal(t, "/search?q=alpha&coll=news&coll=sidtoday-archive&do_search=1"); got != 2 {
|
||||
t.Errorf("the form's resubmit finds %d, want 2", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A FOIA release's metadata links name every collection, so its value
|
||||
// pages list it under the default scope.
|
||||
func TestFOIADocLinksIncludeFOIA(t *testing.T) {
|
||||
buildFixture(t)
|
||||
// Every value link printed for the release - on its own page, on its
|
||||
// card and its table row in the FOIA tab, in a search result - opens a
|
||||
// page that lists the release.
|
||||
const id = "FISC_-_fixture_order"
|
||||
reValueLink := regexp.MustCompile(`href="(/browse/[a-z]+/[^"]+)"`)
|
||||
for _, page := range []string{"/doc/" + id, "/browse?collection=foia", "/browse?collection=foia&view=table", "/search?q=court&coll=foia"} {
|
||||
links := reValueLink.FindAllStringSubmatch(get(t, page).Body.String(), -1)
|
||||
if len(links) < 2 {
|
||||
t.Fatalf("%s: %d value links, want the release's agency and classification at least", page, len(links))
|
||||
}
|
||||
for _, m := range links {
|
||||
link := html.UnescapeString(m[1])
|
||||
if !strings.Contains(get(t, withQuery(link, "view=table")).Body.String(), `href="/doc/`+id+`"`) {
|
||||
t.Errorf("%s links %s, which does not list the release", page, link)
|
||||
}
|
||||
}
|
||||
}
|
||||
// /explore/classification counts every collection, and so do its links.
|
||||
for _, m := range regexp.MustCompile(`href="(/browse/[^"]+)"`).FindAllStringSubmatch(get(t, "/explore/classification").Body.String(), -1) {
|
||||
if !strings.Contains(m[1], "coll=foia") {
|
||||
t.Errorf("/explore/classification links %s without its scope", m[1])
|
||||
}
|
||||
}
|
||||
|
||||
body := get(t, "/doc/FISC_-_fixture_order").Body.String()
|
||||
m := regexp.MustCompile(`href="(/browse/agency/FISC[^"]*)"`).FindStringSubmatch(body)
|
||||
if m == nil || !strings.Contains(m[1], "coll=foia") {
|
||||
t.Fatalf("agency link %v does not include FOIA", m)
|
||||
}
|
||||
if got := resultTotal(t, html.UnescapeString(m[1])); got != 1 {
|
||||
t.Errorf("the agency link lists %d documents, want 1", got)
|
||||
}
|
||||
if strings.Contains(get(t, "/doc/FISC_-_fixture_order", foiaOn).Body.String(), "/browse/agency/FISC?") {
|
||||
t.Error("FOIA on: the links need no scope")
|
||||
}
|
||||
}
|
||||
|
||||
// On a publisher page scoped to a publication year, the links built from
|
||||
// its documents carry that year.
|
||||
func TestPublisherYearLinksCarryTheYear(t *testing.T) {
|
||||
buildFixture(t)
|
||||
body := get(t, "/browse/publisher/The%20Intercept?year=2016").Body.String()
|
||||
if !strings.Contains(body, "date_pub_start=2016") {
|
||||
t.Error("no link on the year-scoped page carries the publication year")
|
||||
}
|
||||
}
|
||||
|
||||
// collScope is the one place a page's collection scope is decided: FOIA
|
||||
// releases are opt-in when the URL names no collection, an explicit coll=
|
||||
// wins, and only the search pins a defaulted scope into its links.
|
||||
func TestCollScope(t *testing.T) {
|
||||
off, on := Prefs{}, Prefs{IncludeFOIA: true}
|
||||
none := url.Values{}
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
v url.Values
|
||||
p Prefs
|
||||
pin bool
|
||||
foia, def bool
|
||||
qs string
|
||||
}{
|
||||
{"default, browse", none, off, false, false, true, ""},
|
||||
{"default, search", none, off, true, false, true, "coll=news&coll=sidtoday-archive"},
|
||||
{"opted in, browse", none, on, false, true, true, ""},
|
||||
{"opted in, search", none, on, true, true, true, "coll=news&coll=sidtoday-archive&coll=foia"},
|
||||
{"explicit foia", url.Values{"coll": {"foia"}}, off, false, true, false, "coll=foia"},
|
||||
{"explicit news over opt-in", url.Values{"coll": {"news"}}, on, false, false, false, "coll=news"},
|
||||
{"explicit everything", url.Values{"coll": {"news", "foia", "sidtoday-archive"}}, off, false, true, false,
|
||||
"coll=news&coll=sidtoday-archive&coll=foia"},
|
||||
} {
|
||||
s := pageScope(c.v, c.p, c.pin)
|
||||
foia := s.shows(&Document{Collection: collectionFOIA})
|
||||
if foia != c.foia || s.defaulted != c.def || s.qs() != c.qs || s.shown[collectionFOIA] != c.foia {
|
||||
t.Errorf("%s: shows FOIA %v, defaulted %v, qs %q; want %v, %v, %q",
|
||||
c.name, foia, s.defaulted, s.qs(), c.foia, c.def, c.qs)
|
||||
}
|
||||
}
|
||||
|
||||
// A FOIA release's links widen the default to the whole archive and
|
||||
// say so; a reader who opted in needs no widening.
|
||||
if got := pageScope(none, off, false).including(collectionFOIA).suffix(); got != "?coll=news&coll=sidtoday-archive&coll=foia" {
|
||||
t.Errorf("widened default: %q", got)
|
||||
}
|
||||
if got := pageScope(none, on, false).including(collectionFOIA).suffix(); got != "" {
|
||||
t.Errorf("opted in, widened: %q, want no scope", got)
|
||||
}
|
||||
// and adds to the links, not to the documents.
|
||||
s := pageScope(none, off, false).and("date_pub_start=2014")
|
||||
if s.link("/search?q=x") != "/search?q=x&date_pub_start=2014" || s.shows(&Document{Collection: collectionFOIA}) {
|
||||
t.Errorf("and: %q", s.link("/search?q=x"))
|
||||
}
|
||||
}
|
||||
5
tests/go.mod
Normal file
5
tests/go.mod
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
// Keeps this directory out of the snowchive module's ./... patterns. The
|
||||
// files here are package main's tests; tests/run.sh compiles them with it.
|
||||
module snowchive/tests
|
||||
|
||||
go 1.22
|
||||
53
tests/headings_test.go
Normal file
53
tests/headings_test.go
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
var reHeadingTag = regexp.MustCompile(`<h([1-6])\b`)
|
||||
|
||||
func TestTranscriptHeadingsNestUnderThePage(t *testing.T) {
|
||||
for src, want := range map[string]string{
|
||||
"# A\n\n## B\n\n#### C\n\n## D": `<h2 class="md-h1">A</h2> <h3 class="md-h2">B</h3> <h4 class="md-h4">C</h4> <h3 class="md-h2">D</h3>`,
|
||||
"### A\n\n### B\n\n#### C": `<h2 class="md-h3">A</h2> <h2 class="md-h3">B</h2> <h3 class="md-h4">C</h3>`,
|
||||
"```\n# not a heading\n```\n\n## A": `<h2 class="md-h2">A</h2>`,
|
||||
} {
|
||||
var got []string
|
||||
for _, line := range strings.Split(Render(src), "\n") {
|
||||
if strings.HasPrefix(line, "<h") {
|
||||
got = append(got, line)
|
||||
}
|
||||
}
|
||||
if g := strings.Join(got, " "); g != want {
|
||||
t.Errorf("Render(%q) headings:\n got %s\nwant %s", src, g, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPagesHaveOneH1AndNoSkippedLevels(t *testing.T) {
|
||||
for _, path := range []string{
|
||||
"/", "/browse", "/browse?view=table", "/browse?collection=sidtoday-archive",
|
||||
"/browse?collection=sidtoday-archive&view=card", "/browse?collection=foia",
|
||||
"/browse/agency/NSA", "/search?q=prism", "/search?q=zzzqqq", "/doc/prism-slides",
|
||||
"/doc/prism-slides/versions/lemonde", "/glossary", "/articles", "/about", "/pinned",
|
||||
} {
|
||||
body := get(t, path).Body.String()
|
||||
h1, prev := 0, 0
|
||||
for _, m := range reHeadingTag.FindAllStringSubmatch(body, -1) {
|
||||
level, _ := strconv.Atoi(m[1])
|
||||
if level == 1 {
|
||||
h1++
|
||||
}
|
||||
if level > prev+1 {
|
||||
t.Errorf("%s: <h%d> follows <h%d>", path, level, prev)
|
||||
}
|
||||
prev = level
|
||||
}
|
||||
if h1 != 1 {
|
||||
t.Errorf("%s: %d <h1>, want 1", path, h1)
|
||||
}
|
||||
}
|
||||
}
|
||||
231
tests/http_test.go
Normal file
231
tests/http_test.go
Normal file
|
|
@ -0,0 +1,231 @@
|
|||
package main
|
||||
|
||||
// http_test.go - protocol behaviour the browser and the proxy rely on:
|
||||
// HEAD, byte ranges, conditional requests, and method handling. net/http
|
||||
// implements most of it, but only if the handlers hand it the right thing -
|
||||
// and a download path that loses Range support breaks resuming a 120 MB PDF
|
||||
// and the page-by-page fetches a PDF viewer makes.
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestHTTPHead(t *testing.T) {
|
||||
buildFixture(t)
|
||||
for _, p := range []string{"/", "/browse", "/doc/prism-slides",
|
||||
"/files/prism-slides/prism-slides.pdf", "/api/facets", "/static/style.css"} {
|
||||
r := httptest.NewRequest("HEAD", p, nil)
|
||||
w := httptest.NewRecorder()
|
||||
fixtureMux.ServeHTTP(w, r)
|
||||
if w.Code != 200 {
|
||||
t.Errorf("HEAD %s: %d", p, w.Code)
|
||||
continue
|
||||
}
|
||||
// net/http drops the body for HEAD at the transport layer, so the
|
||||
// recorder still sees one; the headers are what matters here.
|
||||
if w.Header().Get("Content-Type") == "" {
|
||||
t.Errorf("HEAD %s: no Content-Type", p)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Range on a document file. http.ServeFile implements it - this test is
|
||||
// here so a future change to the download path (a buffered writer, a
|
||||
// compressing wrapper, a manual io.Copy) cannot quietly remove it.
|
||||
func TestHTTPRangeOnFiles(t *testing.T) {
|
||||
buildFixture(t)
|
||||
const path = "/files/prism-slides/prism-slides.pdf"
|
||||
full := get(t, path)
|
||||
body := full.Body.String()
|
||||
if full.Header().Get("Accept-Ranges") != "bytes" {
|
||||
t.Fatalf("no Accept-Ranges on a document file: %v", full.Header())
|
||||
}
|
||||
|
||||
req := func(rangeHdr string) *httptest.ResponseRecorder {
|
||||
r := httptest.NewRequest("GET", path, nil)
|
||||
r.Header.Set("Range", rangeHdr)
|
||||
w := httptest.NewRecorder()
|
||||
fixtureMux.ServeHTTP(w, r)
|
||||
return w
|
||||
}
|
||||
|
||||
w := req("bytes=0-3")
|
||||
if w.Code != http.StatusPartialContent {
|
||||
t.Fatalf("Range bytes=0-3: status %d, want 206", w.Code)
|
||||
}
|
||||
if got, want := w.Body.String(), body[:4]; got != want {
|
||||
t.Errorf("range body %q, want %q", got, want)
|
||||
}
|
||||
if cr := w.Header().Get("Content-Range"); cr != fmt.Sprintf("bytes 0-3/%d", len(body)) {
|
||||
t.Errorf("Content-Range %q", cr)
|
||||
}
|
||||
|
||||
// A suffix range: what a resumed download asks for.
|
||||
w = req(fmt.Sprintf("bytes=%d-", len(body)-4))
|
||||
if w.Code != http.StatusPartialContent || w.Body.String() != body[len(body)-4:] {
|
||||
t.Errorf("suffix range: %d %q", w.Code, w.Body.String())
|
||||
}
|
||||
|
||||
// Past the end is a 416 with the length, not a 200 of the whole file.
|
||||
w = req(fmt.Sprintf("bytes=%d-%d", len(body)+10, len(body)+20))
|
||||
if w.Code != http.StatusRequestedRangeNotSatisfiable {
|
||||
t.Errorf("unsatisfiable range: status %d, want 416", w.Code)
|
||||
}
|
||||
|
||||
// A malformed Range is refused with 416 - net/http's choice, not this
|
||||
// server's; it is recorded here so a change in behaviour is noticed.
|
||||
if w = req("bytes=abc"); w.Code != http.StatusRequestedRangeNotSatisfiable {
|
||||
t.Errorf("malformed Range: status %d, want 416", w.Code)
|
||||
}
|
||||
// An unknown range unit is refused the same way. RFC 7233 says to
|
||||
// ignore it and serve the whole representation; net/http does not, and
|
||||
// nothing this archive serves sends one. Recorded, not worked around.
|
||||
if w = req("pages=1-2"); w.Code != http.StatusRequestedRangeNotSatisfiable {
|
||||
t.Errorf("unknown range unit: status %d, want net/http's 416", w.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// Conditional requests: a document file and a static asset both carry
|
||||
// validators, so a reader's second visit is a 304 instead of a re-download.
|
||||
func TestHTTPConditional(t *testing.T) {
|
||||
buildFixture(t)
|
||||
// A document file comes off disk and carries a modification time.
|
||||
const doc = "/files/prism-slides/prism-slides.pdf"
|
||||
lm := get(t, doc).Header().Get("Last-Modified")
|
||||
if lm == "" {
|
||||
t.Fatalf("%s: no Last-Modified", doc)
|
||||
}
|
||||
r := httptest.NewRequest("GET", doc, nil)
|
||||
r.Header.Set("If-Modified-Since", lm)
|
||||
w := httptest.NewRecorder()
|
||||
fixtureMux.ServeHTTP(w, r)
|
||||
if w.Code != http.StatusNotModified || w.Body.Len() != 0 {
|
||||
t.Errorf("%s with If-Modified-Since: status %d, %d bytes", doc, w.Code, w.Body.Len())
|
||||
}
|
||||
|
||||
// An embedded asset has no modification time - embed.FS zeroes it - so
|
||||
// it carries a content ETag instead. Without one, every page view
|
||||
// re-downloads the stylesheet on an unstamped build.
|
||||
for _, asset := range []string{"/static/style.css", "/static/main.js"} {
|
||||
first := get(t, asset)
|
||||
tag := first.Header().Get("ETag")
|
||||
if tag == "" {
|
||||
t.Errorf("%s: no ETag", asset)
|
||||
continue
|
||||
}
|
||||
r := httptest.NewRequest("GET", asset, nil)
|
||||
r.Header.Set("If-None-Match", tag)
|
||||
w := httptest.NewRecorder()
|
||||
fixtureMux.ServeHTTP(w, r)
|
||||
if w.Code != http.StatusNotModified || w.Body.Len() != 0 {
|
||||
t.Errorf("%s with If-None-Match: status %d, %d bytes (want an empty 304)",
|
||||
asset, w.Code, w.Body.Len())
|
||||
}
|
||||
// A different asset's tag must not satisfy it.
|
||||
r = httptest.NewRequest("GET", asset, nil)
|
||||
r.Header.Set("If-None-Match", `"0000000000000000"`)
|
||||
w = httptest.NewRecorder()
|
||||
fixtureMux.ServeHTTP(w, r)
|
||||
if w.Code != 200 {
|
||||
t.Errorf("%s with a stale ETag: status %d, want the asset", asset, w.Code)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Method handling: the routes declare GET or POST, so anything else is a
|
||||
// 405 from the mux rather than a handler running on the wrong verb.
|
||||
func TestHTTPMethods(t *testing.T) {
|
||||
buildFixture(t)
|
||||
for _, c := range []struct {
|
||||
method, path string
|
||||
want int
|
||||
}{
|
||||
// Methods the archive never answers stop at the wrapper.
|
||||
{"PUT", "/api/docs", http.StatusMethodNotAllowed},
|
||||
{"DELETE", "/doc/prism-slides", http.StatusMethodNotAllowed},
|
||||
{"OPTIONS", "/", http.StatusMethodNotAllowed},
|
||||
{"TRACE", "/browse", http.StatusMethodNotAllowed},
|
||||
// A supported method on the wrong route is a 404: the catch-all
|
||||
// route claims every path, which is what keeps unknown paths on the
|
||||
// archive's own 404 page, and it claims them for POST too.
|
||||
{"POST", "/browse", http.StatusNotFound},
|
||||
{"GET", "/preferences/save", http.StatusNotFound},
|
||||
{"GET", "/pin/prism-slides", http.StatusNotFound},
|
||||
} {
|
||||
r := httptest.NewRequest(c.method, c.path, nil)
|
||||
w := httptest.NewRecorder()
|
||||
fixtureMux.ServeHTTP(w, r)
|
||||
if w.Code != c.want {
|
||||
t.Errorf("%s %s: status %d, want %d", c.method, c.path, w.Code, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Odd but legal requests must not break anything: repeated parameters,
|
||||
// unknown parameters, a very long path, an empty query.
|
||||
func TestHTTPOddRequests(t *testing.T) {
|
||||
buildFixture(t)
|
||||
for _, c := range []struct {
|
||||
path string
|
||||
want int
|
||||
}{
|
||||
{"/browse?sort=title&sort=date", 200},
|
||||
{"/browse?unknown=1&another=2", 200},
|
||||
{"/search?q=a&q=b&do_search=1", 200},
|
||||
{"/browse/program/" + strings.Repeat("x", 2000), 200}, // an empty page, not a crash
|
||||
{"/doc/" + strings.Repeat("x", 2000), 404},
|
||||
{"/browse?", 200},
|
||||
{"/search?do_search=1&fqv=&fqv=&fqv=", 200},
|
||||
} {
|
||||
if w := get(t, c.path); w.Code != c.want {
|
||||
t.Errorf("%.60s: status %d, want %d", c.path, w.Code, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// With -accel-files the file is nginx's to send: the handler answers only
|
||||
// the files it would have served itself, with the redirect and no body.
|
||||
func TestAccelFiles(t *testing.T) {
|
||||
buildFixture(t)
|
||||
accelPrefix = "/_files/"
|
||||
defer func() { accelPrefix = "" }()
|
||||
|
||||
w := get(t, "/files/prism-slides/prism-slides.pdf")
|
||||
rel, err := filepath.Rel(dataDir, filepath.Join(corpus().docIndex["prism-slides"].Dir, "prism-slides.pdf"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got, want := w.Header().Get("X-Accel-Redirect"), "/_files/"+filepath.ToSlash(rel); got != want {
|
||||
t.Errorf("X-Accel-Redirect %q, want %q", got, want)
|
||||
}
|
||||
if w.Code != http.StatusOK || w.Body.Len() != 0 {
|
||||
t.Errorf("accel answer: status %d, %d body bytes, want an empty 200", w.Code, w.Body.Len())
|
||||
}
|
||||
if w.Header().Get("Cache-Control") == "" {
|
||||
t.Error("accel answer carries no Cache-Control for nginx to pass on")
|
||||
}
|
||||
|
||||
for _, p := range []string{"/files/prism-slides/zz-stray.pdf", "/files/prism-slides/../glossary.json", "/files/no-such-doc/x.pdf"} {
|
||||
w := get(t, p)
|
||||
if w.Header().Get("X-Accel-Redirect") != "" || w.Code == http.StatusOK {
|
||||
t.Errorf("%s: status %d, X-Accel-Redirect %q - a file the server does not serve was handed to nginx", p, w.Code, w.Header().Get("X-Accel-Redirect"))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAccelPathEscapes(t *testing.T) {
|
||||
old, oldPrefix := dataDir, accelPrefix
|
||||
dataDir, accelPrefix = "/srv/data", "/_files/"
|
||||
defer func() { dataDir, accelPrefix = old, oldPrefix }()
|
||||
if got, ok := accelPath("/srv/data/FOIA/a doc/50%?.pdf"); !ok || got != "/_files/FOIA/a%20doc/50%25%3F.pdf" {
|
||||
t.Errorf("accelPath = %q, %v", got, ok)
|
||||
}
|
||||
if got, ok := accelPath("/srv/other/x.pdf"); ok {
|
||||
t.Errorf("a file outside the data directory mapped to %q", got)
|
||||
}
|
||||
}
|
||||
99
tests/ids_test.go
Normal file
99
tests/ids_test.go
Normal file
|
|
@ -0,0 +1,99 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The /explore/ids table: one row per document in id order, versions nested
|
||||
// under their document, and a checksum file that sha256sum -c can read from
|
||||
// the corpus root.
|
||||
func TestBuildIDRows(t *testing.T) {
|
||||
oldDocs, oldDir := corpus().archive.Docs, dataDir
|
||||
t.Cleanup(func() {
|
||||
// The id tables are package state the routes serve: rebuild them
|
||||
// from the restored corpus, or the fake rows below leak into
|
||||
// every later test.
|
||||
corpus().archive.Docs, dataDir = oldDocs, oldDir
|
||||
corpus().buildIDRows()
|
||||
})
|
||||
|
||||
dataDir = "/corpus"
|
||||
corpus().archive.Docs = []Document{
|
||||
{
|
||||
ID: "zebra", Dir: filepath.Join(dataDir, "FOIA", "zebra"), Collection: "foia",
|
||||
PDFFile: "zebra.pdf", PDFSize: 2 << 20, Pages: 3,
|
||||
},
|
||||
{
|
||||
ID: "alpha", Dir: filepath.Join(dataDir, "alpha"), Collection: "news",
|
||||
PDFFile: "alpha.pdf", PDFSize: 1500, Pages: 1,
|
||||
PDFSha256: "abc123", AkaID: []AkaEntry{{ID: "alpha-lemonde"}},
|
||||
Versions: []DocVersion{{Slug: "entier", PDFFile: "copy.pdf", PDFSize: 4096, Images: []string{"a", "b"}, SHA: "def456"}},
|
||||
},
|
||||
}
|
||||
corpus().buildIDRows()
|
||||
|
||||
if len(corpus().idRows) != 2 || corpus().idRows[0].ID != "alpha" || corpus().idRows[1].ID != "zebra" {
|
||||
t.Fatalf("rows out of order: %v", corpus().idRows)
|
||||
}
|
||||
if got := corpus().idRows[0].Size; got != "1 KB" {
|
||||
t.Errorf("size %q, want 1 KB", got)
|
||||
}
|
||||
if v := corpus().idRows[0].Versions; len(v) != 1 || v[0].Pages != 2 || v[0].Size != "4 KB" || v[0].SHA != "def456" {
|
||||
t.Errorf("version row %+v", v)
|
||||
}
|
||||
if len(corpus().idRows[1].Versions) != 0 {
|
||||
t.Errorf("zebra has no versions, got %+v", corpus().idRows[1].Versions)
|
||||
}
|
||||
|
||||
// sha256sum -c format: checksum, two spaces, path from the corpus root,
|
||||
// a document's versions after it. Files without a checksum are absent.
|
||||
if corpus().idChecksums != "abc123 alpha/alpha.pdf\ndef456 alpha/versions/entier/copy.pdf\n" {
|
||||
t.Errorf("checksums %q", corpus().idChecksums)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHumanSize(t *testing.T) {
|
||||
for n, want := range map[int64]string{0: "", -1: "", 900: "900 B", 2048: "2 KB",
|
||||
3 << 20: "3.0 MB", 5 << 30: "5.00 GB"} {
|
||||
if got := humanSize(n); got != want {
|
||||
t.Errorf("humanSize(%d) = %q, want %q", n, got, want)
|
||||
}
|
||||
}
|
||||
if !strings.HasSuffix(humanSize(1<<20+1<<19), " MB") {
|
||||
t.Errorf("1.5 MiB should render in MB")
|
||||
}
|
||||
}
|
||||
|
||||
// A version's PDF checksum comes from the metadata's
|
||||
// "versions"/<slug>/pdf_sha256 only; a version without one is warned about
|
||||
// and has no line in ids.txt.
|
||||
func TestVersionChecksums(t *testing.T) {
|
||||
root := assemblyCorpus(t, "")
|
||||
writeAssemblyDoc(t, root, "listed", map[string]interface{}{
|
||||
"title": "Listed",
|
||||
"versions": map[string]interface{}{"copy": map[string]interface{}{"pdf_sha256": " ABC999 "}},
|
||||
}, []string{"page-1.jpg"}, map[string][]string{"copy": {"page-1.jpg"}})
|
||||
var logged bytes.Buffer
|
||||
log.SetOutput(&logged)
|
||||
withCorpus(t, root)
|
||||
log.SetOutput(os.Stderr)
|
||||
|
||||
if v := corpus().docIndex["listed"].VersionFor("copy"); v == nil || v.SHA != "abc999" {
|
||||
t.Errorf("metadata version checksum: %+v, want abc999 from the metadata", v)
|
||||
}
|
||||
if v := corpus().docIndex["deck"].VersionFor("fuller"); v == nil || v.SHA != "" {
|
||||
t.Errorf("a version without pdf_sha256 in the metadata got a checksum: %+v", v)
|
||||
}
|
||||
if !strings.Contains(logged.String(), "deck: versions/fuller has no pdf_sha256") {
|
||||
t.Errorf("no warning for the version without a checksum:\n%s", logged.String())
|
||||
}
|
||||
if !strings.Contains(corpus().idChecksums, "abc999 listed/versions/copy/copy.pdf\n") ||
|
||||
strings.Contains(corpus().idChecksums, "deck/versions/fuller/") {
|
||||
t.Errorf("ids.txt version lines:\n%s", corpus().idChecksums)
|
||||
}
|
||||
}
|
||||
53
tests/image_sizes_test.go
Normal file
53
tests/image_sizes_test.go
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestPageImagesCarryTheirSize(t *testing.T) {
|
||||
for _, path := range []string{
|
||||
"/doc/prism-slides",
|
||||
"/doc/prism-slides/pages",
|
||||
"/doc/prism-slides/versions/lemonde",
|
||||
"/doc/prism-slides/versions/lemonde/pages",
|
||||
} {
|
||||
body := get(t, path).Body.String()
|
||||
if n := strings.Count(body, `page-1.jpg" width="4" height="3"`); n < 2 {
|
||||
t.Errorf("%s: %d page-1 images with their size, want the page and its full-screen view", path, n)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssemblySlidesCarryTheirSize(t *testing.T) {
|
||||
root := assemblyCorpus(t, `[{"id": "sized", "title": "Sized", "slides": [{"doc": "deck", "pages": "1"}]}]`)
|
||||
withCorpus(t, root)
|
||||
for _, path := range []string{"/explore/assemblies", "/explore/assemblies/sized"} {
|
||||
if body := get(t, path).Body.String(); !strings.Contains(body, `page-1.jpg" width="4" height="3"`) {
|
||||
t.Errorf("%s: slide image without its size", path)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnreadablePageImageWarns(t *testing.T) {
|
||||
root := assemblyCorpus(t, "")
|
||||
writeAssemblyDoc(t, root, "broken", map[string]interface{}{"title": "Broken"}, []string{"page-1.jpg"}, nil)
|
||||
writeFixtureFile(t, filepath.Join(root, "broken", "images", "page-1.jpg"), "not an image")
|
||||
var logged bytes.Buffer
|
||||
log.SetOutput(&logged)
|
||||
withCorpus(t, root)
|
||||
log.SetOutput(os.Stderr)
|
||||
|
||||
if !strings.Contains(logged.String(), "page image unreadable, served without its size: broken/images/page-1.jpg") {
|
||||
t.Errorf("no warning for the unreadable image:\n%s", logged.String())
|
||||
}
|
||||
body := get(t, "/doc/broken").Body.String()
|
||||
if !strings.Contains(body, `<img src="/files/broken/images/page-1.jpg"
|
||||
`) || strings.Contains(body, `page-1.jpg" width=`) {
|
||||
t.Errorf("unreadable image: want it served without width/height")
|
||||
}
|
||||
}
|
||||
277
tests/invariants_test.go
Normal file
277
tests/invariants_test.go
Normal file
|
|
@ -0,0 +1,277 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"html"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The archive's central promise: a number on a link is the number the click
|
||||
// returns. Facet counts, refine boxes and histogram bars all assert it, and
|
||||
// every one of them is computed by different code from the page it opens.
|
||||
// These tests walk the real pages and follow their own links.
|
||||
|
||||
var (
|
||||
reFacetLink = regexp.MustCompile(`<a class="facet-link" href="([^"]+)">.*?<span class="facet-count">(\d+)</span></a>`)
|
||||
reBarLink = regexp.MustCompile(`<a class="hist-col" href="([^"]+)"[^>]*>\s*<span class="hist-count">(\d+)</span>`)
|
||||
reTotal = regexp.MustCompile(`([\d,]+)\s+results?`)
|
||||
reListTotal = regexp.MustCompile(`([\d,]+)\s+documents?`)
|
||||
)
|
||||
|
||||
// resultTotal reads the count a /search or a list page reports.
|
||||
func resultTotal(t *testing.T, path string, cookies ...*http.Cookie) int {
|
||||
t.Helper()
|
||||
body := get(t, path, cookies...).Body.String()
|
||||
re := reTotal
|
||||
if !strings.HasPrefix(path, "/search") {
|
||||
re = reListTotal
|
||||
}
|
||||
m := re.FindStringSubmatch(body)
|
||||
if m == nil {
|
||||
t.Fatalf("%s: no result count in the page", path)
|
||||
}
|
||||
n, err := strconv.Atoi(strings.ReplaceAll(m[1], ",", ""))
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", path, err)
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// followLinks checks every (link, count) pair a page prints, as the reader
|
||||
// with these cookies sees it.
|
||||
func followLinks(t *testing.T, page string, re *regexp.Regexp, what string, cookies ...*http.Cookie) int {
|
||||
t.Helper()
|
||||
body := get(t, page, cookies...).Body.String()
|
||||
pairs := re.FindAllStringSubmatch(body, -1)
|
||||
for _, p := range pairs {
|
||||
link := html.UnescapeString(p[1])
|
||||
shown, _ := strconv.Atoi(p[2])
|
||||
if got := resultTotal(t, link, cookies...); got != shown {
|
||||
t.Errorf("%s: %s says %d, its link returns %d\n %s", page, what, shown, got, link)
|
||||
}
|
||||
}
|
||||
return len(pairs)
|
||||
}
|
||||
|
||||
// A category page lists exactly as many documents as its facet counts.
|
||||
func TestInvariantFacetCountsMatchPages(t *testing.T) {
|
||||
buildFixture(t)
|
||||
checked := 0
|
||||
for ci := range categories {
|
||||
c := &categories[ci]
|
||||
for val, want := range c.Facet(corpus()) {
|
||||
if got := len(corpus().categoryMatches(c.Slug, val)); got != want {
|
||||
t.Errorf("%s=%q: facet says %d, the page lists %d", c.Slug, val, want, got)
|
||||
}
|
||||
checked++
|
||||
}
|
||||
}
|
||||
if checked == 0 {
|
||||
t.Fatal("no facet values in the fixture")
|
||||
}
|
||||
}
|
||||
|
||||
// Refine links and histogram bars open searches that must return what they
|
||||
// promise - the two places where a count and its click are computed by
|
||||
// different code paths.
|
||||
func TestInvariantRefineAndBarCounts(t *testing.T) {
|
||||
pages := []string{
|
||||
"/browse/agency/NSA",
|
||||
"/browse/program/PRISM",
|
||||
"/browse/classification/TOP%20SECRET",
|
||||
"/search?q=prism&do_search=1",
|
||||
"/search?do_search=1&fqv=NSA&fqf=CR&fqc=and",
|
||||
}
|
||||
facets, bars := 0, 0
|
||||
for _, p := range pages {
|
||||
facets += followLinks(t, p, reFacetLink, "refine link")
|
||||
bars += followLinks(t, p, reBarLink, "histogram bar")
|
||||
}
|
||||
if facets == 0 || bars == 0 {
|
||||
t.Fatalf("nothing exercised: %d refine links, %d bars", facets, bars)
|
||||
}
|
||||
t.Logf("checked %d refine links and %d histogram bars", facets, bars)
|
||||
}
|
||||
|
||||
// The home page is scoped to the reported collection while the category
|
||||
// pages follow the reader's scope, so its bars and lists link to those
|
||||
// pages with the collection named. Every one of them has to be the number
|
||||
// its click returns.
|
||||
func TestInvariantHomeCounts(t *testing.T) {
|
||||
body := get(t, "/").Body.String()
|
||||
pairs := regexp.MustCompile(`<a class="pill" href="([^"]+)">[^<]*<span class="pill-count">(\d+)</span>`).FindAllStringSubmatch(body, -1)
|
||||
lists := regexp.MustCompile(`<li><a href="(/browse/[^"]+)">[^<]*</a> <span class="muted">(\d+)</span>`).FindAllStringSubmatch(body, -1)
|
||||
if len(lists) < 5 {
|
||||
t.Fatalf("only %d counted list links on the home page - the stat lists should link to their browse pages", len(lists))
|
||||
}
|
||||
pairs = append(pairs, lists...)
|
||||
pairs = append(pairs, regexp.MustCompile(`<a class="hist-col" href="([^"]+)" title="(\d+) documents`).FindAllStringSubmatch(body, -1)...)
|
||||
if len(pairs) < 5 {
|
||||
t.Fatalf("only %d counted links on the home page", len(pairs))
|
||||
}
|
||||
for _, m := range pairs {
|
||||
link, shown := html.UnescapeString(m[1]), m[2]
|
||||
if got := resultTotal(t, link); got != mustAtoi(t, shown) {
|
||||
t.Errorf("home says %s, its link returns %d: %s", shown, got, link)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func mustAtoi(t *testing.T, s string) int {
|
||||
t.Helper()
|
||||
n, err := strconv.Atoi(s)
|
||||
if err != nil {
|
||||
t.Fatalf("%q: %v", s, err)
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// The type pills follow the same rule: the count is over the full match set,
|
||||
// and clicking one narrows to exactly that many.
|
||||
func TestInvariantTypePillCounts(t *testing.T) {
|
||||
body := get(t, "/browse").Body.String()
|
||||
re := regexp.MustCompile(`href="([^"]*type=[^"]*)"[^>]*>\s*([A-Z]+)\s*<span[^>]*>\(?(\d+)\)?</span>`)
|
||||
for _, m := range re.FindAllStringSubmatch(body, -1) {
|
||||
link := html.UnescapeString(m[1])
|
||||
shown, _ := strconv.Atoi(m[3])
|
||||
if got := resultTotal(t, link); got != shown {
|
||||
t.Errorf("type pill %s says %d, its link returns %d (%s)", m[2], shown, got, link)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Everything the sitemap advertises must exist: those URLs are built in Go,
|
||||
// away from the templates that normally escape ids.
|
||||
func TestInvariantAdvertisedURLsResolve(t *testing.T) {
|
||||
for _, src := range []string{"/sitemap.xml"} {
|
||||
body := get(t, src).Body.String()
|
||||
locs := regexp.MustCompile(`<loc>([^<]+)</loc>|<link[^>]+href="([^"]+)"`).FindAllStringSubmatch(body, -1)
|
||||
if len(locs) == 0 {
|
||||
t.Fatalf("%s advertises nothing", src)
|
||||
}
|
||||
seen := 0
|
||||
for _, m := range locs {
|
||||
raw := m[1]
|
||||
if raw == "" {
|
||||
raw = m[2]
|
||||
}
|
||||
u, err := url.Parse(html.UnescapeString(raw))
|
||||
if err != nil || u.Path == "" {
|
||||
continue
|
||||
}
|
||||
p := u.Path
|
||||
if u.RawQuery != "" {
|
||||
p += "?" + u.RawQuery
|
||||
}
|
||||
if w := get(t, p); w.Code != 200 {
|
||||
t.Errorf("%s advertises %s, which answers %d", src, p, w.Code)
|
||||
}
|
||||
seen++
|
||||
}
|
||||
t.Logf("%s: %d URLs resolve", src, seen)
|
||||
}
|
||||
}
|
||||
|
||||
// The API's pagination envelope has to describe the page it returned, at
|
||||
// every page size, and per_page=all is the export mode that skips snippets.
|
||||
func TestInvariantAPIPagination(t *testing.T) {
|
||||
type envelope struct {
|
||||
Total int `json:"total"`
|
||||
Page int `json:"page"`
|
||||
PerPage int `json:"per_page"`
|
||||
TotalPages int `json:"total_pages"`
|
||||
Omitted bool `json:"snippets_omitted"`
|
||||
Docs []struct {
|
||||
ID string `json:"id"`
|
||||
} `json:"docs"`
|
||||
Results []struct {
|
||||
ID string `json:"id"`
|
||||
Snippet string `json:"snippet"`
|
||||
} `json:"results"`
|
||||
}
|
||||
read := func(path string) envelope {
|
||||
var e envelope
|
||||
w := get(t, path)
|
||||
if w.Code != 200 {
|
||||
t.Fatalf("%s: status %d", path, w.Code)
|
||||
}
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &e); err != nil {
|
||||
t.Fatalf("%s: %v", path, err)
|
||||
}
|
||||
return e
|
||||
}
|
||||
|
||||
for _, path := range []string{"/api/docs", "/api/docs?per_page=1", "/api/docs?per_page=all",
|
||||
"/api/search?q=prism", "/api/search?q=prism&per_page=1", "/api/search?q=prism&per_page=all"} {
|
||||
e := read(path)
|
||||
n := len(e.Docs) + len(e.Results)
|
||||
switch {
|
||||
case e.Page < 1:
|
||||
t.Errorf("%s: page %d", path, e.Page)
|
||||
case e.PerPage < 1:
|
||||
t.Errorf("%s: per_page %d", path, e.PerPage)
|
||||
case n > e.PerPage:
|
||||
t.Errorf("%s: returned %d rows for a page of %d", path, n, e.PerPage)
|
||||
case e.TotalPages < 1:
|
||||
t.Errorf("%s: total_pages %d", path, e.TotalPages)
|
||||
}
|
||||
if want := (e.Total + e.PerPage - 1) / e.PerPage; e.Total > 0 && e.TotalPages != want {
|
||||
t.Errorf("%s: total_pages %d, want %d for %d/%d", path, e.TotalPages, want, e.Total, e.PerPage)
|
||||
}
|
||||
}
|
||||
|
||||
// An out-of-range page clamps rather than 500s or returns junk.
|
||||
if e := read("/api/docs?p=9999"); e.Page != e.TotalPages {
|
||||
t.Errorf("p=9999 gave page %d of %d", e.Page, e.TotalPages)
|
||||
}
|
||||
// Export mode returns everything and says why there are no snippets.
|
||||
all := read("/api/search?q=prism&per_page=all")
|
||||
if !all.Omitted || len(all.Results) != all.Total {
|
||||
t.Errorf("per_page=all: omitted=%v, %d of %d results", all.Omitted, len(all.Results), all.Total)
|
||||
}
|
||||
for _, r := range all.Results {
|
||||
if r.Snippet != "" {
|
||||
t.Errorf("per_page=all returned a snippet for %s", r.ID)
|
||||
}
|
||||
}
|
||||
// A normal page does carry them.
|
||||
paged := read("/api/search?q=prism")
|
||||
if paged.Omitted {
|
||||
t.Error("a paged search should not omit snippets")
|
||||
}
|
||||
}
|
||||
|
||||
// Snippet work is bounded by maxSnippetDocs, not by the page size, and the
|
||||
// page says so when the two differ.
|
||||
func TestInvariantSnippetCap(t *testing.T) {
|
||||
buildFixture(t)
|
||||
doc := corpus().docIndex["prism-slides"]
|
||||
if doc == nil {
|
||||
t.Fatal("fixture document missing")
|
||||
}
|
||||
docs := make([]*Document, maxSnippetDocs+5)
|
||||
for i := range docs {
|
||||
docs[i] = doc
|
||||
}
|
||||
page, capped := decorateResults(docs, []string{"collection"}, []string{"TE"}, []string{"and"})
|
||||
if !capped {
|
||||
t.Error("a page past the budget should report itself capped")
|
||||
}
|
||||
withSnippets := 0
|
||||
for i := range page {
|
||||
if len(page[i].Snippets) > 0 {
|
||||
withSnippets++
|
||||
}
|
||||
if page[i].Title == "" {
|
||||
t.Fatalf("row %d lost its title highlighting", i)
|
||||
}
|
||||
}
|
||||
if withSnippets != maxSnippetDocs {
|
||||
t.Errorf("%d rows got snippets, want %d", withSnippets, maxSnippetDocs)
|
||||
}
|
||||
}
|
||||
166
tests/loader_test.go
Normal file
166
tests/loader_test.go
Normal file
|
|
@ -0,0 +1,166 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// What the loader does with a corpus that is wrong in the ways corpora are
|
||||
// wrong in practice. Every case must leave the rest of the archive intact -
|
||||
// a broken document is skipped, never a reason to serve nothing - and say
|
||||
// so in the tally that -check reads.
|
||||
func TestLoaderRejectsBadDocuments(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
write := func(path, content string) {
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
good := func(id string) {
|
||||
write(filepath.Join(root, id, "metadata.json"), `{"title": "`+id+`", "type": "REPORTED"}`)
|
||||
write(filepath.Join(root, id, id+".pdf"), "%PDF-1.4")
|
||||
}
|
||||
good("keeper-one")
|
||||
good("keeper-two")
|
||||
|
||||
// No PDF: this is an archive OF the PDFs, so the document is skipped.
|
||||
write(filepath.Join(root, "no-pdf", "metadata.json"), `{"title": "No PDF"}`)
|
||||
// Invalid JSON: skipped, everything else still loads.
|
||||
write(filepath.Join(root, "bad-json", "metadata.json"), `{"title": "oops"`)
|
||||
write(filepath.Join(root, "bad-json", "x.pdf"), "%PDF-1.4")
|
||||
// aka_id that is not a list of objects: refused, not silently half-read.
|
||||
write(filepath.Join(root, "bad-aka", "metadata.json"), `{"title": "Aka", "aka_id": "other-id"}`)
|
||||
write(filepath.Join(root, "bad-aka", "x.pdf"), "%PDF-1.4")
|
||||
// A directory that is not a document at all: skipped without a warning.
|
||||
write(filepath.Join(root, "notes", "README.txt"), "just notes")
|
||||
// A directory holding a PDF and nothing else: the shape worth warning
|
||||
// about, since it looks like a document nobody described.
|
||||
write(filepath.Join(root, "loose", "orphan.pdf"), "%PDF-1.4")
|
||||
|
||||
withCorpus(t, root)
|
||||
clean := loadWarnings
|
||||
|
||||
// A document in the SIDtoday archive that is not typed SIDTODAY is on
|
||||
// no browse tab: served, and warned about.
|
||||
write(filepath.Join(root, "SIDtoday-archive", "mistyped", "metadata.json"), `{"title": "Mistyped", "type": "REPORTED"}`)
|
||||
write(filepath.Join(root, "SIDtoday-archive", "mistyped", "x.pdf"), "%PDF-1.4")
|
||||
loadData()
|
||||
if corpus().docIndex["mistyped"] == nil || loadWarnings != clean+1 {
|
||||
t.Errorf("a REPORTED document in SIDtoday-archive: loaded %v, %d warnings, want it loaded with %d", corpus().docIndex["mistyped"] != nil, loadWarnings, clean+1)
|
||||
}
|
||||
if err := os.RemoveAll(filepath.Join(root, "SIDtoday-archive")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
loadData()
|
||||
|
||||
if len(corpus().archive.Docs) != 2 {
|
||||
ids := make([]string, 0, len(corpus().archive.Docs))
|
||||
for _, d := range corpus().archive.Docs {
|
||||
ids = append(ids, d.ID)
|
||||
}
|
||||
t.Fatalf("loaded %v, want the two good documents", ids)
|
||||
}
|
||||
if loadErrors < 3 { // no-pdf, bad-aka and bad JSON are errors: not served
|
||||
t.Errorf("loadErrors = %d, want at least 3", loadErrors)
|
||||
}
|
||||
if loadWarnings < 1 { // the loose PDF directory: served nothing, lost nothing
|
||||
t.Errorf("loadWarnings = %d, want at least 1", loadWarnings)
|
||||
}
|
||||
for _, id := range []string{"no-pdf", "bad-json", "bad-aka", "notes", "loose"} {
|
||||
if corpus().docIndex[id] != nil {
|
||||
t.Errorf("%s should not have loaded", id)
|
||||
}
|
||||
}
|
||||
// The archive is coherent: facets, search and the sitemap all see the
|
||||
// same two documents.
|
||||
if got := corpus().archive.CollectionCounts[collectionNews]; got != 2 {
|
||||
t.Errorf("news collection count %d, want 2", got)
|
||||
}
|
||||
if got, _ := corpus().executeSearch(searchParams{Queries: []string{""}, Fields: []string{"TE"}, Conjs: []string{"and"}}); len(got) != 2 {
|
||||
t.Errorf("match-all search found %d, want 2", len(got))
|
||||
}
|
||||
}
|
||||
|
||||
// An empty corpus is a corpus: the server loads, every page renders, and
|
||||
// nothing divides by zero.
|
||||
func TestLoaderEmptyCorpus(t *testing.T) {
|
||||
withCorpus(t, t.TempDir())
|
||||
if len(corpus().archive.Docs) != 0 {
|
||||
t.Fatalf("loaded %d documents from an empty directory", len(corpus().archive.Docs))
|
||||
}
|
||||
for _, p := range []string{"/", "/browse", "/search?q=x&do_search=1", "/explore", "/explore/ids",
|
||||
"/articles", "/api/docs", "/api/facets", "/sitemap.xml"} {
|
||||
if w := get(t, p); w.Code != 200 {
|
||||
t.Errorf("%s on an empty corpus: %d", p, w.Code)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Values are tidied before anything reads them, and a copy the loader skips
|
||||
// leaves no trace - not even a claim on its ID.
|
||||
func TestLoaderTidiesBeforeUse(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
write := func(path, content string) {
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
// FOIA/ sorts before the flat root, so the PDF-less copy is seen first.
|
||||
write(filepath.Join(root, "FOIA", "dup-doc", "metadata.json"), `{"title": "Shadow"}`)
|
||||
write(filepath.Join(root, "dup-doc", "metadata.json"), `{"title": "Real", "type": "REPORTED"}`)
|
||||
write(filepath.Join(root, "dup-doc", "x.pdf"), "%PDF-1.4")
|
||||
|
||||
write(filepath.Join(root, "padded", "metadata.json"), `{
|
||||
"title": "Padded", "type": "REPORTED",
|
||||
"document": {"creator": " NSA ", "series": {"name": "Series X ", "part": 1}},
|
||||
"publication": [{"publisher": "Le Monde", "date_published": "June 2014", "version": "entier"}]
|
||||
}`)
|
||||
write(filepath.Join(root, "padded", "x.pdf"), "%PDF-1.4")
|
||||
write(filepath.Join(root, "padded", "versions", "entier", "entier.pdf"), "%PDF-1.4")
|
||||
|
||||
withCorpus(t, root)
|
||||
|
||||
if d := corpus().docIndex["dup-doc"]; d == nil || d.Title != "Real" {
|
||||
t.Errorf("dup-doc = %+v, want the copy with a PDF", d)
|
||||
}
|
||||
d := corpus().docIndex["padded"]
|
||||
if d == nil {
|
||||
t.Fatal("padded not loaded")
|
||||
}
|
||||
if d.DocMeta.Creator != "NSA" || d.DocMeta.Series.Name != "Series X" {
|
||||
t.Errorf("creator %q, series %q: want both trimmed", d.DocMeta.Creator, d.DocMeta.Series.Name)
|
||||
}
|
||||
// The malformed date is dropped everywhere, the version label included.
|
||||
if v := d.VersionFor("entier"); v == nil || v.Label != "Le Monde" {
|
||||
t.Errorf("version label %+v, want \"Le Monde\" without the invalid date", v)
|
||||
}
|
||||
}
|
||||
|
||||
// Two page images with one page number show that page twice; the loader
|
||||
// says so instead of linking changed pages to whichever sorted first.
|
||||
func TestLoadPageImagesWarnsOnDuplicatePageNumbers(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.MkdirAll(filepath.Join(dir, "images"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, name := range []string{"page-1.jpg", "page-2.jpg", "page-02.png"} {
|
||||
if err := os.WriteFile(filepath.Join(dir, "images", name), nil, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
before := loadWarnings
|
||||
imgs := loadPageImages(dir, "doc")
|
||||
if len(imgs) != 3 || imgs[0] != "page-1.jpg" {
|
||||
t.Errorf("images %v, want all three in page order", imgs)
|
||||
}
|
||||
if loadWarnings != before+1 {
|
||||
t.Errorf("%d warnings for one duplicated page number, want 1", loadWarnings-before)
|
||||
}
|
||||
}
|
||||
74
tests/main_test.go
Normal file
74
tests/main_test.go
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// TestFileWriteBudget: a download slower than the server's WriteTimeout still
|
||||
// completes, because handleFiles replaces that page-sized budget with
|
||||
// fileWriteBudget. The control - the same slow download with the budget cut
|
||||
// short - must be truncated, or the test would not be exercising the deadline
|
||||
// at all.
|
||||
func TestFileWriteBudget(t *testing.T) {
|
||||
const size = 64 << 20 // well past the loopback socket buffers
|
||||
path := filepath.Join(t.TempDir(), "big.pdf")
|
||||
if err := os.WriteFile(path, bytes.Repeat([]byte("snowchive"), size/9+1)[:size], 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
old := fileWriteBudget
|
||||
defer func() { fileWriteBudget = old }()
|
||||
|
||||
// The same two lines handleFiles uses, over a server whose page budget is
|
||||
// far shorter than the download takes.
|
||||
ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
_ = http.NewResponseController(w).SetWriteDeadline(time.Now().Add(fileWriteBudget))
|
||||
http.ServeFile(w, r, path)
|
||||
}))
|
||||
ts.Config.WriteTimeout = 300 * time.Millisecond
|
||||
ts.Start()
|
||||
defer ts.Close()
|
||||
|
||||
// fetch reads 1 MB at a time with a pause after each chunk, returning the
|
||||
// bytes received before the body ended.
|
||||
fetch := func(pause time.Duration) (int64, time.Duration) {
|
||||
start := time.Now()
|
||||
resp, err := http.Get(ts.URL)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
var got int64
|
||||
buf := make([]byte, 1<<20)
|
||||
for {
|
||||
n, err := io.ReadFull(resp.Body, buf)
|
||||
got += int64(n)
|
||||
if err != nil {
|
||||
return got, time.Since(start)
|
||||
}
|
||||
time.Sleep(pause)
|
||||
}
|
||||
}
|
||||
|
||||
fileWriteBudget = 30 * time.Second
|
||||
got, elapsed := fetch(40 * time.Millisecond)
|
||||
if got != size {
|
||||
t.Errorf("slow download got %d of %d bytes", got, size)
|
||||
}
|
||||
if elapsed < 2*ts.Config.WriteTimeout {
|
||||
t.Errorf("download took %v: too fast to prove it outlasted the %v WriteTimeout",
|
||||
elapsed, ts.Config.WriteTimeout)
|
||||
}
|
||||
|
||||
fileWriteBudget = 200 * time.Millisecond
|
||||
if got, _ := fetch(40 * time.Millisecond); got >= size {
|
||||
t.Errorf("control: a %v budget did not cut the download (got all %d bytes)",
|
||||
fileWriteBudget, got)
|
||||
}
|
||||
}
|
||||
145
tests/mdparser_test.go
Normal file
145
tests/mdparser_test.go
Normal file
|
|
@ -0,0 +1,145 @@
|
|||
package main
|
||||
|
||||
import "testing"
|
||||
|
||||
// Emphasis follows CommonMark's flanking rules, so the loose asterisks and
|
||||
// underscores of OCR - footnote marks, flattened bullets, star pages,
|
||||
// form blanks - stay text, and real emphasis nests properly.
|
||||
func TestParseInlineEmphasis(t *testing.T) {
|
||||
for in, want := range map[string]string{
|
||||
// Real emphasis.
|
||||
"*it* and **bold**": "<em>it</em> and <strong>bold</strong>",
|
||||
"***bold italic***": "<em><strong>bold italic</strong></em>",
|
||||
"**2** | **61**": "<strong>2</strong> | <strong>61</strong>",
|
||||
"An *n*-clique": "An <em>n</em>-clique",
|
||||
"*a **b** c*": "<em>a <strong>b</strong> c</em>",
|
||||
"**bold *nested* x**": "<strong>bold <em>nested</em> x</strong>",
|
||||
"*`code`*": "<em><code>code</code></em>",
|
||||
"_em_ and __init__.py": "<em>em</em> and <strong>init</strong>.py",
|
||||
`"On Language,"* a *New York Times*`: ""On Language,"* a <em>New York Times</em>",
|
||||
// A closer takes the nearest opener, not the first one on the line.
|
||||
"Inc.*, 321 F.3d; see also *North v. Ashcroft*, 308": "Inc.*, 321 F.3d; see also <em>North v. Ashcroft</em>, 308",
|
||||
// Not emphasis.
|
||||
"PINWALE (Sweet* and Sour* partitions)": "PINWALE (Sweet* and Sour* partitions)",
|
||||
"NRO*, CIA, SCEs* and field sites": "NRO*, CIA, SCEs* and field sites",
|
||||
"works with: * NCSC * TAO": "works with: * NCSC * TAO",
|
||||
"*** Currently Inactive ***": "*** Currently Inactive ***",
|
||||
"a Driving Force ******": "a Driving Force ******",
|
||||
"2013 WL 5741573, at *3-*9": "2013 WL 5741573, at *3-*9",
|
||||
"2 * 3 * 4": "2 * 3 * 4",
|
||||
"snake_case_name": "snake_case_name",
|
||||
"D/DIR____,IG_____,OGC___": "D/DIR____,IG_____,OGC___",
|
||||
"SO ORDERED, this _____ day of July": "SO ORDERED, this _____ day of July",
|
||||
"datastore<br>_ target user<br>_ time": "datastore<br>_ target user<br>_ time",
|
||||
`\*not em\*`: "*not em*",
|
||||
} {
|
||||
if got := parseInline(in); got != want {
|
||||
t.Errorf("parseInline(%q)\n got %q\nwant %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A list never mixes bullets and numbers: a number kept in a <ul> would
|
||||
// lose itself to a bullet, a bullet kept in an <ol> would gain a number.
|
||||
func TestRenderListKindChange(t *testing.T) {
|
||||
for in, want := range map[string]string{
|
||||
"1. Fact:\n* South Korea\n* Japan\n\n2. Existence": "<ol>\n<li>Fact:</li>\n</ol>\n" +
|
||||
"<ul>\n<li>South Korea</li>\n<li>Japan</li>\n</ul>\n" +
|
||||
"<ol>\n<li value=\"2\">Existence</li>\n</ol>\n",
|
||||
"* Insider claims\n1. ↑ one\n2. ↑ two": "<ul>\n<li>Insider claims</li>\n</ul>\n" +
|
||||
"<ol>\n<li>↑ one</li>\n<li>↑ two</li>\n</ol>\n",
|
||||
// Both nested lists stay inside the item they hang from.
|
||||
"- a\n - b\n 1. c": "<ul>\n<li>a\n<ul>\n<li>b</li>\n</ul>\n<ol>\n<li>c</li>\n</ol>\n</li>\n</ul>\n",
|
||||
// Unindented sub-bullets: "5." picks up where "4." left off.
|
||||
"4. Demonstrate:\n* one\n* two\n5. Thank": "<ol>\n<li value=\"4\">Demonstrate:</li>\n</ol>\n" +
|
||||
"<ul>\n<li>one</li>\n<li>two</li>\n</ul>\n" +
|
||||
"<ol>\n<li value=\"5\">Thank</li>\n</ol>\n",
|
||||
} {
|
||||
if got := Render(in); got != want {
|
||||
t.Errorf("Render(%q)\n got %q\nwant %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A number only starts a list right under a line of text when it is a 1
|
||||
// with content (CommonMark), so OCR runover - a year, a section number, a
|
||||
// page cite wrapped onto the next line - stays part of the sentence.
|
||||
func TestRenderRunoverNumbers(t *testing.T) {
|
||||
for in, want := range map[string]string{
|
||||
"as added by the FISA Amendments Act of\n2008.": "<p>as added by the FISA Amendments Act of<br>\n2008.</p>\n",
|
||||
"extended until August\n2013. He will be joined": "<p>extended until August<br>\n2013. He will be joined</p>\n",
|
||||
"* last visited NSA/CSS in May\n2012.": "<ul>\n<li>last visited NSA/CSS in May<br>\n2012.</li>\n</ul>\n",
|
||||
"See Alexander Decl. at 39-\n42. While terrorists": "<p>See Alexander Decl. at 39-<br>\n42. While terrorists</p>\n",
|
||||
"First Events\n+\nLast Events": "<p>First Events<br>\n+<br>\nLast Events</p>\n",
|
||||
// These still start a list.
|
||||
"Steps:\n1. one\n2. two": "<p>Steps:</p>\n<ol>\n<li>one</li>\n<li>two</li>\n</ol>\n",
|
||||
"Steps:\n- one": "<p>Steps:</p>\n<ul>\n<li>one</li>\n</ul>\n",
|
||||
"\n2008. after a blank": "<ol>\n<li value=\"2008\">after a blank</li>\n</ol>\n",
|
||||
"5. five\n6.\n7. seven": "<ol>\n<li value=\"5\">five</li>\n<li></li>\n<li>seven</li>\n</ol>\n",
|
||||
"1. one\n3. gap\n10. ten": "<ol>\n<li>one</li>\n<li value=\"3\">gap</li>\n<li value=\"10\">ten</li>\n</ol>\n",
|
||||
} {
|
||||
if got := Render(in); got != want {
|
||||
t.Errorf("Render(%q)\n got %q\nwant %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// "0." is a number like any other; it used to double as "not ordered" and
|
||||
// render as whatever the running count was.
|
||||
func TestRenderZeroNumber(t *testing.T) {
|
||||
for in, want := range map[string]string{
|
||||
"0. zero\n1. one": "<ol>\n<li value=\"0\">zero</li>\n<li>one</li>\n</ol>\n",
|
||||
"20. a\n10. b\n0.": "<ol>\n<li value=\"20\">a</li>\n<li value=\"10\">b</li>\n<li value=\"0\"></li>\n</ol>\n",
|
||||
} {
|
||||
if got := Render(in); got != want {
|
||||
t.Errorf("Render(%q)\n got %q\nwant %q", in, got, want)
|
||||
}
|
||||
}
|
||||
// Ten digits is text, not a list item.
|
||||
if got := Render("1234567890. x"); got != "<p>1234567890. x</p>\n" {
|
||||
t.Errorf("Render ten-digit number = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Thematic breaks may space out their characters, as the court filings'
|
||||
// asterisms do; such a line is a rule, never a bullet.
|
||||
func TestRenderSpacedRule(t *testing.T) {
|
||||
for _, in := range []string{"* * *", "- - - - -", "_ _ _", " * *\t* ", "***", "---"} {
|
||||
if got := Render(in); got != "<hr>\n" {
|
||||
t.Errorf("Render(%q) = %q, want a rule", in, got)
|
||||
}
|
||||
}
|
||||
if got := Render("1. a\n* * *\n2. b"); got != "<ol>\n<li>a</li>\n</ol>\n<hr>\n<ol>\n<li value=\"2\">b</li>\n</ol>\n" {
|
||||
t.Errorf("a spaced rule inside a list = %q", got)
|
||||
}
|
||||
for _, in := range []string{"* * x", "--", "- -", "*-*"} {
|
||||
if got := Render(in); got == "<hr>\n" {
|
||||
t.Errorf("Render(%q) is a rule", in)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Only the one outer pipe at each end of a row is dropped, so empty edge
|
||||
// cells keep their columns, and "\|" is a pipe inside a cell.
|
||||
func TestSplitCells(t *testing.T) {
|
||||
for in, want := range map[string][]string{
|
||||
"| a | b |": {"a", "b"},
|
||||
"||||| ECIs. |": {"", "", "", "", "ECIs."},
|
||||
"| B. (U) ASSOCIATIONS|||||": {"B. (U) ASSOCIATIONS", "", "", "", ""},
|
||||
"| | |": {"", ""},
|
||||
`| Service \| Rights | x |`: {"Service | Rights", "x"},
|
||||
`| a \|`: {"a |"},
|
||||
} {
|
||||
got := splitCells(in)
|
||||
if len(got) != len(want) {
|
||||
t.Errorf("splitCells(%q) = %q, want %q", in, got, want)
|
||||
continue
|
||||
}
|
||||
for i := range got {
|
||||
if got[i] != want[i] {
|
||||
t.Errorf("splitCells(%q) = %q, want %q", in, got, want)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
54
tests/outlets_test.go
Normal file
54
tests/outlets_test.go
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// A document counts for an outlet in the years of THAT outlet's entries: The
|
||||
// Guardian's 2013 publication must not put The Intercept's 2014 copy in
|
||||
// The Intercept's 2013 column, and the cell links a page listing exactly the
|
||||
// documents it counts.
|
||||
func TestOutletYearsArePerPublisher(t *testing.T) {
|
||||
root := assemblyCorpus(t, "")
|
||||
writeAssemblyDoc(t, root, "shared", map[string]interface{}{
|
||||
"title": "Shared",
|
||||
"publication": []map[string]interface{}{
|
||||
{"publisher": "The Guardian", "date_published": "2013-06-06"},
|
||||
{"publisher": "The Intercept", "date_published": "2014-02-10"},
|
||||
{"publisher": "Partner", "date_published": "2012-01-01", "reporting_only": true},
|
||||
},
|
||||
}, []string{"page-1.jpg"}, nil)
|
||||
withCorpus(t, root)
|
||||
|
||||
cells := map[string]outletCell{}
|
||||
table := corpus().outletTables[""]
|
||||
for _, row := range table.Rows {
|
||||
for i, y := range table.Years {
|
||||
cells[row.Name+" "+y] = row.Cells[i]
|
||||
}
|
||||
}
|
||||
for k, want := range map[string]int{
|
||||
"The Guardian 2013": 1, "The Guardian 2014": 0,
|
||||
"The Intercept 2013": 0, "The Intercept 2014": 1,
|
||||
} {
|
||||
if got := cells[k].Docs; got != want {
|
||||
t.Errorf("%s: %d documents, want %d", k, got, want)
|
||||
}
|
||||
}
|
||||
for _, y := range table.Years {
|
||||
if y == "2012" {
|
||||
t.Errorf("a reporting-only entry made a 2012 column")
|
||||
}
|
||||
}
|
||||
link := cells["The Intercept 2014"].URL
|
||||
if link != "/browse/publisher/The%20Intercept?year=2014&coll=news&coll=sidtoday-archive&coll=foia" {
|
||||
t.Errorf("cell link %q", link)
|
||||
}
|
||||
if got := resultTotal(t, link); got != 1 {
|
||||
t.Errorf("%s lists %d documents, want 1", link, got)
|
||||
}
|
||||
if body := get(t, "/browse/publisher/The%20Intercept?year=2013").Body.String(); !strings.Contains(body, "0 documents") {
|
||||
t.Errorf("The Intercept's 2013 page is not empty")
|
||||
}
|
||||
}
|
||||
119
tests/portions_test.go
Normal file
119
tests/portions_test.go
Normal file
|
|
@ -0,0 +1,119 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestParsePortion(t *testing.T) {
|
||||
cases := []struct {
|
||||
in string
|
||||
canonical string
|
||||
level string
|
||||
ok bool
|
||||
}{
|
||||
{"U", "U", "U", true},
|
||||
{"U//FOUO", "U//FOUO", "U", true},
|
||||
{"TS//SI//NF", "TS//SI//NF", "TS", true},
|
||||
{"S//REL TO USA, FVEY", "S//REL TO USA, FVEY", "S", true},
|
||||
// OCR writes the separator every way; one marking, one canonical form.
|
||||
{"S/NF", "S//NF", "S", true},
|
||||
{"U/FOUO", "U//FOUO", "U", true},
|
||||
{"TS/SI//REL", "TS//SI//REL", "TS", true},
|
||||
{"TS//SI/NF", "TS//SI//NF", "TS", true},
|
||||
{"TS//STLW//SI//OC/NF", "TS//STLW//SI//OC//NF", "TS", true},
|
||||
{"TS//SI// ECI SOL", "TS//SI//ECI SOL", "TS", true},
|
||||
// FOUO alone is a control on unclassified material.
|
||||
{"FOUO", "U//FOUO", "U", true},
|
||||
// Spelled out.
|
||||
{"TOP SECRET//COMINT", "TS//COMINT", "TS", true},
|
||||
// A level with a part no rule covers: recognised as a marking, but
|
||||
// not counted - the Explore page lists these separately.
|
||||
{"S//SI) (DNI collection", "", "S", false},
|
||||
{"S//WIBBLE", "S//WIBBLE", "S", false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
got := parsePortion(c.in)
|
||||
if got.Level != c.level {
|
||||
t.Errorf("parsePortion(%q).Level = %q, want %q", c.in, got.Level, c.level)
|
||||
}
|
||||
if got.OK != c.ok {
|
||||
t.Errorf("parsePortion(%q).OK = %v, want %v", c.in, got.OK, c.ok)
|
||||
}
|
||||
if c.ok && got.Canonical != c.canonical {
|
||||
t.Errorf("parsePortion(%q).Canonical = %q, want %q", c.in, got.Canonical, c.canonical)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The things that merely look like markings: list enumerators, FOIA
|
||||
// exemption citations, asides. Each of these appears hundreds of times.
|
||||
func TestParsePortionRejectsNonMarkings(t *testing.T) {
|
||||
for _, s := range []string{"1", "2", "a", "b", "iii", "d", "Cont'd.", "continued...",
|
||||
"Dollars in Thousands", "b)(1), (b)(3", "A", "", " "} {
|
||||
if got := parsePortion(s); got.Level != "" {
|
||||
t.Errorf("parsePortion(%q) read a level %q - it is not a marking", s, got.Level)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractPortions(t *testing.T) {
|
||||
text := "" +
|
||||
"TOP SECRET//COMINT\n" +
|
||||
"\n" +
|
||||
"(U) Some Tips on Portion Marking\n" +
|
||||
"\n" +
|
||||
"(U//FOUO) The lead-in sentence MUST ALWAYS be portion-marked.\n" +
|
||||
"* (S//SI) a bullet carrying a complete thought\n" +
|
||||
"* an unmarked bullet, covered by its lead-in\n" +
|
||||
"## (TS//SI//NF) A marked heading\n" +
|
||||
"1. (C) a numbered portion\n" +
|
||||
"(1) not a marking, a list number\n" +
|
||||
"text mentioning (U//FOUO) mid-line is not a portion mark\n"
|
||||
got := make([]string, 0, 5)
|
||||
for _, m := range extractPortions(text) {
|
||||
got = append(got, m.Canonical)
|
||||
}
|
||||
want := []string{"U", "U//FOUO", "S//SI", "TS//SI//NF", "C"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("extractPortions = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// A document quoting a marking is not marked by it. Fenced blocks and block
|
||||
// quotes are where a transcript puts an example, and they are the curator's
|
||||
// correction surface: wrap the example, and it stops being counted.
|
||||
func TestExtractPortionsSkipsQuotedExamples(t *testing.T) {
|
||||
text := "" +
|
||||
"(U) Wrong way:\n" +
|
||||
"\n" +
|
||||
"> (U) All good men and women should come to the aid of their country through:\n" +
|
||||
"> * (U//FOUO) volunteering for a tdy to the Middle East\n" +
|
||||
"\n" +
|
||||
"```\n" +
|
||||
"(TS//SI//NF) an example inside a fenced block\n" +
|
||||
"```\n" +
|
||||
"(S) a real portion after the block\n"
|
||||
var got []string
|
||||
for _, m := range extractPortions(text) {
|
||||
got = append(got, m.Canonical)
|
||||
}
|
||||
want := []string{"U", "S"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("extractPortions = %v, want %v (quoted and fenced examples must not count)", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// Markings are written in capitals. Court filings enumerate subsections
|
||||
// "(a)", "(b)", "(c)" at the start of a line, and "(c)" is not CONFIDENTIAL.
|
||||
func TestParsePortionNeedsCapitals(t *testing.T) {
|
||||
for _, s := range []string{"c", "s", "u", "ts", "s//si", "Ts", "secret"} {
|
||||
if got := parsePortion(s); got.Level != "" {
|
||||
t.Errorf("parsePortion(%q) read level %q from a lower-case label", s, got.Level)
|
||||
}
|
||||
}
|
||||
// The controls after a capital level are read whatever their case.
|
||||
if got := parsePortion("S//si"); got.Canonical != "S//SI" || !got.OK {
|
||||
t.Errorf("parsePortion(S//si) = %+v, want S//SI", got)
|
||||
}
|
||||
}
|
||||
153
tests/prefs_matrix_test.go
Normal file
153
tests/prefs_matrix_test.go
Normal file
|
|
@ -0,0 +1,153 @@
|
|||
package main
|
||||
|
||||
// prefs_matrix_test.go - every preference, asserted against the markup it
|
||||
// is supposed to change. The cookies are documented in prefs.go and their
|
||||
// effects are spread across four templates; before this, nothing checked
|
||||
// that setting one did anything at all.
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// prefCase is one preference: the cookie that turns it on, a page that
|
||||
// shows it, and a marker that must be present in exactly one of the two
|
||||
// states. present says which state the marker belongs to.
|
||||
type prefCase struct {
|
||||
name string
|
||||
cookie string
|
||||
value string
|
||||
path string
|
||||
marker string
|
||||
present bool // true: the marker appears when the cookie is SET
|
||||
}
|
||||
|
||||
func TestPreferenceMatrix(t *testing.T) {
|
||||
buildFixture(t)
|
||||
cases := []prefCase{
|
||||
// --- card display switches (browse lists, search results) -------
|
||||
{"hide thumbnails", "pref_hide_thumbs", "1", "/browse", `class="thumb-container"`, false},
|
||||
{"hide thumbnails (search)", "pref_hide_thumbs", "1", "/search?q=prism&do_search=1", `class="result-thumb"`, false},
|
||||
{"hide summaries", "pref_hide_summary", "1", "/browse", `class="card-summary"`, false},
|
||||
{"hide classification", "pref_hide_classification", "1", "/browse", `class="doc-classification"`, false},
|
||||
{"hide programs", "pref_hide_programs", "1", "/browse", `class="doc-programs"`, false},
|
||||
{"hide people", "pref_hide_people", "1", "/browse", `class="doc-people"`, false},
|
||||
{"hide journalists", "pref_hide_journalists", "1", "/browse", `class="doc-journalist`, false},
|
||||
{"page preview hook", "pref_page_preview", "1", "/browse", "data-pp-id=", true},
|
||||
|
||||
// --- category tables --------------------------------------------
|
||||
{"collection columns", "pref_show_coll_cols", "1", "/browse/program", `id="coll-cols" class="coll-cols-checkbox" checked`, true},
|
||||
|
||||
// --- the document page ------------------------------------------
|
||||
{"info sidebar", "pref_info_position", "beside", "/doc/prism-slides", `class="doc-layout info-sidebar"`, true},
|
||||
{"scroll panes", "pref_scroll_doc", "1", "/doc/prism-slides", "scroll-panes", true},
|
||||
{"expand more info", "pref_open_info", "1", "/doc/prism-slides", `id="meta-toggle" class="meta-checkbox" checked`, true},
|
||||
{"programs one per line", "pref_programs_list", "1", "/doc/prism-slides", `<ul class="meta-value-list">`, true},
|
||||
{"tags one per line", "pref_tags_list", "1", "/doc/prism-slides", `<ul class="meta-value-list">`, true},
|
||||
{"collapse published-together", "pref_collapse_pub", "1", "/doc/prism-slides", `<details class="collapse-section" open>`, false},
|
||||
|
||||
// --- the viewport picker ----------------------------------------
|
||||
{"default view: text", "pref_doc_view", "text", "/doc/prism-slides", `id="view-text" aria-controls="doc-viewport" checked`, true},
|
||||
{"default view: split", "pref_doc_view", "split", "/doc/prism-slides", `id="view-split" aria-controls="doc-viewport" checked`, true},
|
||||
|
||||
// --- theme -------------------------------------------------------
|
||||
{"dark theme", "pref_theme", "dark", "/", `data-theme="dark"`, true},
|
||||
{"light theme", "pref_theme", "light", "/browse", `data-theme="light"`, true},
|
||||
}
|
||||
|
||||
for _, c := range cases {
|
||||
off := get(t, c.path).Body.String()
|
||||
on := get(t, c.path, &http.Cookie{Name: c.cookie, Value: c.value}).Body.String()
|
||||
hasOff, hasOn := strings.Contains(off, c.marker), strings.Contains(on, c.marker)
|
||||
wantOn, wantOff := c.present, !c.present
|
||||
if hasOn != wantOn || hasOff != wantOff {
|
||||
t.Errorf("%s (%s=%s on %s): marker %q present without the cookie=%v, with it=%v; wanted %v / %v",
|
||||
c.name, c.cookie, c.value, c.path, c.marker, hasOff, hasOn, wantOff, wantOn)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Text-only mode is the archive's promise to readers on Tor and metered
|
||||
// links: NO page image, thumbnail or preview, anywhere, whatever else is
|
||||
// set. It also overrides the thumbnail preference rather than being
|
||||
// overridden by it.
|
||||
func TestPreferenceTextOnly(t *testing.T) {
|
||||
buildFixture(t)
|
||||
textOnly := &http.Cookie{Name: "pref_text_only", Value: "1"}
|
||||
thumbsOn := &http.Cookie{Name: "pref_hide_thumbs", Value: "0"}
|
||||
preview := &http.Cookie{Name: "pref_page_preview", Value: "1"}
|
||||
|
||||
for _, p := range []string{"/", "/browse", "/browse?collection=sidtoday-archive&view=card",
|
||||
"/doc/prism-slides", "/doc/prism-slides/pages", "/doc/prism-slides/versions/lemonde",
|
||||
"/search?q=prism&do_search=1", "/pinned", "/articles"} {
|
||||
body := get(t, p, textOnly, thumbsOn, preview).Body.String()
|
||||
if i := strings.Index(body, "<img"); i >= 0 {
|
||||
t.Errorf("%s in text-only mode emits an image: %.80s", p, body[i:])
|
||||
}
|
||||
if strings.Contains(body, "data-pp-imgs") {
|
||||
t.Errorf("%s in text-only mode emits the page-preview payload", p)
|
||||
}
|
||||
}
|
||||
// The document page says so and offers the pages as links instead.
|
||||
doc := get(t, "/doc/prism-slides", textOnly).Body.String()
|
||||
for _, want := range []string{"Text-only mode is on", `class="page-link-list"`} {
|
||||
if !strings.Contains(doc, want) {
|
||||
t.Errorf("the document page in text-only mode lacks %q", want)
|
||||
}
|
||||
}
|
||||
// Without it, that same page does show images.
|
||||
if !strings.Contains(get(t, "/doc/prism-slides").Body.String(), "<img") {
|
||||
t.Error("the document page shows no images even with images on")
|
||||
}
|
||||
}
|
||||
|
||||
// Documents per page drives the list length and the pagination, on browse
|
||||
// and on search alike.
|
||||
func TestPreferencePerPage(t *testing.T) {
|
||||
buildFixture(t)
|
||||
count := func(body string) int { return strings.Count(body, `class="doc-card"`) }
|
||||
all := count(get(t, "/browse", &http.Cookie{Name: "pref_per_page", Value: "all"}).Body.String())
|
||||
if all < 2 {
|
||||
t.Fatalf("the fixture's news collection has %d cards; the test needs at least 2", all)
|
||||
}
|
||||
one := get(t, "/browse", &http.Cookie{Name: "pref_per_page", Value: "10"}).Body.String()
|
||||
if n := count(one); n > 10 {
|
||||
t.Errorf("per_page=10 rendered %d cards", n)
|
||||
}
|
||||
// An unknown value falls back to the default rather than to everything.
|
||||
def := count(get(t, "/browse", &http.Cookie{Name: "pref_per_page", Value: "7"}).Body.String())
|
||||
if def > 20 {
|
||||
t.Errorf("an invalid per_page rendered %d cards, want at most the default 20", def)
|
||||
}
|
||||
}
|
||||
|
||||
// The two history preferences are opt-in: nothing is recorded until they
|
||||
// are on, and turning one off deletes what it recorded.
|
||||
func TestPreferenceHistoryOptIn(t *testing.T) {
|
||||
buildFixture(t)
|
||||
// Off by default: viewing a document sets no history cookie.
|
||||
w := get(t, "/doc/prism-slides")
|
||||
for _, c := range w.Result().Cookies() {
|
||||
if c.Name == recentDocsCookie && c.Value != "" {
|
||||
t.Errorf("a document view recorded history without the preference: %v", c)
|
||||
}
|
||||
}
|
||||
// On: the view is recorded.
|
||||
w = get(t, "/doc/prism-slides", &http.Cookie{Name: "pref_track_recent", Value: "1"})
|
||||
found := false
|
||||
for _, c := range w.Result().Cookies() {
|
||||
if c.Name == recentDocsCookie && strings.Contains(c.Value, "prism-slides") {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Error("with tracking on, a document view was not recorded")
|
||||
}
|
||||
// A page showing history must never be stored by a shared cache.
|
||||
hist := get(t, "/", &http.Cookie{Name: "pref_track_recent", Value: "1"},
|
||||
&http.Cookie{Name: recentDocsCookie, Value: "prism-slides"})
|
||||
if cc := hist.Header().Get("Cache-Control"); !strings.Contains(cc, "no-store") {
|
||||
t.Errorf("the home page showing history: Cache-Control %q", cc)
|
||||
}
|
||||
}
|
||||
117
tests/prefs_test.go
Normal file
117
tests/prefs_test.go
Normal file
|
|
@ -0,0 +1,117 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func reqWithCookies(pairs ...string) *http.Request {
|
||||
r := httptest.NewRequest("GET", "/", nil)
|
||||
for i := 0; i+1 < len(pairs); i += 2 {
|
||||
r.AddCookie(&http.Cookie{Name: pairs[i], Value: pairs[i+1]})
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// Defaults apply when a cookie is absent, and a value the preference does
|
||||
// not accept is a value it does not take.
|
||||
func TestReadPrefsValues(t *testing.T) {
|
||||
p := readPrefs(reqWithCookies())
|
||||
if p.DocView != "images" || p.Theme != "auto" || p.PerPage != 20 || p.InfoSidebar {
|
||||
t.Errorf("defaults wrong: %+v", p)
|
||||
}
|
||||
if !p.ShowThumbs || !p.ShowClassification || p.TrackRecent {
|
||||
t.Errorf("boolean defaults wrong: %+v", p)
|
||||
}
|
||||
|
||||
p = readPrefs(reqWithCookies(
|
||||
"pref_doc_view", "split",
|
||||
"pref_theme", "dark",
|
||||
"pref_per_page", "all",
|
||||
"pref_info_position", "beside",
|
||||
"pref_hide_thumbs", "1",
|
||||
"pref_track_recent", "1",
|
||||
))
|
||||
if p.DocView != "split" || p.Theme != "dark" || p.PerPage != 0 || !p.InfoSidebar {
|
||||
t.Errorf("cookie values not applied: %+v", p)
|
||||
}
|
||||
if p.ShowThumbs || !p.TrackRecent {
|
||||
t.Errorf("boolean cookies not applied: %+v", p)
|
||||
}
|
||||
if p.ThemeAttr() != "dark" {
|
||||
t.Errorf("ThemeAttr = %q", p.ThemeAttr())
|
||||
}
|
||||
|
||||
// Junk in the jar: every value pref falls back to its default.
|
||||
p = readPrefs(reqWithCookies(
|
||||
"pref_doc_view", "sideways",
|
||||
"pref_theme", "neon",
|
||||
"pref_per_page", "7",
|
||||
"pref_info_position", "under",
|
||||
))
|
||||
if p.DocView != "images" || p.Theme != "auto" || p.PerPage != 20 || p.InfoSidebar {
|
||||
t.Errorf("junk cookies not rejected: %+v", p)
|
||||
}
|
||||
if p.ThemeAttr() != "" {
|
||||
t.Errorf("auto must render no data-theme, got %q", p.ThemeAttr())
|
||||
}
|
||||
}
|
||||
|
||||
// Saving writes the cookies a non-default choice needs and clears the rest,
|
||||
// so a default-settings visitor stays cookieless (and their pages stay
|
||||
// publicly cacheable).
|
||||
func TestPreferencesSaveRoundTrip(t *testing.T) {
|
||||
form := url.Values{
|
||||
"doc_view": {"text"}, "theme": {"light"}, "per_page": {"100"},
|
||||
"info_position": {"beside"}, "hide_thumbs": {"1"},
|
||||
}
|
||||
r := httptest.NewRequest("POST", "/preferences/save", strings.NewReader(form.Encode()))
|
||||
r.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
r.Header.Set("Sec-Fetch-Site", "same-origin")
|
||||
w := httptest.NewRecorder()
|
||||
handlePreferencesSave(w, r)
|
||||
|
||||
set := map[string]string{}
|
||||
for _, c := range w.Result().Cookies() {
|
||||
set[c.Name] = c.Value
|
||||
if c.MaxAge < 0 {
|
||||
set[c.Name] = "" // deleted
|
||||
}
|
||||
}
|
||||
for name, want := range map[string]string{
|
||||
"pref_doc_view": "text", "pref_theme": "light", "pref_per_page": "100",
|
||||
"pref_info_position": "beside", "pref_hide_thumbs": "1",
|
||||
"pref_track_recent": "", // unchecked: deleted
|
||||
} {
|
||||
if got, ok := set[name]; !ok || got != want {
|
||||
t.Errorf("%s = %q (present %v), want %q", name, got, ok, want)
|
||||
}
|
||||
}
|
||||
|
||||
// Defaults delete rather than store: nothing in the jar means nothing
|
||||
// to vary the cache on.
|
||||
form = url.Values{"doc_view": {"images"}, "theme": {"auto"}, "per_page": {"20"}, "info_position": {"above"}}
|
||||
r = httptest.NewRequest("POST", "/preferences/save", strings.NewReader(form.Encode()))
|
||||
r.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
r.Header.Set("Sec-Fetch-Site", "same-origin")
|
||||
w = httptest.NewRecorder()
|
||||
handlePreferencesSave(w, r)
|
||||
for _, c := range w.Result().Cookies() {
|
||||
if c.MaxAge >= 0 {
|
||||
t.Errorf("default settings wrote cookie %s=%q", c.Name, c.Value)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// "All" per page is every card up to maxCardsPerPage: a list longer than
|
||||
// that is paged, however large the reader's page size.
|
||||
func TestCardsPerPage(t *testing.T) {
|
||||
for perPage, want := range map[int]int{0: maxCardsPerPage, 20: 20, 100: 100, maxCardsPerPage + 1: maxCardsPerPage} {
|
||||
if got := (Prefs{PerPage: perPage}).cardsPerPage(); got != want {
|
||||
t.Errorf("PerPage %d: %d cards a page, want %d", perPage, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
126
tests/probes_test.go
Normal file
126
tests/probes_test.go
Normal file
|
|
@ -0,0 +1,126 @@
|
|||
package main
|
||||
|
||||
// probes_test.go - the instruments that catch what reading the code cannot:
|
||||
// data races, the parameter space behind each route, and header injection.
|
||||
// Each of these exists because a review that only read the source missed
|
||||
// something a probe found in seconds.
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Concurrency: the archive claims to be read-only while serving. Hammer the
|
||||
// handlers from many goroutines with -race and see.
|
||||
func TestProbeConcurrent(t *testing.T) {
|
||||
if os.Getenv("SNOWCHIVE_BENCH_DATA") == "" {
|
||||
buildFixture(t)
|
||||
}
|
||||
paths := []string{"/", "/browse", "/browse/program", "/browse/tag?sort=count",
|
||||
"/search?q=prism&do_search=1", "/explore/ids", "/api/facets", "/glossary",
|
||||
"/doc/prism-slides", "/articles", "/sitemap.xml"}
|
||||
mux, _ := buildFixture(t)
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < 24; i++ {
|
||||
wg.Add(1)
|
||||
go func(n int) {
|
||||
defer wg.Done()
|
||||
for j := 0; j < 20; j++ {
|
||||
p := paths[(n+j)%len(paths)]
|
||||
r := httptest.NewRequest("GET", p, nil)
|
||||
if j%3 == 0 {
|
||||
r.AddCookie(&http.Cookie{Name: "pref_theme", Value: "dark"})
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
mux.ServeHTTP(w, r)
|
||||
if w.Code != 200 {
|
||||
t.Errorf("%s: %d", p, w.Code)
|
||||
return
|
||||
}
|
||||
}
|
||||
}(i)
|
||||
}
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
// Input space: every route crossed with the parameters it accepts. The
|
||||
// 80 ms page hid behind ?sort=count - a path-only sweep never saw it.
|
||||
|
||||
// Input space: every route crossed with the parameters it accepts. The
|
||||
// 80 ms page hid behind ?sort=count - a path-only sweep never saw it.
|
||||
func TestProbeParamSweep(t *testing.T) {
|
||||
buildFixture(t)
|
||||
base := []string{"/", "/browse", "/browse/program", "/browse/tag", "/browse/docdate",
|
||||
"/browse/program/PRISM", "/browse/docdate/2013", "/search", "/explore/ids",
|
||||
"/explore/terms", "/explore/pages", "/explore/outlets", "/articles",
|
||||
"/api/docs", "/api/search", "/glossary", "/pinned", "/preferences"}
|
||||
params := []string{"", "?sort=count", "?sort=count&order=asc", "?sort=zzz", "?order=desc",
|
||||
"?view=table", "?view=card", "?p=2", "?p=-1", "?p=99999999999999999999",
|
||||
"?per_page=all", "?per_page=0", "?per_page=-5", "?type=REPORTED", "?type=NOPE",
|
||||
"?coll=news", "?coll=news&coll=foia", "?missing=1", "?q=prism&do_search=1",
|
||||
"?q=%22a+b%22~5&do_search=1", "?q=-prism&do_search=1", "?hist=foia,sidtoday-archive",
|
||||
"?date_doc_start=2013&date_doc_end=2013", "?fqv=x&fqf=ZZ&fqc=xx&do_search=1",
|
||||
"?sort=name&order=desc&p=1&type=FOIA&coll=foia"}
|
||||
type slow struct {
|
||||
path string
|
||||
d time.Duration
|
||||
}
|
||||
var slows []slow
|
||||
bad := 0
|
||||
for _, b := range base {
|
||||
for _, p := range params {
|
||||
u := b + p
|
||||
start := time.Now()
|
||||
w := get(t, u)
|
||||
d := time.Since(start)
|
||||
if w.Code != 200 && w.Code != 400 && w.Code != 404 {
|
||||
bad++
|
||||
t.Errorf("%s: status %d", u, w.Code)
|
||||
}
|
||||
slows = append(slows, slow{u, d})
|
||||
}
|
||||
}
|
||||
sort.Slice(slows, func(i, j int) bool { return slows[i].d > slows[j].d })
|
||||
fmt.Printf("%d combinations, %d unexpected statuses; slowest on the fixture:\n", len(slows), bad)
|
||||
for _, s := range slows[:5] {
|
||||
fmt.Printf(" %8s %s\n", s.d.Round(time.Microsecond), s.path)
|
||||
}
|
||||
}
|
||||
|
||||
// Header injection: the Onion-Location header echoes the request URI.
|
||||
|
||||
// Header injection: the Onion-Location header echoes the request URI.
|
||||
func TestProbeHeaderInjection(t *testing.T) {
|
||||
buildFixture(t)
|
||||
for _, raw := range []string{
|
||||
"/browse?x=1%0d%0aX-Injected:+yes",
|
||||
"/browse?x=%0aSet-Cookie:+a=b",
|
||||
"/doc/" + strings.Repeat("a", 200),
|
||||
} {
|
||||
r := httptest.NewRequest("GET", raw, nil)
|
||||
w := httptest.NewRecorder()
|
||||
mux, _ := buildFixture(t)
|
||||
mux.ServeHTTP(w, r)
|
||||
h := w.Header()
|
||||
if h.Get("X-Injected") != "" || h.Get("Set-Cookie") != "" {
|
||||
t.Errorf("%s: header injected: %v", raw, h)
|
||||
}
|
||||
for k, vs := range h {
|
||||
for _, v := range vs {
|
||||
if strings.ContainsAny(v, "\r\n") {
|
||||
t.Errorf("%s: header %s carries a newline: %q", raw, k, v)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Pathological documents: the biggest thing in the corpus is what breaks
|
||||
// first. What IS the biggest thing?
|
||||
43
tests/published_together_test.go
Normal file
43
tests/published_together_test.go
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The "published in the same report" box is capped at pubTogetherCap, and
|
||||
// the overflow links point at the article pages: one per article that holds
|
||||
// other documents, in publication order, so each link's number is the
|
||||
// number its page lists.
|
||||
func TestDocArticles(t *testing.T) {
|
||||
saved := corpus().articleByLink
|
||||
defer func() { corpus().articleByLink = saved }()
|
||||
small := &Article{Key: "aaaaaaaaaaaa", DocIDs: []string{"a", "b"}}
|
||||
big := &Article{Key: "bbbbbbbbbbbb", DocIDs: []string{"a", "b", "c", "d"}}
|
||||
alone := &Article{Key: "cccccccccccc", DocIDs: []string{"a"}}
|
||||
corpus().articleByLink = map[string]*Article{
|
||||
"https://small.example/story": small,
|
||||
"https://big.example/appendix": big,
|
||||
"https://alone.example/x": alone,
|
||||
}
|
||||
var doc Document
|
||||
if err := json.Unmarshal([]byte(`{"publication":[
|
||||
{"link": "https://small.example/story"},
|
||||
{"link": "https://alone.example/x"},
|
||||
{"link": "https://big.example/appendix"},
|
||||
{"link": "https://small.example/story"}]}`), &doc); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := corpus().docArticles(&doc)
|
||||
if len(got) != 2 || got[0] != small || got[1] != big {
|
||||
t.Fatalf("docArticles = %+v, want the two articles with other documents, once each, in order", got)
|
||||
}
|
||||
|
||||
var orphan Document
|
||||
if err := json.Unmarshal([]byte(`{"publication":[{"link": "https://nowhere.example/x"}]}`), &orphan); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if a := corpus().docArticles(&orphan); a != nil {
|
||||
t.Error("a document whose links have no article must produce no overflow link")
|
||||
}
|
||||
}
|
||||
193
tests/query_test.go
Normal file
193
tests/query_test.go
Normal file
|
|
@ -0,0 +1,193 @@
|
|||
package main
|
||||
|
||||
// query_test.go - what the search engine promises, as queries and the
|
||||
// documents they must return. The unit tests cover the pieces (tokenizer,
|
||||
// wildcards, ranges); this covers the sentence a reader types.
|
||||
//
|
||||
// Results come from /api/search, which runs the same engine as the page, so
|
||||
// a refactor that changes what search FINDS fails here even when every page
|
||||
// still renders.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/url"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func searchIDs(t *testing.T, q string) []string {
|
||||
t.Helper()
|
||||
w := get(t, "/api/search?per_page=all&q="+url.QueryEscape(q))
|
||||
if w.Code != 200 {
|
||||
t.Fatalf("%q: status %d", q, w.Code)
|
||||
}
|
||||
var body struct {
|
||||
Results []struct {
|
||||
ID string `json:"id"`
|
||||
} `json:"results"`
|
||||
}
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil {
|
||||
t.Fatalf("%q: %v", q, err)
|
||||
}
|
||||
ids := make([]string, 0, len(body.Results))
|
||||
for _, r := range body.Results {
|
||||
ids = append(ids, r.ID)
|
||||
}
|
||||
sort.Strings(ids)
|
||||
return ids
|
||||
}
|
||||
|
||||
func TestQuerySuite(t *testing.T) {
|
||||
buildFixture(t)
|
||||
const (
|
||||
prism = "prism-slides"
|
||||
companion = "prism-companion"
|
||||
sid = "2003-05-29_SIDToday_-_Fixture_Issue"
|
||||
foia = "FISC_-_fixture_order"
|
||||
undated = "undated-fixture"
|
||||
)
|
||||
cases := []struct {
|
||||
query string
|
||||
want []string
|
||||
why string
|
||||
}{
|
||||
// --- words, stems and case --------------------------------------
|
||||
{"xkeyscore", []string{prism}, "a plain word in the transcript"},
|
||||
{"XKEYSCORE", []string{prism}, "matching ignores case"},
|
||||
{"collections", []string{prism, sid}, "stemming: plural finds the singular"},
|
||||
{"analysts", []string{prism, sid}, "plural finds singular"},
|
||||
// The stemmer deliberately leaves -tion alone (see stem), so
|
||||
// "collecting" and "collection" are different words here. This is
|
||||
// the behaviour, recorded so a change to the stemmer is a decision
|
||||
// rather than a surprise.
|
||||
{"collecting", []string{prism}, "-ing does not reach -tion"},
|
||||
{"zzzznotacorpusword", nil, "a word nobody wrote"},
|
||||
|
||||
// --- phrases and proximity ---------------------------------------
|
||||
{`"upstream collection"`, []string{prism}, "a quoted phrase is adjacency, and SIDtoday has the words apart"},
|
||||
{`"upstream collection"~12`, []string{prism, sid}, "proximity widens it to a window"},
|
||||
{`"cable tapping"`, []string{prism}, "a phrase that is also a tag"},
|
||||
|
||||
// --- punctuation and accents --------------------------------------
|
||||
{"RT-RG", []string{prism}, "a hyphenated term matches its parts in sequence"},
|
||||
{"muller", []string{prism}, "accent folding: muller finds Müller-Maguhn"},
|
||||
{"Müller-Maguhn", []string{prism}, "and the accented form finds itself"},
|
||||
{"sao paulo", []string{prism}, "folding works across a phrase too"},
|
||||
|
||||
// --- wildcards -----------------------------------------------------
|
||||
{"*keyscore", []string{prism}, "a leading wildcard"},
|
||||
{"tapp*", []string{prism, sid}, "a trailing wildcard reaches tapping"},
|
||||
{"*ollectio*", []string{prism, sid}, "both ends open"},
|
||||
|
||||
// --- exact values vs substrings -------------------------------------
|
||||
{`classification:="TOP SECRET"`, []string{prism}, "an exact value is the whole value"},
|
||||
{`classification:="SECRET"`, []string{foia}, "so SECRET does not match TOP SECRET"},
|
||||
{"classification:secret", []string{foia, prism}, "unquoted, it is a word match on both"},
|
||||
|
||||
// --- fielded search ---------------------------------------------------
|
||||
{"program:PRISM", []string{companion, prism}, "a metadata field"},
|
||||
{"agency:FISC", []string{foia}, "creator"},
|
||||
{"agency:NSA", []string{companion, prism, sid}, "creator across collections"},
|
||||
{`journalist:"Greenwald, Glenn"`, []string{companion, prism}, "a quoted byline"},
|
||||
{"publisher:NRK", nil, "a reporting-only partner is not a publisher"},
|
||||
{"publisher:\"The Guardian\"", []string{companion, prism}, "the outlet that published it"},
|
||||
{"tag:collection", []string{prism}, "tags are their own field"},
|
||||
{"country:Brazil", []string{prism}, "countries"},
|
||||
{"people:Snowden", []string{prism}, "people, by surname"},
|
||||
{"type:FOIA", []string{foia}, "document type"},
|
||||
{"series:\"(U) A Day in the Life of...\"", []string{sid}, "a series name with punctuation"},
|
||||
|
||||
// --- boolean and negation ---------------------------------------------
|
||||
{"prism -companion", []string{prism}, "inline negation excludes"},
|
||||
{"upstream -xkeyscore", []string{sid}, "negation against a word in the other document"},
|
||||
{"prism -x", []string{companion, prism}, "a single letter is a literal word, not an empty clause that excludes everything"},
|
||||
{"u.s.", nil, "a dotted abbreviation is matched as written, not dropped"},
|
||||
|
||||
// --- a phrase stays inside one value ------------------------------------
|
||||
{`"united states"`, []string{prism}, "a phrase inside one country"},
|
||||
{`"andy united"`, nil, "a phrase cannot run from one metadata value into the next"},
|
||||
{`"slides slides describing"`, nil, "nor from the title into the description"},
|
||||
{"program:PRISM tag:collection", []string{prism}, "two fields are an AND"},
|
||||
|
||||
// --- dates -------------------------------------------------------------
|
||||
{"date:2013", []string{companion, foia, prism}, "document year, across collections"},
|
||||
{"date:2013-04-02", []string{companion}, "an exact document date"},
|
||||
{"date:2003..2013", []string{companion, foia, prism, sid}, "a range"},
|
||||
{"published:2016", []string{sid}, "publication year, not document year"},
|
||||
|
||||
// --- the corner the archive cares about ---------------------------------
|
||||
{"fairview", []string{prism}, "text that exists only in another published copy"},
|
||||
{"guardian", []string{companion, prism}, "an outlet name is searchable as text"},
|
||||
{"prism-slides", []string{prism}, "the document id itself"},
|
||||
{"prism-slides-guardian", []string{prism}, "and the name the same file was published under"},
|
||||
}
|
||||
|
||||
for _, c := range cases {
|
||||
got := searchIDs(t, c.query)
|
||||
want := append([]string(nil), c.want...)
|
||||
sort.Strings(want)
|
||||
if strings.Join(got, ",") != strings.Join(want, ",") {
|
||||
t.Errorf("%-34s got %v, want %v (%s)", c.query, got, want, c.why)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Relevance is not just membership: a title hit must outrank a body hit,
|
||||
// and the phrase bonus must beat loose words.
|
||||
func TestQueryRanking(t *testing.T) {
|
||||
buildFixture(t)
|
||||
first := func(q string) string {
|
||||
ids := func() []string {
|
||||
w := get(t, "/api/search?q="+url.QueryEscape(q))
|
||||
var body struct {
|
||||
Results []struct {
|
||||
ID string `json:"id"`
|
||||
} `json:"results"`
|
||||
}
|
||||
json.Unmarshal(w.Body.Bytes(), &body)
|
||||
out := make([]string, 0, len(body.Results))
|
||||
for _, r := range body.Results {
|
||||
out = append(out, r.ID)
|
||||
}
|
||||
return out
|
||||
}()
|
||||
if len(ids) == 0 {
|
||||
t.Fatalf("%q returned nothing", q)
|
||||
}
|
||||
return ids[0]
|
||||
}
|
||||
if got := first("collection"); got != "prism-slides" {
|
||||
t.Errorf("collection: first result %s, want the document with it in the title and the tags", got)
|
||||
}
|
||||
if got := first("fixture"); got == "" {
|
||||
t.Error("fixture: no results")
|
||||
}
|
||||
}
|
||||
|
||||
// TestQueryRowParts: a structured row is read as ?q= reads its text - each
|
||||
// quoted phrase is a part of its own and the bare words are one more, all
|
||||
// ANDed. A row with two phrases used to be one substring search for the
|
||||
// text between the outer quotes, quotes included, and found nothing.
|
||||
func TestQueryRowParts(t *testing.T) {
|
||||
buildFixture(t)
|
||||
for _, c := range []struct {
|
||||
row string
|
||||
want int
|
||||
}{
|
||||
{`"upstream collection" "cable tapping"`, 1},
|
||||
{`"upstream collection" xkeyscore`, 1},
|
||||
{`"collection upstream" xkeyscore`, 0},
|
||||
} {
|
||||
w := get(t, "/api/search?per_page=all&fqf=TE&fqc=and&fqv="+url.QueryEscape(c.row))
|
||||
var body struct {
|
||||
Total int `json:"total"`
|
||||
}
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil {
|
||||
t.Fatalf("%s: %v", c.row, err)
|
||||
}
|
||||
if body.Total != c.want {
|
||||
t.Errorf("row %s: %d results, want %d", c.row, body.Total, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
77
tests/related_test.go
Normal file
77
tests/related_test.go
Normal file
|
|
@ -0,0 +1,77 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The regression this guards: a SIDtoday piece called "The Life of an Exec"
|
||||
// used to list five FISC court filings as related, because their titles say
|
||||
// "of" eight times and every occurrence scored like a shared codename. A
|
||||
// shared word is worth only as much as it is rare, and it counts once.
|
||||
func TestRelatedIgnoresFillerTitleWords(t *testing.T) {
|
||||
filler := strings.Repeat("Motion of the Court for Disclosure of Records of the Court of Appeal ", 3)
|
||||
docs := []Document{
|
||||
{ID: "exec", Title: "The Life of an Exec (repost)", Collection: "sidtoday-archive"},
|
||||
{ID: "sibling", Title: "A Day in the Life of an Exec", Collection: "sidtoday-archive"},
|
||||
{ID: "filing", Title: filler, Collection: "foia"},
|
||||
}
|
||||
// Enough other titles saying "of" and "court" that those words are
|
||||
// common - which is exactly what the real corpus looks like.
|
||||
for i := 0; i < 30; i++ {
|
||||
docs = append(docs, Document{
|
||||
ID: fmt.Sprintf("noise-%02d", i),
|
||||
Title: fmt.Sprintf("Order of the Court of Review number %d", i),
|
||||
Collection: "foia",
|
||||
})
|
||||
}
|
||||
|
||||
got := corpus().buildRelatedDocs(docs[:1], docs)["exec"]
|
||||
if len(got) == 0 || got[0] != "sibling" {
|
||||
t.Fatalf("related = %v, want sibling first", got)
|
||||
}
|
||||
for _, id := range got {
|
||||
if id == "filing" || strings.HasPrefix(id, "noise-") {
|
||||
t.Errorf("related = %v, want no documents matched only on filler words", got)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Shared metadata counts each distinct value once: a tag listed twice in one
|
||||
// document is one tag, not two matches.
|
||||
func TestRelatedScoreCountsEachValueOnce(t *testing.T) {
|
||||
a := Document{ID: "a", Title: "Alpha", Tags: []string{"PRISM", "prism", "SSO"}}
|
||||
b := Document{ID: "b", Title: "Beta", Tags: []string{"PRISM", "PRISM"}}
|
||||
if got := corpus().relatedScore(&a, &b, titleWeights{}); got != 2.0 {
|
||||
t.Errorf("score = %v, want 2 (one shared tag)", got)
|
||||
}
|
||||
|
||||
// A shared program, tag, agency and country each still count.
|
||||
a.DocMeta.Programs, a.DocMeta.Creator, a.DocMeta.Countries = []string{"XKEYSCORE"}, "NSA", []string{"Norway"}
|
||||
b.DocMeta.Programs, b.DocMeta.Creator, b.DocMeta.Countries = []string{"xkeyscore"}, "nsa", []string{"NORWAY", "Norway"}
|
||||
if got := corpus().relatedScore(&a, &b, titleWeights{}); got != 2.0+4.0+1.0+0.5 {
|
||||
t.Errorf("score = %v, want 7.5", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A word in most titles is worth almost nothing; one in a couple of titles is
|
||||
// worth nearly full marks.
|
||||
func TestTitleWeights(t *testing.T) {
|
||||
tw := titleWeights{n: 1000, df: map[string]int{"of": 800, "exec": 2}}
|
||||
if w := tw.weight("of"); w > 0.05 {
|
||||
t.Errorf("weight(of) = %v, want near 0", w)
|
||||
}
|
||||
if w := tw.weight("exec"); w < 0.8 {
|
||||
t.Errorf("weight(exec) = %v, want near 1", w)
|
||||
}
|
||||
// An unknown word is treated as unique, and a corpus too small to have
|
||||
// frequencies weighs everything equally.
|
||||
if w := tw.weight("never-seen"); w < 0.99 {
|
||||
t.Errorf("weight(unseen) = %v, want ~1", w)
|
||||
}
|
||||
if w := (titleWeights{}).weight("anything"); w != 1 {
|
||||
t.Errorf("empty weights = %v, want 1", w)
|
||||
}
|
||||
}
|
||||
49
tests/reporting_only_test.go
Normal file
49
tests/reporting_only_test.go
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// A reporting-only partner keeps its entry, but is not a source of the file:
|
||||
// publisher: must not match it and the source category must not list it.
|
||||
func TestReportingOnlyIsNotASource(t *testing.T) {
|
||||
var d Document
|
||||
if err := json.Unmarshal([]byte(`{"publication": [
|
||||
{"publisher": "The Intercept", "title": "Norway Used NSA Technology",
|
||||
"date_published": "2018-03-01", "reporters": ["Gallagher, Ryan"]},
|
||||
{"publisher": "NRK", "title": "Antennene som samler inn data",
|
||||
"date_published": "2018-03-01", "reporters": ["Hopland, Standahl"],
|
||||
"reporting_only": true}
|
||||
]}`), &d); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if got := fmt.Sprint(facetValues(&d, "PU")); got != "[The Intercept]" {
|
||||
t.Errorf("publisher values %s, want only [The Intercept]", got)
|
||||
}
|
||||
// The /browse/publisher/{outlet} index is built from the same values, so
|
||||
// the page and the facet agree: the publisher is a source, the partner
|
||||
// that only reported the story is not. The index is package state, so
|
||||
// put the corpus back for whatever runs next.
|
||||
prev := corpus().archive.Docs
|
||||
t.Cleanup(func() { corpus().archive.Docs = prev; corpus().buildCategoryDocs() })
|
||||
corpus().archive.Docs = []Document{d}
|
||||
corpus().buildCategoryDocs()
|
||||
if got := len(corpus().categoryMatches("publisher", "The Intercept")); got != 1 {
|
||||
t.Errorf("the publishing outlet's page lists %d documents, want 1", got)
|
||||
}
|
||||
if got := len(corpus().categoryMatches("publisher", "NRK")); got != 0 {
|
||||
t.Errorf("the reporting-only partner's page lists %d documents, want 0", got)
|
||||
}
|
||||
// The entry itself, its byline and its article survive.
|
||||
if len(d.Publication) != 2 || d.Publication[1].Publisher != "NRK" ||
|
||||
len(d.Publication[1].Journalists) != 1 {
|
||||
t.Errorf("the reporting-only entry lost data: %+v", d.Publication)
|
||||
}
|
||||
if !strings.Contains(fmt.Sprint(d.JournalistsDeduped()), "Hopland") {
|
||||
t.Error("a partner's reporters should still be credited")
|
||||
}
|
||||
}
|
||||
705
tests/routes_test.go
Normal file
705
tests/routes_test.go
Normal file
|
|
@ -0,0 +1,705 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"html/template"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Every route, through the real mux: the status it must answer, and a
|
||||
// fragment of the body that proves the page is the one it should be. The
|
||||
// handlers had no coverage at all before this - a missing template
|
||||
// function, a nil map or a renamed data key reached production as a 500.
|
||||
func TestRoutes(t *testing.T) {
|
||||
cases := []struct {
|
||||
path string
|
||||
code int
|
||||
want string // substring of the body
|
||||
}{
|
||||
{"/", 200, "Snowden"},
|
||||
{"/browse", 200, "Reported"},
|
||||
{"/browse?collection=foia", 200, "FOIA"},
|
||||
{"/browse?collection=sidtoday-archive", 200, "SIDtoday Archive"},
|
||||
{"/browse?collection=sidtoday-archive&view=card", 200, "SIDtoday Archive"},
|
||||
{"/browse?collection=sidtoday-archive&view=table", 200, "SIDtoday Archive"},
|
||||
{"/browse?collection=bogus", 200, "Reported"}, // unknown collection falls back
|
||||
{"/browse/program", 200, "PRISM"},
|
||||
{"/browse/program/PRISM", 200, "PRISM Collection Slides"},
|
||||
{"/browse/program/NOSUCHTHING", 200, "0 document"},
|
||||
{"/browse/tag/PRISM", 200, "PRISM Collection Slides"},
|
||||
{"/browse/agency/NSA", 200, "PRISM Collection Slides"},
|
||||
{"/browse/publisher/The%20Guardian", 200, "PRISM Collection Slides"},
|
||||
{"/browse/publisher/NRK", 200, "0 document"}, // reporting-only: not a publisher
|
||||
{"/browse/journalist/Greenwald%2C%20Glenn", 200, "PRISM"},
|
||||
{"/browse/classification/TOP%20SECRET", 200, "PRISM"},
|
||||
{"/browse/dissemination/NOFORN", 200, "PRISM"},
|
||||
{"/browse/docdate/2013", 200, "PRISM"},
|
||||
{"/browse/docdate/2013-04", 200, "PRISM"},
|
||||
{"/browse/docdate/20", 200, "0 document"}, // not a valid date prefix
|
||||
{"/browse/published/2013", 200, "PRISM"},
|
||||
{"/browse/series", 200, "A Day in the Life"},
|
||||
{"/browse/nosuchcategory", 404, ""},
|
||||
{"/browse/nosuchcategory/value", 404, ""},
|
||||
{"/doc/prism-slides", 200, "PRISM Collection Slides"},
|
||||
{"/doc/prism-slides/pages", 200, "All pages"},
|
||||
{"/doc/prism-slides/versions/lemonde", 200, "lemonde"},
|
||||
{"/doc/prism-slides/versions/lemonde/pages", 200, "lemonde"},
|
||||
{"/doc/prism-slides/versions/nope", 404, ""},
|
||||
{"/doc/broken-no-pdf", 404, ""}, // skipped at load: no PDF
|
||||
{"/doc/nope", 404, ""},
|
||||
{"/search", 200, "Search"},
|
||||
{"/search?q=prism&do_search=1", 200, "Collection Slides"}, // the term itself is <mark>ed
|
||||
{"/search?q=zzzznomatch&do_search=1", 200, "No matches"},
|
||||
{"/search?q=fairview&do_search=1", 200, "Copy:"}, // matched in a version
|
||||
{"/articles", 200, "NSA Prism program"},
|
||||
{"/explore", 200, "Explore"},
|
||||
{"/explore/pages", 200, "Longest"},
|
||||
{"/explore/ids", 200, "prism-slides"},
|
||||
{"/explore/ids.txt", 200, "prism-slides.pdf"},
|
||||
{"/explore/ids?missing=1", 200, "Technical IDs"},
|
||||
{"/explore/outlets", 200, "Guardian"},
|
||||
{"/explore/terms", 200, "PRISM"},
|
||||
{"/explore/classification", 200, "Classification"},
|
||||
{"/explore/nosuchmetric", 404, ""},
|
||||
{"/glossary", 200, "XKEYSCORE"},
|
||||
{"/changelog", 200, ""},
|
||||
{"/about", 200, "About"},
|
||||
{"/preferences", 200, "Preferences"},
|
||||
{"/pinned", 200, "Pinned"},
|
||||
{"/pinned/export", 200, ""},
|
||||
{"/random", 302, ""},
|
||||
{"/robots.txt", 200, "Sitemap"},
|
||||
{"/sitemap.xml", 200, "/doc/prism-slides"},
|
||||
{"/feed.xml", 404, ""},
|
||||
{"/api/docs", 200, `"total"`},
|
||||
{"/api/docs?per_page=all", 200, `"docs"`},
|
||||
{"/api/doc/prism-slides", 200, `"full_text"`},
|
||||
{"/api/doc/nope", 404, `"error"`},
|
||||
{"/api/search?q=prism", 200, `"results"`},
|
||||
{"/api/search", 400, `"error"`},
|
||||
{"/api/facets", 200, `"programs"`},
|
||||
{"/api/nosuch", 404, `"error"`},
|
||||
{"/static/style.css", 200, "--"},
|
||||
{"/static/nosuch.css", 404, ""},
|
||||
{"/files/prism-slides/prism-slides.pdf", 200, "%PDF"},
|
||||
{"/nosuchpage", 404, ""},
|
||||
}
|
||||
|
||||
for _, c := range cases {
|
||||
w := get(t, c.path)
|
||||
if w.Code != c.code {
|
||||
t.Errorf("%s: status %d, want %d", c.path, w.Code, c.code)
|
||||
continue
|
||||
}
|
||||
if c.want != "" && !strings.Contains(w.Body.String(), c.want) {
|
||||
t.Errorf("%s: body does not contain %q", c.path, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The document file handler is the one place a URL reaches the filesystem.
|
||||
func TestFileHandlerContainment(t *testing.T) {
|
||||
for _, path := range []string{
|
||||
"/files/prism-slides/images/page-1.jpg",
|
||||
"/files/prism-slides/versions/lemonde/lemonde.pdf",
|
||||
"/files/prism-slides/versions/lemonde/images/page-1.jpg",
|
||||
} {
|
||||
if w := get(t, path); w.Code != 200 {
|
||||
t.Fatalf("%s: a file of the document should serve, got %d", path, w.Code)
|
||||
}
|
||||
}
|
||||
// A file the loader never indexed is not served, whatever its type: a
|
||||
// document directory gathers backups and second copies.
|
||||
_, root := buildFixture(t)
|
||||
dir := filepath.Join(root, "prism-slides")
|
||||
for _, stray := range []string{"zz-stray.pdf", "images/notes/scan.jpg", "old-versions/draft.pdf"} {
|
||||
writeFixtureFile(t, filepath.Join(dir, filepath.FromSlash(stray)), "%PDF-1.4\n")
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
for _, stray := range []string{"zz-stray.pdf", "images/notes", "old-versions"} {
|
||||
os.RemoveAll(filepath.Join(dir, stray))
|
||||
}
|
||||
})
|
||||
for _, path := range []string{
|
||||
"/files/prism-slides/zz-stray.pdf", // a second PDF
|
||||
"/files/prism-slides/images/notes/scan.jpg", // not a page image
|
||||
"/files/prism-slides/old-versions/draft.pdf",
|
||||
"/files/prism-slides/versions/nosuch/x.pdf",
|
||||
"/files/prism-slides/thumb.jpg", // the fixture has no thumbnail
|
||||
"/files/prism-slides/../../../etc/passwd",
|
||||
"/files/prism-slides/..%2f..%2fetc%2fpasswd",
|
||||
"/files/prism-slides/versions/lemonde/metadata.json", // a version has none
|
||||
"/files/prism-slides/glossary.json",
|
||||
"/files/prism-slides/", // no file
|
||||
"/files/prism-slides/nope.pdf", // inside the doc, missing
|
||||
"/files/nosuchdoc/x.pdf", // unknown id
|
||||
"/files/prism-slides/versions/lemonde/", // directory
|
||||
} {
|
||||
if w := get(t, path); w.Code == 200 {
|
||||
t.Errorf("%s: served 200, want a refusal", path)
|
||||
}
|
||||
}
|
||||
// The curated metadata is a served file like the PDF it describes.
|
||||
if m := get(t, "/files/prism-slides/metadata.json"); m.Code != 200 || !strings.HasPrefix(m.Header().Get("Content-Type"), "application/json") {
|
||||
t.Errorf("metadata.json: status %d, Content-Type %q, want 200 application/json", m.Code, m.Header().Get("Content-Type"))
|
||||
}
|
||||
// A .md transcript is served as plain text, never as HTML.
|
||||
w := get(t, "/files/prism-slides/prism-slides.md")
|
||||
if w.Code != 200 {
|
||||
t.Fatalf("transcript status %d", w.Code)
|
||||
}
|
||||
if ct := w.Header().Get("Content-Type"); !strings.HasPrefix(ct, "text/plain") {
|
||||
t.Errorf("transcript Content-Type %q, want text/plain", ct)
|
||||
}
|
||||
}
|
||||
|
||||
// Cacheability is decided by one rule - a request carrying any cookie is
|
||||
// private - and personal pages go further with no-store. The hosting story
|
||||
// (a shared microcache in front) rests on this.
|
||||
func TestCacheHeaders(t *testing.T) {
|
||||
anon := get(t, "/browse")
|
||||
if cc := anon.Header().Get("Cache-Control"); cc != "public, max-age=10" {
|
||||
t.Errorf("cookieless page: Cache-Control %q", cc)
|
||||
}
|
||||
if v := anon.Header().Get("Vary"); v != "Cookie" {
|
||||
t.Errorf("cookieless page: Vary %q, want Cookie", v)
|
||||
}
|
||||
|
||||
cookied := get(t, "/browse", &http.Cookie{Name: "pref_theme", Value: "dark"})
|
||||
if cc := cookied.Header().Get("Cache-Control"); cc != "private, no-cache" {
|
||||
t.Errorf("cookied page: Cache-Control %q", cc)
|
||||
}
|
||||
|
||||
for _, p := range []string{"/preferences", "/pinned", "/pinned/export"} {
|
||||
if cc := get(t, p).Header().Get("Cache-Control"); !strings.Contains(cc, "no-store") {
|
||||
t.Errorf("%s: Cache-Control %q, want no-store", p, cc)
|
||||
}
|
||||
}
|
||||
|
||||
if cc := get(t, "/files/prism-slides/prism-slides.pdf").Header().Get("Cache-Control"); cc != "public, max-age=86400" {
|
||||
t.Errorf("document file: Cache-Control %q", cc)
|
||||
}
|
||||
// A 404 must never inherit the immutable static cache.
|
||||
if cc := get(t, "/static/nosuch.css?v=abc123").Header().Get("Cache-Control"); strings.Contains(cc, "immutable") {
|
||||
t.Errorf("missing static asset marked immutable: %q", cc)
|
||||
}
|
||||
}
|
||||
|
||||
// Every response carries the baseline security headers, whatever it is.
|
||||
func TestSecurityHeaders(t *testing.T) {
|
||||
for _, p := range []string{"/", "/doc/prism-slides", "/api/facets", "/nosuchpage", "/files/prism-slides/prism-slides.pdf"} {
|
||||
h := get(t, p).Header()
|
||||
for _, want := range []string{"Content-Security-Policy", "X-Frame-Options", "X-Content-Type-Options", "Referrer-Policy"} {
|
||||
if h.Get(want) == "" {
|
||||
t.Errorf("%s: missing %s", p, want)
|
||||
}
|
||||
}
|
||||
if csp := h.Get("Content-Security-Policy"); !strings.Contains(csp, "default-src 'self'") {
|
||||
t.Errorf("%s: CSP %q", p, csp)
|
||||
}
|
||||
}
|
||||
// The method gate's 405 too: scanner traffic is what it answers.
|
||||
mux, _ := buildFixture(t)
|
||||
w := record(mux, httptest.NewRequest("DELETE", "/", nil))
|
||||
if w.Code != http.StatusMethodNotAllowed || w.Header().Get("Content-Security-Policy") == "" {
|
||||
t.Errorf("DELETE: status %d, CSP %q; want 405 with the headers", w.Code, w.Header().Get("Content-Security-Policy"))
|
||||
}
|
||||
}
|
||||
|
||||
// State-changing POSTs are cookie-backed with no login, so a cross-site
|
||||
// submission must be refused rather than silently flipping preferences.
|
||||
func TestCrossSitePOSTRejected(t *testing.T) {
|
||||
mux, _ := buildFixture(t)
|
||||
for _, p := range []string{"/preferences/save", "/pin/prism-slides", "/history/clear", "/pinsearch"} {
|
||||
// same-site is a sibling subdomain: another origin, not this site.
|
||||
for _, site := range []string{"cross-site", "same-site"} {
|
||||
r := newPost(p)
|
||||
r.Header.Set("Sec-Fetch-Site", site)
|
||||
if w := record(mux, r); w.Code != http.StatusForbidden {
|
||||
t.Errorf("%s from %s: status %d, want 403", p, site, w.Code)
|
||||
}
|
||||
}
|
||||
r := newPost(p)
|
||||
r.Header.Set("Sec-Fetch-Site", "same-origin")
|
||||
if w := record(mux, r); w.Code == http.StatusForbidden {
|
||||
t.Errorf("%s from this site: rejected", p)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Loading is idempotent: every build* starts from empty, so a second load
|
||||
// (a test with another corpus, a future reload) produces the same archive
|
||||
// rather than a doubled one. Several builders used to append to package
|
||||
// slices that nothing reset.
|
||||
func TestLoadDataIsIdempotent(t *testing.T) {
|
||||
buildFixture(t)
|
||||
type snapshot struct {
|
||||
docs, timeline, codewords, groups, titled, glossary, sitemap, articles int
|
||||
}
|
||||
take := func() snapshot {
|
||||
years := 0
|
||||
for _, y := range corpus().archiveTimeline {
|
||||
years += len(y.Months)
|
||||
}
|
||||
return snapshot{
|
||||
docs: len(corpus().archive.Docs), timeline: years, codewords: len(corpus().codewords),
|
||||
groups: len(corpus().articleGroups), titled: corpus().articlesTitled,
|
||||
glossary: len(corpus().glossaryIndex) + len(corpus().glossaryGroups), sitemap: len(corpus().sitemapPaths), articles: len(corpus().articles),
|
||||
}
|
||||
}
|
||||
before := take()
|
||||
loadData()
|
||||
if after := take(); after != before {
|
||||
t.Errorf("a second load changed the archive:\n before %+v\n after %+v", before, after)
|
||||
}
|
||||
}
|
||||
|
||||
// robots.txt keeps crawlers off the routes that cost a corpus scan, and the
|
||||
// sitemap never advertises a path robots.txt disallows.
|
||||
func TestRobotsAndSitemapAgree(t *testing.T) {
|
||||
robots := get(t, "/robots.txt").Body.String()
|
||||
for _, want := range []string{"Disallow: /search", "Disallow: /api/", "Sitemap: "} {
|
||||
if !strings.Contains(robots, want) {
|
||||
t.Errorf("robots.txt lacks %q:\n%s", want, robots)
|
||||
}
|
||||
}
|
||||
var disallowed []string
|
||||
for _, line := range strings.Split(robots, "\n") {
|
||||
if v := strings.TrimPrefix(line, "Disallow:"); v != line {
|
||||
disallowed = append(disallowed, strings.TrimSpace(v))
|
||||
}
|
||||
}
|
||||
sitemap := get(t, "/sitemap.xml").Body.String()
|
||||
for _, m := range regexp.MustCompile(`<loc>([^<]+)</loc>`).FindAllStringSubmatch(sitemap, -1) {
|
||||
path := m[1]
|
||||
if i := strings.Index(path[len("https://"):], "/"); i >= 0 {
|
||||
path = path[len("https://")+i:]
|
||||
}
|
||||
for _, d := range disallowed {
|
||||
if d != "" && strings.HasPrefix(path, d) {
|
||||
t.Errorf("sitemap advertises %s, which robots.txt disallows (%s)", path, d)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// With a canonical base configured, /sitemap.xml is marshaled once at load
|
||||
// rather than per request - and still says the same thing.
|
||||
func TestCanonicalSitemapPrebuilt(t *testing.T) {
|
||||
buildFixture(t)
|
||||
prev := canonicalBase
|
||||
t.Cleanup(func() {
|
||||
canonicalBase = prev
|
||||
corpus().canonicalSitemap = nil
|
||||
corpus().buildCanonicalSitemap()
|
||||
})
|
||||
canonicalBase = "https://archive.example.org"
|
||||
corpus().buildCanonicalSitemap()
|
||||
if corpus().canonicalSitemap == nil {
|
||||
t.Fatal("no sitemap was built for the canonical base")
|
||||
}
|
||||
body := get(t, "/sitemap.xml").Body.String()
|
||||
if !strings.Contains(body, "<loc>https://archive.example.org/doc/prism-slides</loc>") {
|
||||
t.Errorf("sitemap does not use the canonical base:\n%s", body[:200])
|
||||
}
|
||||
if body != string(corpus().canonicalSitemap) {
|
||||
t.Error("the served sitemap is not the one built at load")
|
||||
}
|
||||
}
|
||||
|
||||
// Without a canonical base the sitemap is written from parts cut at load,
|
||||
// with the request's base between them: the document marshalSitemap gives
|
||||
// for that base, to the byte - a base that needs escaping included.
|
||||
func TestSitemapFromParts(t *testing.T) {
|
||||
h, _ := buildFixture(t)
|
||||
if canonicalBase != "" || corpus().sitemapParts == nil {
|
||||
t.Fatalf("the fixture should serve its sitemap from parts (base %q, %d parts)", canonicalBase, len(corpus().sitemapParts))
|
||||
}
|
||||
for _, host := range []string{"archive.example.org", "odd&host<x>.example"} {
|
||||
r := httptest.NewRequest("GET", "/sitemap.xml", nil)
|
||||
r.Host = host
|
||||
want, err := corpus().marshalSitemap(requestBase(r))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := record(h, r).Body.String(); got != string(want) {
|
||||
t.Errorf("host %q: the sitemap written from parts differs from marshalSitemap", host)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The big visitor-independent page bodies are executed once at startup
|
||||
// (prerenderPages). The page must look exactly the same whether it used
|
||||
// that copy or rendered its rows inline - which is what dev mode does, and
|
||||
// what every one of these pages falls back to if a block ever fails to
|
||||
// pre-render.
|
||||
func TestPrerenderedMatchesLive(t *testing.T) {
|
||||
buildFixture(t)
|
||||
if len(corpus().prerendered) == 0 {
|
||||
t.Fatal("nothing was pre-rendered")
|
||||
}
|
||||
pages := []string{
|
||||
"/explore/ids",
|
||||
"/explore/ids?missing=1",
|
||||
"/glossary",
|
||||
"/browse?collection=sidtoday-archive",
|
||||
"/browse/program",
|
||||
"/browse/tag",
|
||||
// Category orderings are ASSEMBLED from pre-rendered rows rather
|
||||
// than pre-rendered whole, so each one has to match too.
|
||||
"/browse/tag?sort=count",
|
||||
"/browse/program?sort=count&order=asc",
|
||||
"/browse/agency?sort=name&order=desc",
|
||||
"/browse/people?sort=foia",
|
||||
// So are the doc tables, out of pre-rendered document rows.
|
||||
"/browse?view=table",
|
||||
"/browse?collection=sidtoday-archive&view=table&sort=title&order=desc",
|
||||
"/browse/program/PRISM?view=table",
|
||||
}
|
||||
was, wasRows, wasAZ, wasDocRows := corpus().prerendered, corpus().categoryRowHTML, corpus().azRowHTML, corpus().docRowHTML
|
||||
t.Cleanup(func() {
|
||||
corpus().prerendered, corpus().categoryRowHTML, corpus().azRowHTML, corpus().docRowHTML = was, wasRows, wasAZ, wasDocRows
|
||||
})
|
||||
|
||||
live := make([]string, len(pages))
|
||||
corpus().prerendered = map[string]template.HTML{} // force the inline path
|
||||
corpus().categoryRowHTML, corpus().azRowHTML, corpus().docRowHTML = nil, nil, nil
|
||||
for i, p := range pages {
|
||||
live[i] = get(t, p).Body.String()
|
||||
}
|
||||
corpus().prerendered, corpus().categoryRowHTML, corpus().azRowHTML, corpus().docRowHTML = was, wasRows, wasAZ, wasDocRows
|
||||
for i, p := range pages {
|
||||
if got := get(t, p).Body.String(); got != live[i] {
|
||||
t.Errorf("%s: pre-rendered output differs from the live one", p)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The table view's headings are its sorters, and the one marked active is
|
||||
// the order the rows are actually in - the page's default when the URL
|
||||
// names none (the SIDtoday tab sorts by date, not title).
|
||||
func TestDocTableActiveHeading(t *testing.T) {
|
||||
for path, want := range map[string]string{
|
||||
"/browse?collection=sidtoday-archive&view=table": "Doc Date ▲",
|
||||
"/browse?view=table": "Title ▲",
|
||||
"/browse?view=table&sort=country&order=desc": "Countries ▼",
|
||||
"/browse?view=table&sort=bogus": "Title ▲",
|
||||
} {
|
||||
body := get(t, path).Body.String()
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("%s: no active heading %q", path, want)
|
||||
}
|
||||
if n := strings.Count(body, "▲") + strings.Count(body, "▼"); n != 1 {
|
||||
t.Errorf("%s: %d active headings, want 1", path, n)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A reporter credited by two co-publishing outlets is one name in the
|
||||
// table's Journalists column, as on the cards.
|
||||
func TestDocTableJournalistsDeduped(t *testing.T) {
|
||||
buildFixture(t)
|
||||
d := testDoc(t, `{"id": "x", "publication": [
|
||||
{"publisher": "A", "reporters": ["Greenwald, Glenn"]},
|
||||
{"publisher": "B", "reporters": ["Greenwald, Glenn", "Poitras, Laura"]}]}`)
|
||||
var b strings.Builder
|
||||
err := tmplSet["list.html"].ExecuteTemplate(&b, "doc-table", listView{Docs: []*Document{&d}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n := strings.Count(b.String(), "/browse/journalist/Greenwald"); n != 1 {
|
||||
t.Errorf("Greenwald linked %d times, want once", n)
|
||||
}
|
||||
if !strings.Contains(b.String(), "/browse/journalist/Poitras") {
|
||||
t.Error("Poitras missing")
|
||||
}
|
||||
}
|
||||
|
||||
// Without fetch metadata the Origin decides, and it omits a default port
|
||||
// the Host header may carry.
|
||||
func TestOriginIsHost(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
origin, host string
|
||||
want bool
|
||||
}{
|
||||
{"https://example.org", "example.org", true},
|
||||
{"https://example.org", "example.org:443", true},
|
||||
{"https://EXAMPLE.org", "example.org", true},
|
||||
{"http://127.0.0.1:8080", "127.0.0.1:8080", true},
|
||||
{"http://[::1]:8080", "[::1]:8080", true},
|
||||
{"https://example.org", "example.org:8443", false},
|
||||
{"https://example.org:8443", "example.org", false},
|
||||
{"http://example.org", "example.org:443", false},
|
||||
{"https://evil.example", "example.org", false},
|
||||
} {
|
||||
u, err := url.Parse(c.origin)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := originIsHost(u, c.host); got != c.want {
|
||||
t.Errorf("originIsHost(%s, %s) = %v, want %v", c.origin, c.host, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnvBool(t *testing.T) {
|
||||
for v, want := range map[string]bool{"": false, "1": true, "true": true, "TRUE": true, "0": false, "false": false} {
|
||||
t.Setenv("SNOWCHIVE_TEST_BOOL", v)
|
||||
if got := envBool("SNOWCHIVE_TEST_BOOL"); got != want {
|
||||
t.Errorf("envBool(%q) = %v, want %v", v, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestBrowseValueSpelling: a value matches ignoring case, so another
|
||||
// spelling of it redirects to the one the lists link instead of serving a
|
||||
// second page for the same documents; the query string goes along.
|
||||
func TestBrowseValueSpelling(t *testing.T) {
|
||||
w := get(t, "/browse/program/prism?view=table")
|
||||
if w.Code != http.StatusMovedPermanently || w.Header().Get("Location") != "/browse/program/PRISM?view=table" {
|
||||
t.Errorf("/browse/program/prism: %d to %q, want 301 to /browse/program/PRISM?view=table", w.Code, w.Header().Get("Location"))
|
||||
}
|
||||
if w := get(t, "/browse/program/NOSUCHTHING"); w.Code != 200 {
|
||||
t.Errorf("an unknown value: %d, want the empty page", w.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCardHighlightNeedsTheCategory: the cards on a value's page mark the
|
||||
// links to that value - not every link that happens to read the same, such
|
||||
// as the document date 2013 on the publication-year page for 2013.
|
||||
func TestCardHighlightNeedsTheCategory(t *testing.T) {
|
||||
if body := get(t, "/browse/published/2013").Body.String(); strings.Contains(body, "doc-created active") {
|
||||
t.Error("/browse/published/2013 marks the document-date links")
|
||||
}
|
||||
if body := get(t, "/browse/docdate/2013").Body.String(); !strings.Contains(body, "doc-created active") {
|
||||
t.Error("/browse/docdate/2013 does not mark the document-date links")
|
||||
}
|
||||
if body := get(t, "/browse/program/PRISM").Body.String(); !strings.Contains(body, `class="active">PRISM</a>`) {
|
||||
t.Error("/browse/program/PRISM does not mark the program links")
|
||||
}
|
||||
}
|
||||
|
||||
// TestUnknownSortIsNotEchoed: a sort key no list has is the default sort,
|
||||
// and stays out of the page's links and labels.
|
||||
func TestUnknownSortIsNotEchoed(t *testing.T) {
|
||||
for _, path := range []string{"/browse?sort=bogus", "/browse/program/PRISM?sort=bogus", "/search?q=prism&do_search=1&sort=bogus"} {
|
||||
body := get(t, path).Body.String()
|
||||
if strings.Contains(body, "sort=bogus") || strings.Contains(body, "Bogus") {
|
||||
t.Errorf("%s: the unknown sort is echoed", path)
|
||||
}
|
||||
}
|
||||
if body := get(t, "/search?q=prism&do_search=1").Body.String(); strings.Contains(body, "sort=relevance&order=desc") {
|
||||
t.Error("relevance offers a direction it does not have")
|
||||
}
|
||||
}
|
||||
|
||||
// TestOutletYearLinksKeepTheScope: on the reported-only outlets table the
|
||||
// year headings open the reported-only year, like the rows and cells do.
|
||||
func TestOutletYearLinksKeepTheScope(t *testing.T) {
|
||||
body := get(t, "/explore/outlets?coll=news").Body.String()
|
||||
if !strings.Contains(body, `href="/browse/published/2013?coll=news"`) {
|
||||
t.Error("the 2013 heading drops coll=news")
|
||||
}
|
||||
}
|
||||
|
||||
// pinSearch posts qs to /pinsearch from this site, with the given pinned
|
||||
// searches already in the cookie.
|
||||
func pinSearch(t *testing.T, qs string, pinned []string) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
mux, _ := buildFixture(t)
|
||||
r := httptest.NewRequest("POST", "/pinsearch", strings.NewReader(url.Values{"qs": {qs}, "back": {"/search"}}.Encode()))
|
||||
r.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
r.Header.Set("Sec-Fetch-Site", "same-origin")
|
||||
if len(pinned) > 0 {
|
||||
r.AddCookie(&http.Cookie{Name: pinnedSearchesCookie, Value: strings.Join(pinned, "|")})
|
||||
}
|
||||
return record(mux, r)
|
||||
}
|
||||
|
||||
// TestPinListFull: a pin the cookie has no room for is refused, and the
|
||||
// reader lands on /pinned, which says so, rather than back on a page whose
|
||||
// button still reads "Pin" as though nothing happened.
|
||||
func TestPinListFull(t *testing.T) {
|
||||
qs := url.Values{"do_search": {"1"}, "fqc": {"and"}, "fqf": {"TE"}, "fqv": {"prism"}}.Encode()
|
||||
if w := pinSearch(t, qs, nil); w.Code != http.StatusSeeOther || w.Header().Get("Location") != "/search" {
|
||||
t.Fatalf("pinning into an empty list: %d to %q", w.Code, w.Header().Get("Location"))
|
||||
}
|
||||
// Filler searches up to the byte budget: stored entries are not
|
||||
// checked on read, so any escaped query string stands in for one.
|
||||
var fill []string
|
||||
for i := 0; ; i++ {
|
||||
e := url.QueryEscape(fmt.Sprintf("fqv=filler%02d", i) + strings.Repeat("x", 180))
|
||||
if len(strings.Join(append(fill, e), "|")) > cookieValueMax {
|
||||
break
|
||||
}
|
||||
fill = append(fill, e)
|
||||
}
|
||||
// The last filler takes the cookie to exactly its budget.
|
||||
fill[len(fill)-1] += strings.Repeat("x", cookieValueMax-len(strings.Join(fill, "|")))
|
||||
w := pinSearch(t, qs, fill)
|
||||
if w.Code != http.StatusSeeOther || w.Header().Get("Location") != "/pinned?full=1" {
|
||||
t.Errorf("pinning into a full list: %d to %q, want 303 to /pinned?full=1", w.Code, w.Header().Get("Location"))
|
||||
}
|
||||
if body := get(t, "/pinned?full=1").Body.String(); !strings.Contains(body, "Your pinned list is full") {
|
||||
t.Error("/pinned?full=1 does not say the list is full")
|
||||
}
|
||||
}
|
||||
|
||||
// TestSavableSearchMatchesPinSearch: the search page offers the pin button
|
||||
// for exactly the searches /pinsearch accepts - one size limit for both.
|
||||
func TestSavableSearchMatchesPinSearch(t *testing.T) {
|
||||
long := strings.TrimSpace(strings.Repeat("abcdefgh ", 58)) // 58 words, under the word cap
|
||||
q := url.Values{"do_search": {"1"}, "fqc": {"and"}, "fqf": {"TE"}, "fqv": {long}}
|
||||
if len(q.Encode()) <= searchQSMax {
|
||||
t.Fatalf("test query is %d bytes, not over the %d limit", len(q.Encode()), searchQSMax)
|
||||
}
|
||||
if body := get(t, "/search?"+q.Encode()).Body.String(); strings.Contains(body, `action="/pinsearch"`) {
|
||||
t.Error("the search page offers to pin a search /pinsearch refuses")
|
||||
}
|
||||
if w := pinSearch(t, q.Encode(), nil); w.Code != http.StatusBadRequest {
|
||||
t.Errorf("/pinsearch with an over-long search: %d, want 400", w.Code)
|
||||
}
|
||||
short := url.Values{"do_search": {"1"}, "fqc": {"and"}, "fqf": {"TE"}, "fqv": {"prism"}}
|
||||
if body := get(t, "/search?"+short.Encode()).Body.String(); !strings.Contains(body, `action="/pinsearch"`) {
|
||||
t.Error("the search page does not offer to pin a short search")
|
||||
}
|
||||
}
|
||||
|
||||
// TestAssetStamp: pages link each asset by its content hash; that URL is
|
||||
// cached for a year, and any other stamp only revalidates.
|
||||
func TestAssetStamp(t *testing.T) {
|
||||
body := get(t, "/").Body.String()
|
||||
want := "/static/style.css?v=" + staticHashes["style.css"]
|
||||
if !strings.Contains(body, `href="`+want+`"`) {
|
||||
t.Fatalf("home page does not link %s", want)
|
||||
}
|
||||
if cc := get(t, want).Header().Get("Cache-Control"); !strings.Contains(cc, "immutable") {
|
||||
t.Errorf("%s: Cache-Control %q, want immutable", want, cc)
|
||||
}
|
||||
w := get(t, "/static/style.css?v=0000000000000000")
|
||||
if cc := w.Header().Get("Cache-Control"); strings.Contains(cc, "immutable") || w.Header().Get("ETag") == "" {
|
||||
t.Errorf("a stale stamp: Cache-Control %q, ETag %q; want revalidation", cc, w.Header().Get("ETag"))
|
||||
}
|
||||
}
|
||||
|
||||
// TestSitemapIgnoresFile: the sitemap is always the one built from the
|
||||
// loaded corpus - a sitemap.xml lying in the data or project directory is
|
||||
// never served, with -base or without it.
|
||||
func TestSitemapIgnoresFile(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
writeFixtureFile(t, filepath.Join(root, "sitemap.xml"), xmlHeaderForTest+"<urlset><url><loc>https://stale.example/</loc></url></urlset>\n")
|
||||
withCorpus(t, root)
|
||||
prev := canonicalBase
|
||||
t.Cleanup(func() { canonicalBase = prev })
|
||||
for _, base := range []string{"", "https://archive.example.org"} {
|
||||
canonicalBase = base
|
||||
loadData()
|
||||
if body := get(t, "/sitemap.xml").Body.String(); strings.Contains(body, "stale.example") {
|
||||
t.Errorf("base %q: a sitemap.xml file was served", base)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const xmlHeaderForTest = `<?xml version="1.0" encoding="UTF-8"?>` + "\n"
|
||||
|
||||
// TestPinnedSearchListed: /pinned lists a pinned search with its label,
|
||||
// its link and the unpin form carrying the canonical query string.
|
||||
func TestPinnedSearchListed(t *testing.T) {
|
||||
qs := url.Values{"do_search": {"1"}, "fqc": {"and"}, "fqf": {"TE"}, "fqv": {"prism"}}.Encode()
|
||||
body := get(t, "/pinned", &http.Cookie{Name: pinnedSearchesCookie, Value: url.QueryEscape(qs)}).Body.String()
|
||||
for _, want := range []string{`href="/search?` + template.HTMLEscapeString(qs) + `"`, `name="qs" value="` + template.HTMLEscapeString(qs) + `"`, ">prism<"} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("/pinned lacks %s", want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestPageTitles: every page names itself in the tab, not just the site.
|
||||
func TestPageTitles(t *testing.T) {
|
||||
buildFixture(t) // the article key comes from the loaded corpus
|
||||
for path, want := range map[string]string{
|
||||
"/browse/program/PRISM": "<title>Program: PRISM - Snowden Archive</title>",
|
||||
"/browse/program": "<title>Browse by Programs - Snowden Archive</title>",
|
||||
"/search?q=prism&do_search=1": "<title>prism - Search - Snowden Archive</title>",
|
||||
"/search": "<title>Search - Snowden Archive</title>",
|
||||
"/browse?collection=foia": "<title>FOIA documents - Snowden Archive</title>",
|
||||
"/pinned": "<title>Pinned - Snowden Archive</title>",
|
||||
"/nosuchpage": "<title>Page not found - Snowden Archive</title>",
|
||||
"/explore/pages": "- Snowden Archive</title>",
|
||||
"/article/" + corpus().articleByLink["https://example.org/prism"].Key: "<title>NSA Prism program - Snowden Archive</title>",
|
||||
} {
|
||||
if body := get(t, path).Body.String(); !strings.Contains(body, want) {
|
||||
t.Errorf("%s: no %q", path, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestNoBlockContentInHeadings: a heading holds text and inline controls
|
||||
// only - a form or a disclosure inside one is invalid HTML and lands in
|
||||
// the heading's accessible name. Forms tie their buttons from outside.
|
||||
func TestNoBlockContentInHeadings(t *testing.T) {
|
||||
heading := regexp.MustCompile(`(?s)<h([1-6])\b[^>]*>(.*?)</h[1-6]>`)
|
||||
block := regexp.MustCompile(`<(form|details|div|p|ul|ol|table)\b`)
|
||||
cookie := &http.Cookie{Name: pinnedCookie, Value: "prism-slides"}
|
||||
recent := []*http.Cookie{{Name: "pref_track_recent", Value: "1"}, {Name: recentDocsCookie, Value: "prism-slides"}}
|
||||
pages := map[string][]*http.Cookie{
|
||||
"/pinned": {cookie},
|
||||
"/": recent,
|
||||
"/search?q=prism&do_search=1": nil,
|
||||
"/doc/prism-slides": nil,
|
||||
}
|
||||
for path, cookies := range pages {
|
||||
body := get(t, path, cookies...).Body.String()
|
||||
for _, m := range heading.FindAllStringSubmatch(body, -1) {
|
||||
if b := block.FindString(m[2]); b != "" {
|
||||
t.Errorf("%s: %s inside <h%s>", path, b, m[1])
|
||||
}
|
||||
}
|
||||
}
|
||||
if body := get(t, "/pinned", cookie).Body.String(); !strings.Contains(body, `form="unpin-prism-slides"`) || !strings.Contains(body, `id="unpin-prism-slides"`) {
|
||||
t.Error("/pinned: the unpin button is not tied to its form")
|
||||
}
|
||||
}
|
||||
|
||||
// TestGlossarySeeAlsoResolves: a see-also name links to the entry it
|
||||
// names - through an alias or another case too - and a name no entry has
|
||||
// is plain text rather than a link to nowhere.
|
||||
func TestGlossarySeeAlsoResolves(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
writeFixtureFile(t, filepath.Join(root, "glossary.json"), `[
|
||||
{"term": "XKEYSCORE", "definition": "A search system.", "aka": ["XKS"], "see": ["rt-rg", "TURMOIL", "NOSUCHTERM"]},
|
||||
{"term": "RT-RG", "definition": "Real time regional gateway."},
|
||||
{"term": "TURMOIL", "definition": "Passive collection.", "see": ["xks"]}
|
||||
]`)
|
||||
withCorpus(t, root)
|
||||
body := get(t, "/glossary").Body.String()
|
||||
for _, want := range []string{`<a href="#term-RT-RG">rt-rg</a>`, `<a href="#term-TURMOIL">TURMOIL</a>`, `<a href="#term-XKEYSCORE">xks</a>`} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("/glossary lacks %s", want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(body, `href="#term-NOSUCHTERM"`) {
|
||||
t.Error("/glossary links a see-also name no entry has")
|
||||
}
|
||||
}
|
||||
|
||||
// A pager or sort link appends its parameter to queryBase, with nothing
|
||||
// left over when the page has no query of its own.
|
||||
func TestQueryBase(t *testing.T) {
|
||||
for want, q := range map[string]url.Values{
|
||||
"/browse?": {},
|
||||
"/browse?view=table&": {"view": {"table"}},
|
||||
"/browse?coll=foia&view=card&": {"view": {"card"}, "coll": {"foia"}},
|
||||
} {
|
||||
if got := queryBase("/browse", q); got != want {
|
||||
t.Errorf("queryBase(%v) = %q, want %q", q, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
22
tests/run.sh
Executable file
22
tests/run.sh
Executable file
|
|
@ -0,0 +1,22 @@
|
|||
#!/bin/sh
|
||||
# Runs the tests in this directory. They are package main's own tests, and Go
|
||||
# only compiles those from the package's directory, so an overlay presents
|
||||
# them to the go tool as if they were there. Arguments go to "go test":
|
||||
#
|
||||
# tests/run.sh
|
||||
# tests/run.sh -run TestCorpus -v
|
||||
set -e
|
||||
cd "$(dirname "$0")/.."
|
||||
root=$(pwd)
|
||||
overlay=$(mktemp)
|
||||
trap 'rm -f "$overlay"' EXIT
|
||||
{
|
||||
printf '{"Replace":{'
|
||||
sep=
|
||||
for f in tests/*_test.go; do
|
||||
printf '%s"%s/%s":"%s/%s"' "$sep" "$root" "${f#tests/}" "$root" "$f"
|
||||
sep=,
|
||||
done
|
||||
printf '}}\n'
|
||||
} >"$overlay"
|
||||
go test -overlay "$overlay" "$@" .
|
||||
1000
tests/search_test.go
Normal file
1000
tests/search_test.go
Normal file
File diff suppressed because it is too large
Load diff
58
tests/sidtoday_scope_test.go
Normal file
58
tests/sidtoday_scope_test.go
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The home figures describe the Reported collection; /?sidtoday=1 adds the
|
||||
// SIDtoday archive, and the note above them links between the two. In the
|
||||
// consistency corpus only s1 (SIDtoday archive) is dated 2011-05.
|
||||
func TestHomeStatsWithSIDtoday(t *testing.T) {
|
||||
withCorpus(t, writeConsistencyCorpus(t))
|
||||
plain := get(t, "/").Body.String()
|
||||
if !strings.Contains(plain, "exclude SIDtoday archive documents") || !strings.Contains(plain, `href="/?sidtoday=1"`) {
|
||||
t.Error("/: no note linking to the figures with the SIDtoday archive")
|
||||
}
|
||||
if strings.Contains(plain, "coll=sidtoday-archive") {
|
||||
t.Error("/: a figure links into the SIDtoday archive")
|
||||
}
|
||||
with := get(t, "/?sidtoday=1").Body.String()
|
||||
if !strings.Contains(with, "include SIDtoday archive documents") {
|
||||
t.Error("/?sidtoday=1: the note does not say the figures include the SIDtoday archive")
|
||||
}
|
||||
if !strings.Contains(with, `href="/browse/docdate/2011?coll=news&coll=sidtoday-archive"`) {
|
||||
t.Error("/?sidtoday=1: the document-date bars do not link with both collections")
|
||||
}
|
||||
}
|
||||
|
||||
// The "reported" badge tells an individually reported SIDtoday issue apart
|
||||
// from the bulk release, so only the SIDtoday tab shows it: every view of
|
||||
// that tab, and no other page.
|
||||
func TestReportedBadgeOnlyOnSIDtodayTab(t *testing.T) {
|
||||
root := writeConsistencyCorpus(t)
|
||||
d := filepath.Join(root, "r1")
|
||||
writeFixtureFile(t, filepath.Join(d, "r1.pdf"), "%PDF-1.4\n")
|
||||
writeFixtureFile(t, filepath.Join(d, "metadata.json"), `{"title": "Reported issue", "type": "SIDTODAY",
|
||||
"document": {"date_created": "2011-06-01"},
|
||||
"publication": [{"publisher": "The Intercept", "date_published": "2014-03-01", "link": "https://example.org/r"}]}`)
|
||||
withCorpus(t, root)
|
||||
|
||||
const badge = `class="issue-reported" title=`
|
||||
for _, p := range []string{"/browse?collection=sidtoday-archive", "/browse?collection=sidtoday-archive&view=card",
|
||||
"/browse?collection=sidtoday-archive&view=table"} {
|
||||
if !strings.Contains(get(t, p).Body.String(), badge) {
|
||||
t.Errorf("%s: no reported badge", p)
|
||||
}
|
||||
}
|
||||
for _, p := range []string{"/browse", "/browse?view=table", "/search?q=reported&do_search=1", "/browse/publisher/The%20Intercept"} {
|
||||
body := get(t, p).Body.String()
|
||||
if !strings.Contains(body, `href="/doc/r1"`) {
|
||||
t.Errorf("%s does not list r1", p)
|
||||
}
|
||||
if strings.Contains(body, badge) {
|
||||
t.Errorf("%s shows the reported badge", p)
|
||||
}
|
||||
}
|
||||
}
|
||||
464
tests/tmplcheck_test.go
Normal file
464
tests/tmplcheck_test.go
Normal file
|
|
@ -0,0 +1,464 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
"text/template/parse"
|
||||
)
|
||||
|
||||
// Every template is checked against the struct it is executed with, without
|
||||
// executing it: each field or method it reads must exist on the type it is
|
||||
// read from - in every branch, including the ones no test request takes.
|
||||
// html/template only finds a wrong name when the action runs.
|
||||
//
|
||||
// The checker follows the data through range, with, variables and template
|
||||
// calls. Where a type is not known statically it does not guess: a value
|
||||
// behind an interface{} (a dict's missing key, the argument of a partial
|
||||
// called with a literal) is accepted whatever is read from it.
|
||||
func TestTemplatesMatchPages(t *testing.T) {
|
||||
buildFixture(t)
|
||||
for _, p := range pageTypes {
|
||||
name := p.templateName()
|
||||
set, err := parsePage(name) // a fresh parse: executing rewrites the trees
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", name, err)
|
||||
}
|
||||
c := &tmplCheck{seen: map[string]bool{}, records: map[reflect.Type]bool{}}
|
||||
c.lookup = func(n string) *parse.Tree {
|
||||
if tt := set.Lookup(n); tt != nil {
|
||||
return tt.Tree
|
||||
}
|
||||
return nil
|
||||
}
|
||||
c.call(name, reflect.TypeOf(p), 0)
|
||||
sort.Strings(c.errs)
|
||||
for _, e := range c.errs {
|
||||
t.Errorf("%s (%T): %s", name, p, e)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Each template file is some page's, and each page's template exists.
|
||||
func TestEveryTemplateHasAPage(t *testing.T) {
|
||||
buildFixture(t)
|
||||
files, err := fs.Glob(templateFS, "*.html")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
byName := map[string]bool{}
|
||||
for _, p := range pageTypes {
|
||||
if byName[p.templateName()] {
|
||||
t.Errorf("%s is the template of two page types", p.templateName())
|
||||
}
|
||||
byName[p.templateName()] = true
|
||||
}
|
||||
for _, f := range files {
|
||||
if f == "base.html" || f == "partials.html" {
|
||||
continue
|
||||
}
|
||||
if !byName[f] {
|
||||
t.Errorf("%s has no page type in pageTypes", f)
|
||||
}
|
||||
delete(byName, f)
|
||||
}
|
||||
for n := range byName {
|
||||
t.Errorf("page type for %s, which is not a template file", n)
|
||||
}
|
||||
}
|
||||
|
||||
// The checker itself: a misspelled field is found in a branch that would
|
||||
// not run, behind a range, a variable and a partial; a dict's keys carry
|
||||
// their types into the partial they are passed to.
|
||||
func TestTemplateCheckerFindsWrongNames(t *testing.T) {
|
||||
type row struct{ Name string }
|
||||
type page struct {
|
||||
Rows []row
|
||||
Doc *Document
|
||||
M map[string]int
|
||||
}
|
||||
for src, want := range map[string]string{
|
||||
`{{ if .Nope }}x{{ end }}`: "Nope",
|
||||
`{{ if false }}{{ range .Rows }}{{ .Nmae }}{{ end }}{{ end }}`: "Nmae",
|
||||
`{{ range $i, $r := .Rows }}{{ $r.Name }}{{ $.Rowz }}{{ end }}`: "Rowz",
|
||||
`{{ with .Doc }}{{ .Title }}{{ .DocMeta.Creatr }}{{ end }}`: "Creatr",
|
||||
`{{ $d := .Doc }}{{ $d.Titel }}`: "Titel",
|
||||
`{{ template "p" .Doc }}{{ define "p" }}{{ .ID }}{{ .Idd }}{{ end }}`: "Idd",
|
||||
`{{ template "p" (dict "D" .Doc) }}{{ define "p" }}{{ .D.Idd }}{{ end }}`: "Idd",
|
||||
`{{ index .M "a" }}{{ (index .Rows 0).Nmae }}`: "Nmae",
|
||||
`{{ .Rows.Name }}`: "Name",
|
||||
`{{ range .Rows }}{{ .Name }}{{ else }}{{ .Name }}{{ end }}`: "Name",
|
||||
`{{ template "p" (dict "D" .Doc) }}{{ define "p" }}{{ .Missing.X }}{{ end }}`: "",
|
||||
`{{ range .Rows }}{{ .Name }}{{ end }}{{ .Doc.Title }}{{ len .M }}`: "",
|
||||
`{{ $x := "" }}{{ if .Doc }}{{ $x = .Doc }}{{ end }}{{ $x.Anything }}`: "",
|
||||
} {
|
||||
trees, err := parse.Parse("t", src, "", "", checkerFuncs())
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", src, err)
|
||||
}
|
||||
c := &tmplCheck{seen: map[string]bool{}, records: map[reflect.Type]bool{}}
|
||||
c.lookup = func(n string) *parse.Tree { return trees[n] }
|
||||
c.call("t", reflect.TypeOf(page{}), 0)
|
||||
got := strings.Join(c.errs, "; ")
|
||||
if (want == "") != (got == "") || !strings.Contains(got, want) {
|
||||
t.Errorf("%s\n errors: %q, want one naming %q", src, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// checkerFuncs is funcMap as the parser wants it, with the builtins the
|
||||
// templates use.
|
||||
func checkerFuncs() map[string]any {
|
||||
m := map[string]any{}
|
||||
for k, v := range funcMap {
|
||||
m[k] = v
|
||||
}
|
||||
for _, b := range []string{"and", "or", "not", "eq", "ne", "lt", "le", "gt", "ge", "len", "index",
|
||||
"slice", "print", "printf", "println", "html", "js", "urlquery", "call"} {
|
||||
m[b] = true
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
type tmplCheck struct {
|
||||
lookup func(name string) *parse.Tree
|
||||
seen map[string]bool // template name + argument type, already walked
|
||||
records map[reflect.Type]bool // types made from a dict call: a missing key is nil, not an error
|
||||
errs []string
|
||||
where string // the template being walked
|
||||
}
|
||||
|
||||
type tmplVar struct {
|
||||
name string
|
||||
typ reflect.Type // nil: not known
|
||||
}
|
||||
|
||||
var (
|
||||
stringType = reflect.TypeOf("")
|
||||
intType = reflect.TypeOf(0)
|
||||
boolType = reflect.TypeOf(false)
|
||||
floatType = reflect.TypeOf(0.0)
|
||||
anyType = reflect.TypeOf((*any)(nil)).Elem()
|
||||
)
|
||||
|
||||
func (c *tmplCheck) errorf(n parse.Node, format string, a ...any) {
|
||||
loc := c.where
|
||||
if tree := c.lookup(c.where); tree != nil {
|
||||
loc, _ = tree.ErrorContext(n)
|
||||
}
|
||||
c.errs = append(c.errs, fmt.Sprintf("%s (in %q): %s", loc, c.where, fmt.Sprintf(format, a...)))
|
||||
}
|
||||
|
||||
// call walks the named template with dot of type arg.
|
||||
func (c *tmplCheck) call(name string, arg reflect.Type, depth int) {
|
||||
key := name + "|"
|
||||
if arg != nil {
|
||||
key += arg.String()
|
||||
if c.records[arg] {
|
||||
key += fmt.Sprintf("|%p", arg) // records print alike only when they are alike; keep them apart anyway
|
||||
}
|
||||
}
|
||||
if c.seen[key] || depth > 20 {
|
||||
return
|
||||
}
|
||||
c.seen[key] = true
|
||||
tree := c.lookup(name)
|
||||
if tree == nil || tree.Root == nil {
|
||||
c.errs = append(c.errs, fmt.Sprintf("template %q is not defined", name))
|
||||
return
|
||||
}
|
||||
prev := c.where
|
||||
c.where = name
|
||||
vars := []tmplVar{{"$", arg}}
|
||||
c.walk(tree.Root, arg, &vars, depth)
|
||||
c.where = prev
|
||||
}
|
||||
|
||||
func (c *tmplCheck) walk(n parse.Node, dot reflect.Type, vars *[]tmplVar, depth int) {
|
||||
switch n := n.(type) {
|
||||
case *parse.ListNode:
|
||||
if n == nil {
|
||||
return
|
||||
}
|
||||
for _, x := range n.Nodes {
|
||||
c.walk(x, dot, vars, depth)
|
||||
}
|
||||
case *parse.ActionNode:
|
||||
c.pipe(n.Pipe, dot, vars)
|
||||
case *parse.IfNode:
|
||||
mark := len(*vars)
|
||||
c.pipe(n.Pipe, dot, vars)
|
||||
c.walk(n.List, dot, vars, depth)
|
||||
c.walk(n.ElseList, dot, vars, depth)
|
||||
*vars = (*vars)[:mark]
|
||||
case *parse.WithNode:
|
||||
mark := len(*vars)
|
||||
t := c.pipe(n.Pipe, dot, vars)
|
||||
c.walk(n.List, t, vars, depth)
|
||||
c.walk(n.ElseList, dot, vars, depth)
|
||||
*vars = (*vars)[:mark]
|
||||
case *parse.RangeNode:
|
||||
mark := len(*vars)
|
||||
t := c.cmds(n.Pipe, dot, vars)
|
||||
key, elem := rangeTypes(t)
|
||||
switch len(n.Pipe.Decl) {
|
||||
case 1:
|
||||
*vars = append(*vars, tmplVar{n.Pipe.Decl[0].Ident[0], elem})
|
||||
case 2:
|
||||
*vars = append(*vars, tmplVar{n.Pipe.Decl[0].Ident[0], key}, tmplVar{n.Pipe.Decl[1].Ident[0], elem})
|
||||
}
|
||||
c.walk(n.List, elem, vars, depth)
|
||||
*vars = (*vars)[:mark]
|
||||
c.walk(n.ElseList, dot, vars, depth)
|
||||
case *parse.TemplateNode:
|
||||
var arg reflect.Type
|
||||
if n.Pipe != nil {
|
||||
arg = c.pipe(n.Pipe, dot, vars)
|
||||
}
|
||||
c.call(n.Name, arg, depth+1)
|
||||
}
|
||||
}
|
||||
|
||||
func rangeTypes(t reflect.Type) (key, elem reflect.Type) {
|
||||
t = indirectType(t)
|
||||
if t == nil {
|
||||
return nil, nil
|
||||
}
|
||||
switch t.Kind() {
|
||||
case reflect.Slice, reflect.Array:
|
||||
return intType, known(t.Elem())
|
||||
case reflect.Map:
|
||||
return known(t.Key()), known(t.Elem())
|
||||
case reflect.Int:
|
||||
return intType, intType
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// known is t, or nil when t says nothing about the value (an interface).
|
||||
func known(t reflect.Type) reflect.Type {
|
||||
if t != nil && t.Kind() == reflect.Interface {
|
||||
return nil
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
func indirectType(t reflect.Type) reflect.Type {
|
||||
for t != nil && t.Kind() == reflect.Pointer {
|
||||
t = t.Elem()
|
||||
}
|
||||
return known(t)
|
||||
}
|
||||
|
||||
// pipe is the type a pipeline yields, declaring or assigning its variable.
|
||||
func (c *tmplCheck) pipe(p *parse.PipeNode, dot reflect.Type, vars *[]tmplVar) reflect.Type {
|
||||
if p == nil {
|
||||
return nil
|
||||
}
|
||||
t := c.cmds(p, dot, vars)
|
||||
for _, d := range p.Decl {
|
||||
name := d.Ident[0]
|
||||
if !p.IsAssign {
|
||||
*vars = append(*vars, tmplVar{name, t})
|
||||
continue
|
||||
}
|
||||
for i := len(*vars) - 1; i >= 0; i-- {
|
||||
if (*vars)[i].name == name {
|
||||
if (*vars)[i].typ != t {
|
||||
(*vars)[i].typ = nil // assigned two types: whichever branch ran decides
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
func (c *tmplCheck) cmds(p *parse.PipeNode, dot reflect.Type, vars *[]tmplVar) reflect.Type {
|
||||
var t reflect.Type
|
||||
for i, cmd := range p.Cmds {
|
||||
t = c.cmd(cmd, dot, vars, i > 0, t)
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
func (c *tmplCheck) cmd(cmd *parse.CommandNode, dot reflect.Type, vars *[]tmplVar, piped bool, prev reflect.Type) reflect.Type {
|
||||
first := cmd.Args[0]
|
||||
if id, ok := first.(*parse.IdentifierNode); ok {
|
||||
if id.Ident == "dict" {
|
||||
return c.dict(cmd, dot, vars)
|
||||
}
|
||||
args := make([]reflect.Type, 0, len(cmd.Args))
|
||||
for _, a := range cmd.Args[1:] {
|
||||
args = append(args, c.arg(a, dot, vars))
|
||||
}
|
||||
if piped {
|
||||
args = append(args, prev)
|
||||
}
|
||||
return c.fn(id, args)
|
||||
}
|
||||
for _, a := range cmd.Args[1:] { // a method's arguments
|
||||
c.arg(a, dot, vars)
|
||||
}
|
||||
return c.arg(first, dot, vars)
|
||||
}
|
||||
|
||||
func (c *tmplCheck) arg(n parse.Node, dot reflect.Type, vars *[]tmplVar) reflect.Type {
|
||||
switch n := n.(type) {
|
||||
case *parse.DotNode:
|
||||
return dot
|
||||
case *parse.FieldNode:
|
||||
return c.fields(n, dot, n.Ident)
|
||||
case *parse.VariableNode:
|
||||
for i := len(*vars) - 1; i >= 0; i-- {
|
||||
if (*vars)[i].name == n.Ident[0] {
|
||||
return c.fields(n, (*vars)[i].typ, n.Ident[1:])
|
||||
}
|
||||
}
|
||||
c.errorf(n, "undefined variable %s", n.Ident[0])
|
||||
case *parse.ChainNode:
|
||||
return c.fields(n, c.arg(n.Node, dot, vars), n.Field)
|
||||
case *parse.PipeNode:
|
||||
return c.pipe(n, dot, vars)
|
||||
case *parse.IdentifierNode:
|
||||
return c.fn(n, nil)
|
||||
case *parse.StringNode:
|
||||
return stringType
|
||||
case *parse.BoolNode:
|
||||
return boolType
|
||||
case *parse.NumberNode:
|
||||
if n.IsInt {
|
||||
return intType
|
||||
}
|
||||
return floatType
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// fields follows a chain of names from t.
|
||||
func (c *tmplCheck) fields(n parse.Node, t reflect.Type, names []string) reflect.Type {
|
||||
for _, name := range names {
|
||||
if t == nil {
|
||||
return nil
|
||||
}
|
||||
// A method, on the value or on its address.
|
||||
recv := t
|
||||
if recv.Kind() != reflect.Pointer && recv.Kind() != reflect.Interface {
|
||||
recv = reflect.PointerTo(recv)
|
||||
}
|
||||
if m, ok := recv.MethodByName(name); ok {
|
||||
t = nil
|
||||
if m.Type.NumOut() > 0 {
|
||||
t = known(m.Type.Out(0))
|
||||
}
|
||||
continue
|
||||
}
|
||||
s := indirectType(t)
|
||||
if s == nil {
|
||||
return nil
|
||||
}
|
||||
switch s.Kind() {
|
||||
case reflect.Struct:
|
||||
f, ok := s.FieldByName(name)
|
||||
if !ok || !f.IsExported() {
|
||||
if c.records[s] {
|
||||
return nil // a key this dict call did not pass
|
||||
}
|
||||
c.errorf(n, "no field or method %s in %s", name, s)
|
||||
return nil
|
||||
}
|
||||
t = known(f.Type)
|
||||
case reflect.Map:
|
||||
if s.Key().Kind() != reflect.String {
|
||||
c.errorf(n, "%s read from %s, a map without string keys", name, s)
|
||||
return nil
|
||||
}
|
||||
t = known(s.Elem())
|
||||
default:
|
||||
c.errorf(n, "%s read from %s, which has no fields", name, s)
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
// dict is the record a dict call builds: each literal key with the type of
|
||||
// the value passed for it.
|
||||
func (c *tmplCheck) dict(cmd *parse.CommandNode, dot reflect.Type, vars *[]tmplVar) reflect.Type {
|
||||
var fs []reflect.StructField
|
||||
args := cmd.Args[1:]
|
||||
for i := 0; i+1 < len(args); i += 2 {
|
||||
k, ok := args[i].(*parse.StringNode)
|
||||
t := c.arg(args[i+1], dot, vars)
|
||||
if !ok {
|
||||
return nil // a computed key: nothing is known about the record
|
||||
}
|
||||
if t == nil {
|
||||
t = anyType
|
||||
}
|
||||
fs = append(fs, reflect.StructField{Name: k.Text, Type: t})
|
||||
}
|
||||
defer func() { recover() }() // a key that is not a Go identifier: no record
|
||||
rt := reflect.StructOf(fs)
|
||||
c.records[rt] = true
|
||||
return rt
|
||||
}
|
||||
|
||||
// fn is the type a function call yields.
|
||||
func (c *tmplCheck) fn(id *parse.IdentifierNode, args []reflect.Type) reflect.Type {
|
||||
switch id.Ident {
|
||||
case "not", "eq", "ne", "lt", "le", "gt", "ge":
|
||||
return boolType
|
||||
case "len":
|
||||
return intType
|
||||
case "print", "printf", "println", "html", "js", "urlquery":
|
||||
return stringType
|
||||
case "and", "or":
|
||||
for _, a := range args {
|
||||
if a == nil || a != args[0] {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
if len(args) > 0 {
|
||||
return args[0]
|
||||
}
|
||||
return nil
|
||||
case "slice":
|
||||
if len(args) > 0 {
|
||||
return args[0]
|
||||
}
|
||||
return nil
|
||||
case "index":
|
||||
if len(args) == 0 {
|
||||
return nil
|
||||
}
|
||||
t := args[0]
|
||||
for range args[1:] {
|
||||
t = indirectType(t)
|
||||
if t == nil {
|
||||
return nil
|
||||
}
|
||||
switch t.Kind() {
|
||||
case reflect.Map, reflect.Slice, reflect.Array:
|
||||
t = known(t.Elem())
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return t
|
||||
case "call":
|
||||
return nil
|
||||
}
|
||||
f, ok := funcMap[id.Ident]
|
||||
if !ok {
|
||||
return nil // the parser has already refused an unknown function
|
||||
}
|
||||
ft := reflect.TypeOf(f)
|
||||
if ft.NumOut() == 0 {
|
||||
return nil
|
||||
}
|
||||
return known(ft.Out(0))
|
||||
}
|
||||
59
tests/versions_pages_test.go
Normal file
59
tests/versions_pages_test.go
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestChangedPages(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
v := filepath.Join(dir, "versions", "entier")
|
||||
writeTestFile(t, filepath.Join(v, "copy.pdf"), "%PDF-1.4")
|
||||
for _, n := range []string{"1", "2", "7", "8"} {
|
||||
writeTestFile(t, filepath.Join(v, "images", "image-"+n+".jpg"), "")
|
||||
}
|
||||
|
||||
var d Document
|
||||
if err := json.Unmarshal([]byte(`{
|
||||
"publication": [{"publisher": "Le Monde", "version": "entier"}],
|
||||
"versions": {"entier": {"changed_pages": [8, 7, 12, 7]}, "ghost": {"changed_pages": [1]}}
|
||||
}`), &d); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
d.ID, d.Dir = "prism", dir
|
||||
corpus().loadVersions(&d)
|
||||
|
||||
e := d.VersionFor("entier")
|
||||
if e == nil {
|
||||
t.Fatal("version entier not loaded")
|
||||
}
|
||||
// Numbers come from the filenames; positions are where those images sit
|
||||
// in the page grid (image-7 is the 3rd image). 12 has no image: kept,
|
||||
// without a position.
|
||||
if got := fmt.Sprint(e.ChangedPages); got != "[{7 3} {8 4} {12 0}]" {
|
||||
t.Errorf("changed pages %s, want [{7 3} {8 4} {12 0}]", got)
|
||||
}
|
||||
for pos, want := range map[int]bool{0: false, 1: false, 2: false, 3: true, 4: true} {
|
||||
if e.PageChanged(pos) != want {
|
||||
t.Errorf("PageChanged(%d) = %v, want %v", pos, !want, want)
|
||||
}
|
||||
}
|
||||
if e.DocID != "prism" {
|
||||
t.Errorf("DocID %q, want prism", e.DocID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPageNumber(t *testing.T) {
|
||||
for name, want := range map[string]int{"page-12.jpg": 12, "image-07.png": 7} {
|
||||
if n, err := pageNumber(name); err != nil || n != want {
|
||||
t.Errorf("pageNumber(%q) = %d, %v; want %d", name, n, err, want)
|
||||
}
|
||||
}
|
||||
for _, name := range []string{"cover.jpg", "page-a.jpg"} {
|
||||
if _, err := pageNumber(name); err == nil {
|
||||
t.Errorf("pageNumber(%q) should fail", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
101
tests/versions_search_test.go
Normal file
101
tests/versions_search_test.go
Normal file
|
|
@ -0,0 +1,101 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func versionedDoc(t *testing.T, main string, versions map[string]string, pubsJSON string) Document {
|
||||
t.Helper()
|
||||
var d Document
|
||||
if pubsJSON != "" {
|
||||
if err := json.Unmarshal([]byte(pubsJSON), &d); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
d.ID, d.FullText = "doc", main
|
||||
for _, slug := range []string{"lemonde", "spiegel"} {
|
||||
if text, ok := versions[slug]; ok {
|
||||
d.Versions = append(d.Versions, DocVersion{Slug: slug, FullText: text})
|
||||
}
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
func TestVersionNovelText(t *testing.T) {
|
||||
d := versionedDoc(t, "PRISM Overview\nShared line\n", map[string]string{
|
||||
"lemonde": "shared LINE\r\nNew in Le Monde\n---\nIn both copies\nNew in Le Monde\n",
|
||||
"spiegel": "In both copies\nOnly Der Spiegel\n",
|
||||
}, "")
|
||||
combined := versionNovelText(&d)
|
||||
|
||||
if got := d.Versions[0].NovelText; got != "New in Le Monde\nIn both copies\nNew in Le Monde" {
|
||||
t.Errorf("lemonde new lines %q", got)
|
||||
}
|
||||
if got := d.Versions[1].NovelText; got != "In both copies\nOnly Der Spiegel" {
|
||||
t.Errorf("spiegel new lines %q", got)
|
||||
}
|
||||
if combined != "New in Le Monde\nIn both copies\nOnly Der Spiegel" {
|
||||
t.Errorf("combined new lines %q, want each line once", combined)
|
||||
}
|
||||
}
|
||||
|
||||
func TestVersionLabel(t *testing.T) {
|
||||
d := versionedDoc(t, "", map[string]string{"lemonde": "", "spiegel": ""}, `{"publication": [
|
||||
{"publisher": "Le Monde", "date_published": "2013-10-21"},
|
||||
{"publisher": "Le Monde", "date_published": "2013-10-25"},
|
||||
{"publisher": "The Guardian", "date_published": "2013-10-20"}
|
||||
]}`)
|
||||
d.Versions[0].PubIndex = []int{0, 1, 2}
|
||||
if got := versionLabel(&d, &d.Versions[0]); got != "Le Monde / The Guardian, 2013-10-20" {
|
||||
t.Errorf("label %q", got)
|
||||
}
|
||||
if got := versionLabel(&d, &d.Versions[1]); got != "spiegel" {
|
||||
t.Errorf("label of a version no publication names %q, want its folder name", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestVersionTextIsSearchable(t *testing.T) {
|
||||
d := versionedDoc(t, "The overview ends with alpha", map[string]string{
|
||||
"lemonde": "The overview ends with alpha\nbeta starts the Wanadoo slide",
|
||||
}, "")
|
||||
setBodySearchText(&d)
|
||||
d.StemmedText = d.StemmedFull
|
||||
d.SearchText = d.FoldedFull
|
||||
|
||||
match := func(q, field string) bool {
|
||||
return compileQuery([]string{q}, []string{field}, []string{"and"}).matches(&d)
|
||||
}
|
||||
if !match("wanadoo", "FT") || !match(`"wanadoo slide"`, "FT") || !match("=wanadoo", "FT") || !match("wanad*", "FT") {
|
||||
t.Error("a term only in a version's new lines should match the document")
|
||||
}
|
||||
if match(`"alpha beta"`, "FT") {
|
||||
t.Error("a quoted phrase matched across the join between the main text and a version's lines")
|
||||
}
|
||||
if tok := tokenize("overview")[0]; strings.Count(" "+d.StemmedFull+" ", " "+tok+" ") != 1 {
|
||||
t.Errorf("a line shared with the main text was indexed twice: %q", d.StemmedFull)
|
||||
}
|
||||
|
||||
// buildTFIDF skips the separator only as a token of its own; glued to a
|
||||
// word it would mint terms and hide that word from phrase matching.
|
||||
for _, tok := range strings.Fields(d.StemmedFull) {
|
||||
if tok != valueSep && strings.Contains(tok, valueSep) {
|
||||
t.Errorf("the join separator is glued to a term in StemmedFull: %q", tok)
|
||||
}
|
||||
}
|
||||
idx := buildTFIDF([]Document{d})
|
||||
for w := range idx.DocFreq {
|
||||
if strings.Contains(w, valueSep) {
|
||||
t.Errorf("the join separator was indexed in the TF-IDF term %q", w)
|
||||
}
|
||||
}
|
||||
|
||||
hits := versionHits(&d, "wanadoo", 3)
|
||||
if len(hits) != 1 || hits[0].Version.Slug != "lemonde" || !strings.Contains(hits[0].Snippets[0], "Wanadoo") {
|
||||
t.Errorf("versionHits = %+v, want one lemonde hit with a Wanadoo snippet", hits)
|
||||
}
|
||||
if hits := versionHits(&d, "overview", 3); len(hits) != 0 {
|
||||
t.Errorf("versionHits found %+v for a word only in lines shared with the main text", hits)
|
||||
}
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue