first commit
This commit is contained in:
commit
0d90363116
6510 changed files with 679146 additions and 0 deletions
60
data/selection-forwarding-readme/metadata.json
Normal file
60
data/selection-forwarding-readme/metadata.json
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
{
|
||||
"title": "Forwarding Selected MAILORDER Files Read Me",
|
||||
"type": "REPORTED",
|
||||
"description": "An UNCLASSIFIED//FOR OFFICIAL USE ONLY XKEYSCORE administrator guide explaining how to configure an XKS cluster to forward selected intercepted sessions via MAILORDER, including category throttling limits, output directory configuration and the process setup required to switch all Digital Network Intelligence filtering and selection to the XKEYSCORE collection system.",
|
||||
"long_description": "This is a technical administrator readme for configuring the MAILORDER session forwarding capability within XKEYSCORE, classified at the relatively low level of UNCLASSIFIED//FOR OFFICIAL USE ONLY — suggesting it was intended for a broader technical audience of site administrators across the collection infrastructure rather than restricted to cleared analysts.\n\nMAILORDER is XKEYSCORE's mechanism for forwarding selected intercepted sessions — those that have matched targeting criteria — onward to other NSA systems or repositories for further processing and reporting. The document describes how to enable this forwarding within an XKS cluster's configuration, and introduces an important operational detail: category throttling. The throttle sets a ceiling of 10,000 sessions per category per day and no more than 1GB of forwarded data per category, after which XKEYSCORE stops forwarding matching sessions for that category for the remainder of the day while continuing to process data for other categories. This limit resets every 24 hours.\n\nThe throttling mechanism reveals something significant about the scale of XKEYSCORE's collection: the system collected so much matching traffic that forwarding everything would be operationally unmanageable, requiring an automated rationing system to prioritise what actually gets sent downstream. Sessions that exceed the quota are not lost — the document notes explicitly that they remain scanned, tagged and queryable within XKEYSCORE itself — but they are not forwarded via MAILORDER for that day.\n\nThe configuration process described involves editing the central xks.config file, running setup processes for plugins and processes, pushing configuration changes to slave nodes via rsync, and performing a safe restart of process_data — a standard distributed systems administration workflow that illustrates the industrial-scale engineering underlying what was, in effect, a global wiretapping infrastructure.",
|
||||
"tags": [
|
||||
"session forwarding",
|
||||
"category throttling",
|
||||
"DNI",
|
||||
"Digital Network Intelligence",
|
||||
"XKS",
|
||||
"master server",
|
||||
"slave nodes",
|
||||
"collection infrastructure",
|
||||
"surveillance infrastructure",
|
||||
"quota management",
|
||||
"bulk collection",
|
||||
"sessionization",
|
||||
"administrator guide",
|
||||
"site configuration",
|
||||
"rsync",
|
||||
"process management"
|
||||
],
|
||||
"document": {
|
||||
"creator": "NSA",
|
||||
"date_created": "",
|
||||
"classification": [
|
||||
"UNCLASSIFIED"
|
||||
],
|
||||
"dissemination": [
|
||||
"FOUO"
|
||||
],
|
||||
"banner": "UNCLASSIFED//FOR OFFICIAL USE ONLY",
|
||||
"genre": "manual",
|
||||
"programs": [
|
||||
"MAILORDER",
|
||||
"XKEYSCORE"
|
||||
],
|
||||
"countries": [],
|
||||
"people": []
|
||||
},
|
||||
"publication": [
|
||||
{
|
||||
"publisher": "The Intercept",
|
||||
"date_published": "2015-07-01",
|
||||
"title": "XKEYSCORE: NSA’s Google for the World’s Private Communications",
|
||||
"link": "https://theintercept.com/2015/07/01/nsas-google-worlds-private-communications/",
|
||||
"reporters": [
|
||||
"Marquis-Boire, Morgan",
|
||||
"Greenwald, Glenn",
|
||||
"Lee, Micah"
|
||||
],
|
||||
"contributors": [],
|
||||
"doc_source": "https://theintercept.com/document/2015/07/01/selection-forwarding-readme/",
|
||||
"doc_url": "https://s3.documentcloud.org/documents/2116165/selection-forwarding-readme.pdf"
|
||||
}
|
||||
],
|
||||
"pdf_sha256": "4c3cc521bc519789e3ea6a0b0f7ac70e722e2753563bff94bd5897685faf16d5",
|
||||
"pages": 2
|
||||
}
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
UNCLASSIFED//FOR OFFICIAL USE ONLY
|
||||
|
||||
(U//FOUO) Forwarding Selected MAILORDER Files Read Me
|
||||
|
||||
(U//FOUO) This document explains how to configure an XKS cluster to forward selected sessions via MAILORDER. Sessions are still scanned, tagged, and queryable in XKEYSCORE, even if the quota limit (category throttling) has been exceeded for the day. Use this document to learn the procedures for switching all Digital Network Intelligence (DNI) filtering and selection to the XKEYSCORE collection system.
|
||||
|
||||
(U//FOUO) Configuring DNI Selection in xks.config
|
||||
|
||||
(U//FOUO) Follow these steps to configure `xks.config` to use the XKS collection system for DNI filtering and selection:
|
||||
|
||||
1. (U//FOUO) Logon as the user `oper`.
|
||||
2. (U//FOUO) At the command line from within any directory, type `viconfig` and then press Enter. The `xks.config` file will open.
|
||||
3. (U//FOUO) In the `inputs` section of `xks.config` set the following configurations:
|
||||
a. `forward sessions = yes` : This enables session forwarding on the selected output plugin. You can verify that session forwarding is set to `yes` in the `default.xml` file located in `$XSCORE_DIR/config/plugins`
|
||||
**Note**: Only after running the `xks setup plugins` process, described on page 2, will you see the `selected_output` plugin enabled in `default.xml`.
|
||||
b. `category_throttle = true` (default). This sets the maximum number of sessions per category to 10,000 and the total size forwarded for any given category to be no greater than 1GB. If either of these limits is reached, XKEYSCORE will no longer forward sessions that hit on that particular category for the rest of the day. It will, however, continue to process data for other categories. These limits are reset every 24 hours.
|
||||
c. All MAILORDER files will be stored on the Master server in:
|
||||
`/export/data/xkeyscore/outputs/mailorder`
|
||||
|
||||
1 |
|
||||
UNCLASSIFED//FOR OFFICIAL USE ONLY
|
||||
UNCLASSIFIED//FOR OFFICIAL USE ONLY
|
||||
|
||||
You can check the status of the category throttle using one of the following commands:
|
||||
* `mysqls quota status current`
|
||||
The `quota_limits` forwarding status for all categories.
|
||||
* `mysqls quota status yesterday`
|
||||
The `quota_limits` forwarding status for the previous day.
|
||||
* `mysqls quota_throttled current`
|
||||
The top 20 categories for the current day (The phrase "negative remaining" indicates the category was throttled).
|
||||
d. `output_location = master`: All MAILORDER files will be generated in:
|
||||
`/xks_data/<masterHostname>/outputs/mailorder_working`
|
||||
|
||||
Then the `mailorder_proc` process on the Master will rename and move them to:
|
||||
`/xks_data/<masterHostname>/outputs/mailorder`
|
||||
|
||||
e. `mailorder = yes` (default) : On the Master, this creates the `mailorder_proc` process that polls the `mailorder_working` directory and then moves/renames any MAILORDER files to `$XSCORE_DATA_DIR/outputs/mailorder`.
|
||||
4. (U//FOUO) Type: `:wq!` and then press Enter to save and exit `xks.config`.
|
||||
|
||||
(U//FOUO) Additional Processes
|
||||
|
||||
(U//FOUO) In addition to editing the configurations in `xks.config`, it is important to run several set-up processes.
|
||||
|
||||
1. (U//FOUO) As the `oper` user, at the command prompt, type `xks setup plugins` and press Enter. This ensures any applicable changes to plugin configurations will take effect.
|
||||
2. (U//FOUO) As the `oper` user, at the command prompt, type `xks setup processes` and press Enter. This will create the `mailorder_proc` process on the Master.
|
||||
3. (U//FOUO) At the command prompt, type `rsync push_config` and press Enter. This pushes the latest configuration changes to the slaves.
|
||||
4. (U//FOUO) As the `oper` user, at the command prompt, type `xks proc start` and press Enter. This will ensure all of the running processes pick up any configuration changes.
|
||||
5. (U//FOUO) At the command prompt, type `xks proc saferestart` and press Enter. After setup is complete, this restarts `process_datas` for the new configurations to take effect.
|
||||
|
||||
2 |
|
||||
UNCLASSIFIED//FOR OFFICIAL USE ONLY
|
||||
Loading…
Add table
Add a link
Reference in a new issue