first commit
This commit is contained in:
commit
0d90363116
6510 changed files with 679146 additions and 0 deletions
57
data/trafficthief-readme/metadata.json
Normal file
57
data/trafficthief-readme/metadata.json
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
{
|
||||
"title": "TRAFFICTHIEF Readme",
|
||||
"type": "REPORTED",
|
||||
"description": "A SECRET XKeyscore configuration manual describing TRAFFICTHIEF — NSA's corporate alerting and tipping system that receives real-time events from XKeyscore when targets are actively communicating, enriches them with geolocation data and immediately alerts analysts — covering socket and MAILORDER configuration methods for both Five Eyes and NOFORN tip streams.",
|
||||
"long_description": "This is a SECRET//REL TO USA, FVEY XKeyscore configuration read-me document describing TRAFFICTHIEF. The document is a technical configuration guide rather than an analytical presentation, providing step-by-step instructions for connecting XKeyscore field sites to the TRAFFICTHIEF alerting infrastructure.\n\nTRAFFICTHIEF is defined as NSA's corporate alerting and tipping system. When a tasked selector triggers a hit in XKeyscore, the system generates a real-time message called an event or tip. TRAFFICTHIEF receives these tips in real-time — meaning while targets are actively communicating — enhances each message with geolocation information and immediately alerts analysts. The system functions as a real-time bridge between passive bulk collection and active analyst attention, transforming XKeyscore from a retrospective search tool into a live surveillance alert system.\n\nThe configuration architecture involves several components. The Unified Targeting Tool (UTT) drives selector tasking through Site Selection Distribution Managers (SSDM) at field sites. XKeyscore's process_data_parent handles dictionary scanning, metadata extraction, database storage and content archival. Tips are forwarded to TRAFFICTHIEF via two transport mechanisms — direct socket connection over port 443, or MAILORDER file-based transport via a designated pickup directory. Two separate TRAFFICTHIEF server IP addresses are documented: 7.216.26.130:443 for Five Eyes (51's) tips and 7.216.26.2:443 for NOFORN tips — confirming that the alerting infrastructure maintains separate channels for different releasability levels.\n\nThe configuration process involves editing xks.config to set alert_output, alert_output_type, alert_instance and alert_host parameters, followed by xks setup plugins, xks rsync push_config and xks proc saferestart commands to propagate changes across the XKeyscore cluster's master and slave nodes. The alert_instance parameter embeds the originating site name into the XML of each tip, allowing analysts to determine which field site generated the alert.\n\nThe document is operationally significant for confirming that XKeyscore does not merely store data for retrospective analysis — it actively monitors for tasked selectors in real time and pushes alerts to analysts while targets are communicating, enabling immediate response to target communications activity.",
|
||||
"tags": [
|
||||
"real-time alerting",
|
||||
"tipping",
|
||||
"geolocation",
|
||||
"selectors",
|
||||
"field sites",
|
||||
"cluster",
|
||||
"master slave",
|
||||
"surveillance architecture"
|
||||
],
|
||||
"document": {
|
||||
"creator": "NSA",
|
||||
"date_created": "",
|
||||
"classification": [
|
||||
"SECRET"
|
||||
],
|
||||
"dissemination": [
|
||||
"REL TO USA, FVEY"
|
||||
],
|
||||
"banner": "SECRET//REL TO USA, FVEY",
|
||||
"genre": "manual",
|
||||
"programs": [
|
||||
"TRAFFICTHIEF",
|
||||
"XKEYSCORE",
|
||||
"UTT",
|
||||
"SSDM",
|
||||
"MAILORDER",
|
||||
"PATHMASTER"
|
||||
],
|
||||
"countries": [],
|
||||
"people": []
|
||||
},
|
||||
"publication": [
|
||||
{
|
||||
"publisher": "The Intercept",
|
||||
"date_published": "2015-07-01",
|
||||
"title": "XKEYSCORE: NSA’s Google for the World’s Private Communications",
|
||||
"link": "https://theintercept.com/2015/07/01/nsas-google-worlds-private-communications/",
|
||||
"reporters": [
|
||||
"Marquis-Boire, Morgan",
|
||||
"Greenwald, Glenn",
|
||||
"Lee, Micah"
|
||||
],
|
||||
"contributors": [],
|
||||
"doc_source": "https://theintercept.com/document/2015/07/01/trafficthief-readme/",
|
||||
"doc_url": "https://s3.documentcloud.org/documents/2116187/trafficthief-readme.pdf"
|
||||
}
|
||||
],
|
||||
"aka_id": [],
|
||||
"pdf_sha256": "2bcd00496c239710220b41af9e9db1f78e5eaffd97262e680a62866412aa3f76",
|
||||
"pages": 5
|
||||
}
|
||||
133
data/trafficthief-readme/trafficthief-readme.md
Normal file
133
data/trafficthief-readme/trafficthief-readme.md
Normal file
|
|
@ -0,0 +1,133 @@
|
|||
SECRET//REL TO USA, FVEY
|
||||
|
||||
(U//FOUO) TRAFFICTHIEF
|
||||
Configuration Read Me
|
||||
|
||||
(U) Overview
|
||||
|
||||
(S//REL) TRAFFICTHIEF (TT) is the NSA corporate alerting and tipping system. Independent collection systems such as XKEYSCORE provide real-time messages, called events (tips), to the TRAFFICTHIEF server. TRAFFICTHIEF receives these tips in real-time when targets are actively communicating, enhances messages with geo-location information and then immediately alerts analysts to this activity.
|
||||
|
||||
(U) Pre-Configuration Requirements
|
||||
|
||||
(U/FOUO) Before configuring xks.config for TRAFFICTHIEF tipping, you must:
|
||||
|
||||
1. (S//SI//REL) Confirm Unified Targeting Tool (UTT) tasking is delivered from a Site Selection Distribution Manager (SSDM). Please refer to the UTT Configuration Read Me for configuring UTT tasking.
|
||||
2. (S//SI//REL) Confirm the UTT sends selector targeting information to the SSDM to manage selector tasking at a field site. The SSDM receives load and update requests from UTT and is responsible for any site-specific processing that must be performed before forwarding the appropriate subset of information to the site-local collection system.
|
||||
3. (C) Confirm that port 443 is open if tips will be sent via socket to the TT server (e.g., using the alert_trafficthief plugin).
|
||||
|
||||
NOTE: (U//FOUO) If tips will be sent over MAILORDER (i.e., using the alert_mailorder plugin), then port 443 does not need to be opened.
|
||||
|
||||
4. (C) Confirm MAILORDER is configured to pick up MAILORDER files from the $XSCORE_DATA_DIR/outputs/mailorder directory on the Master server. To do this, create a MAILORDER ticket to set up the pick-up directory for your Master XKS server:
|
||||
a. Type go mailorder in the URL field of a browser on a high-side computer.
|
||||
b. Click PATHMASTER Remedy Ticket near the top right of the screen. The Remedy Ticket Submission screen will appear.
|
||||
c. In the bottom half of the Remedy Ticket Submission screen is two yellow Submit Remedy Ticket buttons. Click the top button of the two. (This ticket pertains to New Data-Flow). The ITSC login screen will appear.
|
||||
|
||||
1 |
|
||||
|
||||
SECRET//REL TO USA, FVEY
|
||||
|
||||
d. Enter the SID of your alternate POC and then click Continue. A screen containing contact information for you and your POC will appear.
|
||||
e. Add/Edit the contact information as appropriate and then click Continue to go to the Data transport service screen.
|
||||
f. Click Dataflow request and then click Continue. The Request for New Data Flow on an Existing Transport System screen will appear.
|
||||
g. Enter as much information as you can. In the Data Flow Change Description field, specifically indicate that the pick-up directory should be:
|
||||
/export/data/xkeyscore/outputs/mailorder
|
||||
h. Click Submit. You will receive a confirmation e-mail indicating the ticket has been received and more information may be requested before the ticket can be completed. Overall, the process might take anywhere from a few days to a week to complete.
|
||||
|
||||
(U//FOUO) Using Socket to Send 5I's Tips to TRAFFICTHIEF
|
||||
|
||||
(U//FOUO) Follow these steps to configure the xks.config file use Socket to send 5I's Tips to TRAFFICTHIEF:
|
||||
|
||||
1. (U//FOUO) Log on as the user oper.
|
||||
2. (U//FOUO) At the command line from within any directory, type viconfig and then press Enter. The xks.config file will open.
|
||||
3. (U//FOUO) In the inputs section of xks.config, set the following configurations:
|
||||
a. alert_output = true: This turns tipping \"on.\"
|
||||
b. alert_output_type = socket: This indicates that all output will be sent via secure socket.
|
||||
c. alert_instance = sitename: This helps the analyst determine where the tip came from. The sitename name is included in the XML of the actual tip sent to TRAFFICTHIEF.
|
||||
d. alert_host = 7.216.26.130:443: This is the IP address and port number where the alerts are sent.
|
||||
|
||||
Note: (U/FOUO) Which ever value is set equal to alert_instance is the value that is put into the instance field of the actual alert.
|
||||
|
||||
4. (U//FOUO) Type wq! and then press Enter to exit xks.config.
|
||||
|
||||
2 |
|
||||
|
||||
SECRET//REL TO USA, FVEY
|
||||
|
||||
5. (U//FOUO) Perform the following commands only after making changes in xks.config. At the command prompt, type:
|
||||
a. xks setup plugins: This ensures any applicable changes to plugin configurations will take effect.
|
||||
b. xks rsync push_config: This pushes the latest configuration files to the slaves in the cluster.
|
||||
c. xks proc saferestart: After setup is complete, this restarts process_data_parent's for the new configuration to take effect. This process loads all the dictionaries and fingerprints and then performs dictionary scanning, metadata extraction, databasing of metadata, and archival of content. When the parent is finished reloading, it will do a staggered restart of its children based on which slave the parent is running on.
|
||||
|
||||
(U//FOUO) Using Socket to Send NOFORN Tips to TRAFFICTHIEF
|
||||
|
||||
(U//FOUO) Follow these steps to configure the xks.config file use Socket to send NOFORN Tips to TRAFFICTHIEF:
|
||||
|
||||
1. (U//FOUO) Log on as the user oper.
|
||||
2. (U//FOUO) At the command line from within any directory, type viconfig and then press Enter. The xks.config file will open.
|
||||
3. (U//FOUO) In the inputs section of xks.config, set the following configurations:
|
||||
a. alert_output = true: This turns tipping \"on.\"
|
||||
b. alert_output_type = socket: This indicates that output will be sent via secure socket.
|
||||
c. alert_instance = sitename: This is the location where the tip came from. The sitename is retrieved from the actual XML of the tip that is generated.
|
||||
d. alert_host = 7.216.26.2:443: This is the IP address and port number where the alerts are sent.
|
||||
|
||||
Note: (U//FOUO) The instance in the actual alert is filled in with the value assigned to alert_instance in xks.config.
|
||||
|
||||
4. (U//FOUO) Type wq! and then press Enter to exit xks.config.
|
||||
|
||||
3 |
|
||||
|
||||
SECRET//REL TO USA, FVEY
|
||||
|
||||
5. (U//FOUO) Perform the following commands only after making changes in xks.config. At the command prompt, type:
|
||||
a. xks setup plugins: This enables the alert_trafficthief plugin with the configurations established in Step 3 above.
|
||||
b. xks rsync push_config: This pushes the latest configurations to the slaves in the cluster.
|
||||
c. xks proc saferestart: After setup is complete, this restarts process_data_parent's for the new configuration to take effect. This process loads all the dictionaries and fingerprints and then performs dictionary scanning, metadata extraction, databasing of metadata, and archival of content.
|
||||
|
||||
(U//FOUO) Using MAILORDER to Send 5I's Tips to TRAFFICTHIEF
|
||||
|
||||
(U//FOUO) Follow these steps to configure the xks.config file to use MAILORDER to send 5I's Tips to TRAFFICTHIEF:
|
||||
|
||||
1. (U//FOUO) Log on as the user oper.
|
||||
2. (U//FOUO) At the command line from within any directory, type viconfig and then press Enter. The xks.config file will open.
|
||||
3. (U//FOUO) In the inputs section of xks.config, set the following configurations:
|
||||
a. alert_output = true: This turns tipping \"on.\"
|
||||
b. alert_output_type = mailorder: This indicates that output will be sent via MAILORDER.
|
||||
c. alert_instance = sitename: This is the location where the tip came from. The sitename is retrieved from the actual XML of the tip that is generated.
|
||||
d. alert_host = : This is the IP address where the alerts are sent. No IP address is required because, in this case, tips are being sent via MAILORDER.
|
||||
|
||||
Note: (U//FOUO) The instance in the actual alert is filled in with the value assigned to alert_instance in xks.config.
|
||||
|
||||
4. (U//FOUO) Type wq! and then press Enter to exit xks.config.
|
||||
5. (U//FOUO) Perform the following commands only after making changes in xks.config. At the command prompt, type:
|
||||
a. xks setup plugins: This ensures any applicable changes to plugin configurations will take effect.
|
||||
|
||||
4 |
|
||||
|
||||
SECRET//REL TO USA, FVEY
|
||||
|
||||
b. xks rsync push_config: This pushes the latest configuration files to the slaves in the cluster.
|
||||
c. xks proc saferestart: After setup is complete, this restarts process_data_parent's for the new configuration to take effect. This process loads all the dictionaries and fingerprints and then performs dictionary scanning, metadata extraction, databasing of metadata, and archival of content.
|
||||
|
||||
(U//FOUO) Using MAILORDER to Send NOFORN Tips to TRAFFICTHIEF
|
||||
|
||||
(U//FOUO) Follow these steps to configure the xks.config file to use MAILORDER to send NOFORN Tips to TRAFFICTHIEF:
|
||||
|
||||
1. (U//FOUO) Log on as the user oper.
|
||||
2. (U//FOUO) At the command line from within any directory, type viconfig and then press Enter. The xks.config file will open.
|
||||
3. (U//FOUO) In the inputs section of xks.config, set the following configurations:
|
||||
a. alert_output = true: This turns tipping \"on.\"
|
||||
b. alert_output_type = mailorder: This indicates that output will be sent via MAILORDER.
|
||||
c. alert_instance = sitename: This is the location where the tip came from. The sitename is retrieved from the actual XML of the tip that is generated.
|
||||
e. alert_host = : This is the IP address where the alerts are sent. No IP address is required because, in this case, tips are being sent via MAILORDER.
|
||||
|
||||
Note: (U//FOUO) The instance in the actual alert is filled in with the value assigned to alert_instance in xks.config.
|
||||
|
||||
4. (U//FOUO) Type :wq! and then press Enter to exit xks.config.
|
||||
5. (C) Perform the following commands only after making changes in xks.config. At the command prompt, type:
|
||||
a. xks setup plugins: This ensures any applicable changes to plugin configurations will take effect.
|
||||
b. xks rsync push_config: This pushes the latest configuration files to the slaves in the cluster.
|
||||
c. xks proc saferestart: After setup is complete, this restarts process_data_parent's for the new configuration to take effect. This process loads all the dictionaries and fingerprints and then performs dictionary scanning, metadata extraction, databasing of metadata, and archival of content.
|
||||
|
||||
5 |
|
||||
|
||||
SECRET//REL TO USA, FVEY
|
||||
Loading…
Add table
Add a link
Reference in a new issue