snowchive-meta/data/xks-system-administration/metadata.json
2026-11-19 04:39:11 +02:00

98 lines
8.9 KiB
JSON
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

{
"title": "XKEYSCORE System Administration",
"type": "REPORTED",
"description": "A December 2012 TOP SECRET XKeyscore System Administration training manual — the detailed technical guide for the engineers who ran XKeyscore field-site clusters — covering the system's Red Hat Linux architecture, master/slave cluster design, database and processing pipeline, the GUI (including built-in Tor, botnet and CNE search menus), the DeepDive full-take packet processor, the xks command-line tool, and monitoring and troubleshooting, with live screenshots from the Timberline site (USF-790) and Menwith Hill Station.",
"long_description": "This is a TOP SECRET//SI//REL TO USA, AUS, CAN, GBR, NZL XKeyscore System Administration course dated December 2012. Unlike the analyst-facing guides, this is the technical manual for the system administrators and engineers who install, configure, monitor and troubleshoot XKeyscore clusters at field sites.\n\nThe introduction and architecture sections describe XKeyscore as a Computer-to-Computer (C2C) exploitation system — a fully distributed processing and query system running on Red Hat AS5u8 Linux, installed in clusters of one master and one or more slaves that scale by adding servers. Three functional roles are described: web server/UI host, real-time processing host, and search/query host, which can be combined into hybrid systems. The data flow moves from front-end sources like WEALTHYCLUSTER2 through processing that extracts metadata into two database types — insert (i0) and query (q0). The processing pipeline is detailed: file_input_proc and sotf_dist receive sessions and load-balance them across process_data_parent instances, which fork child processes to extract metadata; xks_meta_ingester streams metadata to the insert database; and register_metadata_tables indexes completed tables and moves them to the query database.\n\nThe operating system and directory structure sections document the httpd, mysqld, NFS and autofs services, the NFS/automount configuration that lets cluster nodes share the /export/data/xkeyscore directory, and the complete /opt/xkeyscore directory layout including config subdirectories for dictionaries (holding TRAFFICTHIEF, CADENCE and fist tables), plugins and www.\n\nThe GUI section contains significant disclosures through its screenshots and menu listings. The login banner reads \"The system is accredited for USSID 18 and Human Rights Act compliance,\" and a prominent notice describes the Sensitive Targeting Approval (STA) process required at Menwith Hill Station before querying terms specific to a UK, British Dependent Territory or Second Party person, or wildcards likely to retrieve such entries — confirming the document reflects a UK/GCHQ-shared deployment. Most notably, the Search menu tree reveals the operational categories built into XKeyscore by this date: an Anonymizer section with dedicated searches for Tor Onion Address Survey, Tor Routers, Tor Software Downloads, Tor User Survey and Web Anonymizers; a Botnet section including QUANTUMBOT and ROGUESTITCH (with \"Rs Stolen Creds\"); and an extensive CNE section with searches for keyloggers, machine information, registry data, captured credentials, endpoint attached media, Kaspersky and Norton PSP (personal security product) detection, \"Fourth Party Passive Map,\" ANIMALHOUSE and MP TRII access logs, Odnoklassniki (a Russian social network), and vBulletin sessions. A CT section includes searches for Yemen mobile IPs, RAR/archive known-password attacks, Opera Mini headers and persistent evercookies.\n\nThe DeepDive section describes XKeyscore's full-take packet-processing variant, in which XKeyscore's own software handles all packet processing with no upfront filtering. A \"promoter\" component selects the richest traffic for retention — all strong selectors, full-take ASDF user activity metadata, and a subset of GENESIS signatures — typically fully processing 20-30% of site traffic. The front-end processes are named: Packet Splatter (ingests packets from files, network or capture cards), xFip (fast TCP/UDP stream reassembly, preserving original packet bundles with a packet API for microplugins), METTLESOME (reassembly of less common protocols), Promoter (rule-based filtering by keyword, country code or appid/fingerprint), and Defrag (full session rebuilding). The Promoter's example rules — \"allow appid chat.*\", \"allow country_code PK\", \"block country_code US-US\" — demonstrate how traffic retention was tuned by application type and country, explicitly blocking US traffic while promoting Pakistani traffic. DeepDive is noted as supporting new missions including Cyber and Bulk Crypt, with sessions displayable as Wireshark-style packet bundles.\n\nThe xks command-line tool section is exhaustive, documenting dozens of services (including royale_with_cheese for automatic updates, endace for Endace DAG capture cards, myricom for 10GigE cards, and virus_scanner) and actions (compile_genesis, install_slave, tasking_dump, correlation server management). The live xks info output identifies the Timberline site with SIGAD USF-790, PDDG IE, running XKeyscore version 1.5.9-65 with a 14-node cluster (master tlxksvr01 plus 13 slaves). Other processes documented include correlation_server (an in-memory map-reduce correlation engine), mpmr_server (the map-reduce server running the reducer portion of GENESIS v5 microplugins), signal_acquisition_loopback (which feeds modified packets back into the system, supporting packet recursion), and xks_comms_server (managing bandwidth-limited inter-site communications). A live query detail screenshot shows an email_addresses query against a hotmail.com address, with the classification string revealing MUSCULAR among the compartments.\nThe monitoring and troubleshooting sections provide operational commands (mysqls, sotf_stat, xks top, xks tail, xks monitor) with live output from Menwith Hill Station servers (mhxkssvr02-05) showing per-host session input rates and processing statistics, and extensive troubleshooting procedures for full disks, SOTF input problems, MySQL errors, processing failures, mailorder output and query problems.\n\nThe document is significant as the definitive technical reference on how XKeyscore actually operated at the infrastructure level. The built-in Tor, botnet and CNE search categories confirm that anonymity-tool tracking, botnet exploitation and computer-network-exploitation data handling were standard, menu-driven features of the deployed system by 2012 — not experimental additions. The DeepDive promoter's country-based traffic rules, the STA/Human Rights Act compliance apparatus, the MUSCULAR classification marking, and the identification of specific sites (Timberline USF-790, Menwith Hill) tie the abstract capabilities documented elsewhere to concrete operational deployments. The Endace and Myricom hardware references and the 20+ terabyte processing rates confirm the industrial scale of the collection infrastructure.",
"tags": [
"Tor",
"botnets",
"system administrator",
"C2C",
"query database",
"SOTF",
"anonymizers",
"Computer Network Exploitation",
"keylogger",
"registry",
"machine information",
"Kaspersky",
"Norton",
"PSP",
"fourth party",
"Odnoklassniki",
"vBulletin",
"Evercookie",
"Opera",
"PDDG",
"Menwith Hill",
"MHS"
],
"document": {
"creator": "NSA",
"date_created": "2012-12",
"classification": [
"TOP SECRET",
"SI"
],
"dissemination": [
"REL TO USA, AUS, CAN, GBR, NZL"
],
"banner": "TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL",
"genre": "slides",
"programs": [
"XKEYSCORE",
"WEALTHYCLUSTER",
"MAILORDER",
"CADENCE",
"ROGUESTITCH",
"QUANTUMBOT",
"COTTONCHAINSAW",
"MPTRII",
"SCARFACE",
"ZAMENSIS",
"MAPREDUCE",
"GENESIS",
"REDUCER",
"DEEPDIVE",
"ASDF",
"CYBER",
"BULKCRYPT",
"METTLESOME",
"PACKETSPLAYYER",
"PROMOTER",
"DEFRAG",
"XFIP",
"TRAFFICTHIEF",
"ENDACE",
"TIMBERLINE",
"MUSCULAR",
"FIST",
"STARPROC"
],
"countries": [
"Yemen",
"Pakistan",
"China"
],
"people": []
},
"publication": [
{
"publisher": "The Intercept",
"date_published": "2015-07-01",
"title": "XKEYSCORE: NSA’s Google for the World’s Private Communications",
"link": "https://theintercept.com/2015/07/01/nsas-google-worlds-private-communications/",
"reporters": [
"Marquis-Boire, Morgan",
"Greenwald, Glenn",
"Lee, Micah"
],
"contributors": [],
"doc_source": "https://theintercept.com/document/2015/07/01/xks-system-administration/",
"doc_url": "https://s3.documentcloud.org/documents/2116483/xks-system-administration.pdf"
}
],
"aka_id": [],
"pdf_sha256": "3e1ff9da55e81c6afa00e46edc191191b2e509b16b24c7639222cc968045f265",
"pages": 144
}