snowchive-meta/data/xks-cipher-detection-and-you/metadata.json
2026-11-19 04:39:11 +02:00

109 lines
7.9 KiB
JSON
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

{
"title": "XKEYSCORE, Cipher Detection, and You!",
"type": "REPORTED",
"description": "A TOP SECRET presentation from NSA's Mathematics Research Group dated 21 August 2008, explaining how XKeyscore detects encrypted and steganographic communications — describing the fingerprint and C++ plug-in systems used to flag sessions using particular ciphers, VPNs, encrypted archives and hidden-data tools, with worked examples detecting RAR archives, OpenVPN, digital certificates and various encryption products, plus a revealing note that detection of encryption is itself used to identify communications worth targeting.\n",
"long_description": "This is a TOP SECRET//COMINT//REL TO USA, FVEY presentation authored by NSA's Mathematics Research Group, dated 21 August 2008. The presentation focuses specifically on using XKeyscore to detect cipher use, encryption and steganography — a complement to the analyst and developer training documents that focus on traffic classification and querying.\n\nThe presentation situates XKeyscore within the collection architecture. Data sources feeding the system include FORNSAT (satellite downlink), Overhead (uplink), Special Source, Tailored Access, F6 (Special Collection Service), FISA (described as limited) and third party. Front-end processing by WEALTHYCLUSTER2/TURMOIL demodulates, demultiplexes, packetizes and sessionizes the traffic, with a \"Decrypt?\" step noted. XKeyscore then selects tasked CADENCE/UTT terms, sends hits to PINWALE/PRESSUREWAVE, tips to TRAFFICTHIEF, performs fingerprinting and supports SIGINT development through two rolling buffers — metadata (approximately 30 days, searchable in a MySQL database) and content (approximately 7 days, retrievable from disk archive). This document gives specific buffer figures: ~30 days metadata, ~7 days content, somewhat longer than the 3-5 day content figure in other documents, reflecting site variation.\n\nThe fingerprinting system is described in technical detail. Fingerprints are pattern matches that mark a session as metadata without sending it to PINWALE — the analyst must search for the fingerprint to retrieve flagged sessions. Strings require a minimum of three fixed-byte anchors (with an exception for two bytes at the start of a session), regular expressions must contain a non-optional three-byte string, and context-dependent terms are supported, with XKeyscore reserving the right to increase the minimum to four bytes. The worked examples reveal specific encryption detection fingerprints: encryption/helixstronghold matching a Helix Stronghold encrypted file header, encryption/wharfrat matching specific byte sequences on ports 443 or 80, encryption/the_algorithm using a regex, encryption/kryptel matching a hex signature, encryption/cipherpad matching a CPAD1 marker within the first 4000 bytes, and certificate/digital_id matching certificate headers. VPN detection examples include vpn/openvpn/x509/wera matching OpenVPN over UDP.\n\nThe syntax features section documents the fingerprint language's case sensitivity, full Boolean logic with parenthetical grouping, variables, and available functions including port, first, hex, pos, distance, lpos, last, follows, between and order. Fingerprint definitions are updated hourly across the entire enterprise. The plug-in section describes using the full power of C++ when pattern matching is insufficient, typically limited to certain file types due to volume (huge JPEG volume from web surfing is noted). Named steganography and encryption plug-ins include PHOSPHORESSENCE (a library of steganography detectors), SHELLLOCK (steganography detection) and SEDENA (indigenous/foreign encryption software detection). The drawback noted is that plug-ins require a site upgrade to deploy. The compromise solution — snippets of C++ code embedded in fingerprints, deployed hourly with most of the flexibility of a full plug-in — is shown detecting RAR encrypted archives by checking specific byte flags, and an anchorless regex follow-on check for DHCP client-to-server traffic.\n\nThe releasability section contains significant compartment disclosures. Nearly all XKeyscore personnel hold PICARESQUE, with those lacking it holding PRIVAC. XKeyscore distribution comes in two flavours — first and second party, and third party — with no NOFORN capabilities permitted in the shared versions. Special dispensation was granted for some capabilities to SMOKYSINK. PICARESQUE code could be kept running on R1's rednet if absolutely necessary. PICARESQUE is the same ECI compartment named in the BULLRUN briefing as one of the three named exceptionally controlled compartments protecting NSA's cryptanalytic capabilities, confirming that XKeyscore's encryption-detection and exploitation functions were protected under the same compartment that shielded NSA's broader anti-encryption programme.\n\nThe document notes that all data are stored, not just hits, and that a session with detected encryption is marked but not sent to PINWALE. The integration of mathematical/cryptanalytic research into XKeyscore's fingerprinting, the named steganography detection libraries, and the PICARESQUE compartment linkage together show that XKeyscore was not merely a traffic-collection and classification system but an active front-line tool in NSA's effort to identify and defeat encrypted communications.",
"tags": [
"Mathematics Research Group",
"cipher detection",
"encryption detection",
"steganography",
"fingerprints",
"FORNSAT",
"F6",
"FISA",
"RAR",
"OpenVPN",
"x509",
"digital certificate",
"Kryptel",
"CipherPad",
"Helix Stronghold",
"VPN analysis",
"C++ plug-in",
"code-based fingerprint",
"rolling buffer",
"metadata",
"content buffer",
"retrospective retrieval",
"anchor",
"regex",
"DHCP",
"ECI",
"releasability",
"rednet"
],
"document": {
"creator": "NSA",
"date_created": "2008-08-21",
"classification": [
"TOP SECRET",
"COMINT"
],
"dissemination": [
"REL TO USA, FVEY"
],
"banner": "TOP SECRET//COMINT//REL TO USA, FVEY",
"genre": "slides",
"programs": [
"XKEYSCORE",
"WEALTHYCLUSTER",
"TURMOIL",
"CADENCE",
"UTT",
"PINWALE",
"PRESSUREWAVE",
"TRAFFICTHIEF",
"PHOSPHORESSENCE",
"SHELLLOCK",
"SEDENA",
"SMOKYSINK",
"PICARESQUE",
"PRIVAC",
"WHARFRAT"
],
"countries": [],
"people": []
},
"publication": [
{
"publisher": "The Intercept",
"date_published": "2015-07-01",
"title": "XKEYSCORE: NSA’s Google for the World’s Private Communications",
"link": "https://theintercept.com/2015/07/01/nsas-google-worlds-private-communications/",
"reporters": [
"Marquis-Boire, Morgan",
"Greenwald, Glenn",
"Lee, Micah"
],
"contributors": [],
"doc_source": "https://theintercept.com/document/2015/07/01/xks-cipher-detection/",
"doc_url": "https://s3.documentcloud.org/documents/2157058/xks-cipher-detection-and-you.pdf"
},
{
"publisher": "SVT Nyheter",
"date_published": "2013-12-11",
"title": "Read the Snowden Documents From the NSA",
"link": "http://www.svt.se/ug/read-the-snowden-documents-from-the-nsa",
"reporters": [
"Rensfeldt, Gunnar"
],
"contributors": [],
"doc_url": "https://s3.documentcloud.org/documents/894407/xkeyscore-sources-dated-2008.pdf",
"version": "xkeyscore-sources-dated-2008"
}
],
"aka_id": [
{
"id": "EFF_-_20131211-svt-xkeyscore_sources"
}
],
"archive_comment": "SVT Nyheter only published one slide (\"Data Sources\", slide 3) from this deck in December 2013.",
"pdf_sha256": "56cd85589f3a9e3ca5d74d200a55015a15bdff8d603d7b76e83760fc50ad3b9d",
"pages": 17,
"versions": {
"xkeyscore-sources-dated-2008": {
"pdf_sha256": "f0dcfae86d9c6837a4542ede41f5e4c7cb9c977553fe49f121ecf97c158febec"
}
}
}