{ "title": "XKEYSCORE Appids & Fingerprints", "type": "REPORTED", "description": "A TOP SECRET XKeyscore developer training presentation describing the complete AppID and fingerprint programming language — the system NSA analysts and developers use to write pattern-matching rules that automatically classify every piece of intercepted internet traffic by application type, protocol, encryption method and target behaviour — including syntax for keyword matching, regular expressions, binary patterns, context-sensitive scanning and embedded C++ code execution against live intercepted sessions.", "long_description": "This is a TOP SECRET//COMINT//REL TO USA, FVEY XKeyscore technical training presentation from the xkeyscore@nsa development team. Unlike the analyst-facing XKeyscore documents that teach how to query stored data, this presentation teaches developers how to write the pattern-matching rules that classify intercepted internet traffic in real time as it flows through XKeyscore's processing pipeline. Every intercepted session receives one AppID (the most specific matching application identifier) and all matching fingerprints, which are then stored in the database and used for querying, forwarding and alerting.\n\nThe AppID/fingerprint system uses a C-like syntax with two primary functions. AppIDs classify traffic by application type using a hierarchical naming convention (e.g., chat/yahoo/incoming, mail/smtp/to_server, voip/sip/invite) with priority levels from 1.0 (highest) to 9.9 (lowest) determining which classification wins when multiple signatures match. Fingerprints tag all matching patterns without priority competition and are stored alongside the winning AppID. The basic pattern matching supports Boolean logic (and, or, not), raw text keywords with minimum three-character length, regular expressions, binary/hex patterns, case sensitivity flags and UTF-16 matching. Built-in functions provide matching against IP addresses, ports, protocols, MAC addresses, email addresses (with DECODEORDAIN-style permutation) and positional constraints. Chainwords allow reusable pattern variables — predefined chainwords include $tcp, $udp, $http, $ssl, $vbulletin and $user_agent among others.\n\nContext-sensitive scanning is a key capability, allowing patterns to be evaluated only within specific extracted fields rather than across the entire session. Sample contexts include html_title, url, http_host, http_referer, http_cookie, user_agent, web_search, email_body, chat_body, doc_author, doc_org and filename — meaning a fingerprint can target only Google Maps requests appearing in the HTTP Host header rather than matching any mention of maps.google.com anywhere in a session. This improves both accuracy and performance.\n\nThe advanced syntax sections describe three progressively more powerful capabilities. Code-based AppIDs combine a keyword/regex trigger with embedded C++ code for secondary validation — useful for verifying packet structure, checksums, length fields and encoded content that cannot be matched with pattern syntax alone. The examples show OSPF protocol verification, VPN packet inspection examining specific byte positions, and Office document filename matching. Extractors provide simplified regex-based metadata extraction with named capture groups. Flex support enables lex-style tokenised pattern matching against session content. Microplugins represent the most advanced capability — giving code-based AppIDs limited access to XKeyscore's core, including reading session metadata (IP addresses, ports), throwing events (document_metadata events that feed into the document metadata search), and contributing derived fields to database tables like HTTP Activity (for example, extracting and URL-decoding Google Maps search terms for storage in the search_terms field).\n\nThe real-world example slide shows a captured Arabic-language vBulletin forum private message containing a complete \"ASRAR El Mojahedeen v2.0 Encrypted Message\" — the Mujahideen Secrets 2.0 encryption tool — demonstrating the operational context in which these fingerprints operate. The distribution slide confirms that AppID and fingerprint definitions are pushed across the entire global XKeyscore network every hour, with changes taking effect within two hours of check-in. The validation tooling shows AppID definitions loaded from configuration files including anonymizer.appid, bulletin_board.appid, tao_vpn.appid, terminal.appid and voip.appid, with unit testing against sample traffic captures.", "tags": [ "AppID", "fingerprints", "pattern matching", "C++", "microplugins", "chainword", "context-sensitive scanning", "regular expression", "binary pattern", "session metadata", "HTTP Activity", "document metadata", "SMTP", "ICQ", "OSPF", "Skinny", "VPNs", "Kazaa", "Google Maps", "vBulletin", "Mujahideen Secrets" ], "document": { "creator": "NSA", "date_created": "", "classification": [ "TOP SECRET", "COMINT" ], "dissemination": [ "REL TO USA, FVEY" ], "banner": "TOP SECRET//COMINT//REL TO USA, FVEY", "genre": "slides", "programs": [ "XKEYSCORE", "DECODEORDAIN" ], "countries": [], "people": [] }, "publication": [ { "publisher": "The Intercept", "date_published": "2015-07-01", "title": "XKEYSCORE: NSA’s Google for the World’s Private Communications", "link": "https://theintercept.com/2015/07/01/nsas-google-worlds-private-communications/", "reporters": [ "Marquis-Boire, Morgan", "Greenwald, Glenn", "Lee, Micah" ], "contributors": [], "doc_source": "https://theintercept.com/document/2015/07/01/xks-application-ids/", "doc_url": "https://s3.documentcloud.org/documents/2116302/xks-application-ids.pdf" } ], "aka_id": [], "pdf_sha256": "2e15431de13a51d2c49002a9fca5ab411dd646f36e66549847267f6f76e7e53f", "pages": 42 }