{ "title": "X-KEYSCORE as a SIGDEV tool 2009", "type": "REPORTED", "description": "A 2009 TOP SECRET NSA presentation describing XKeyscore as a SIGINT development tool — explaining how it captures and temporarily stores the full content of internet traffic (the 95%+ that was previously dropped because it didn't match any target) across 125 sites worldwide, \"slowing down the internet\" so analysts can retroactively reconstruct a target's entire online session, discover new targets, and search the content of documents, web searches, emails and chats that contain no tasked selector.\n", "long_description": "This is a TOP SECRET//COMINT//ORCON//REL TO USA, AUS, CAN, GBR, NZL NSA presentation from 2009 presenting XKeyscore as a Digital Network Intelligence SIGDEV (signals intelligence development) tool. It is one of the foundational documents explaining XKeyscore's core architecture and purpose, complementing the more technical AppID and analyst-guide documents released in the same batch.\n\nThe presentation's central concept is that XKeyscore captures and retains the vast majority of internet traffic that traditional collection discarded. The slides explain that sites typically select and forward to PINWALE less than 5% of the DNI they process — the remaining 95%+ was previously dropped, but is now retained temporarily and made available through XKeyscore. At one site, XKeyscore was described as seeing more data per day than all of PINWALE. The goal is to store full-take content for 3-5 days — \"slowing down the Internet\" so analysts can recover sessions that the front end would otherwise have dropped — with metadata retained longer, targeting 30 days. At some sites, the 20+ terabytes received per day could only be stored for as little as 24 hours, operating as a rolling buffer where new data pushes out the oldest. Some sites had legal or policy restrictions limiting storage further.\n\nThe discovery options slide positions four systems on a spectrum from low to high data volume: TRAFFICTHIEF (metadata from tasked strong selectors), PINWALE (content selected from dictionary-tasked terms), MARINA (user activity metadata from full-take and selected feeds), and XKeyscore (unique data beyond user activity from front-end full-take feeds). XKeyscore queries metadata tables only, with results linked back to the original content. The metadata plug-ins span email addresses, extracted files, HTTP activity, phone numbers, document metadata (Author, Organization, dates from Office and PDF files), user activity (webmail and chat including buddy lists and machine-specific cookies), Cisco passwords, logins and passwords, CNE, TOR logs, and more. A single session emailing a Word attachment would populate the Full Log, Email Addresses, Extracted Files and Document Metadata tables simultaneously.\n\nThe presentation states XKeyscore was deployed at 125 sites and produced almost 1,300 AppIDs across 28 categories at the time. The AppID/fingerprint examples include PGP-encrypted emails (showing a Launchpad OpenPGP key confirmation captured in full), airline e-tickets (an Airblue Peshawar-to-Dubai ticket with booking and fare details), and extremist forum private messages (a vBulletin private message tagged with the forum/extremist/al-faloja fingerprint, containing an al-Shabaab claim of responsibility for an attack on Ethiopian forces in Mogadishu in Arabic).\n\nThe SIGDEV scenarios demonstrate the tool's operational power. Persona Analysis reconstructs everything a target did during a session — the worked example shows a Yahoo user from Pakistan (Kohat) with their full HTTP timeline across mail.ru, yahoo, facebook, Turkish and Russian sites, browser list, referer summary, extracted files, username summary across multiple webmail providers, and geographic IP summary. The NWFP (North-West Frontier Province) example shows an analyst noticing a Pakistani target looking at London in Google Earth and discovering a new strong selector (badguy@yahoo.com). The HTTP traffic examples show monitoring Google searches from Waziristan — a workflow named \"Waz_NWWFP_Foriegn_Googlers\" captured searches for \"al qaida,\" \"the al-Ikhlas network,\" \"Waziristan\" and jihadist forum terms, with Arabic search terms machine-translated. Another SIGDEV technique tracks who viewed video links passed by targets, discovering new targets who accessed the same files.toand load.toURLs.\n\nThe context-aware tagging and scanning sections describe targeting terms only when they appear inside document bodies (avoiding flooding PINWALE), foreign language tagging (Arabic expressions from Pakistani tribal regions), and regex-based discovery of MAC addresses inside documents. The document metadata exploitation example searches for all files where the Organization field is \"Warid Telecom\" — a Pakistani telecommunications company — returning internal troubleshooting documents, spreadsheets and letters of intent, many of which were captured despite having no tasked strong selector. The workflow list reveals sustained monitoring operations heavily focused on Pakistan's tribal regions: Waziristan internet searches, Zahedan Googlers, foreign peer-to-peer chats, daily Arabic from Waziristan, and proxy/anonymizer monitoring (megaproxy, Guardster, megaproxy).\n\nThe query hierarchy slide confirms the global federated architecture — user queries flow through XKeyscore Central to F6 HQS, FORNSAT sites and SSO sites, and onward to individual F6 (Special Collection Service) sites. One workflow screenshot bears the additional banner \"This system is audited for USSID 18 and Human Rights Act compliance,\" the reference to the UK Human Rights Act indicating the system was used at or shared with GCHQ sites.\n\nThe document is significant as the clearest published explanation of XKeyscore's foundational premise — that it provides retrospective full-take access to internet content that escaped traditional selector-based collection, transforming bulk interception into a searchable archive. The persona analysis and document metadata capabilities demonstrate that analysts could reconstruct a target's complete online behaviour and discover associated documents and selectors without any prior tasking, making XKeyscore a discovery engine rather than merely a retrieval tool.", "tags": [ "SIGDEV", "DNI", "full-take collection", "metadata", "persona analysis", "AppID", "fingerprints", "document metadata", "HTTP Activity", "user activity", "extracted files", "Cisco", "TOR log", "context-aware tagging", "context-aware scanning", "workflow", "Workflow Central", "Special Collection Service", "FORNSAT", "SSO", "125 sites", "rolling buffer", "slowing down the internet", "Waziristan", "NWFP", "Kohat", "Zahedan", "Google Earth", "Google", "al-Shabaab", "al-Faloja", "extremist forums", "web forums", "PGP", "OpenPGP", "encryption", "Warid Telecom", "Airblue", "Peshawar", "target discovery", "persona session collection" ], "document": { "creator": "NSA", "date_created": "2009", "classification": [ "TOP SECRET", "COMINT", "ORCON" ], "dissemination": [ "REL TO USA, AUS, CAN, GBR, NZL" ], "banner": "TOP SECRET//COMINT//ORCON,REL TO USA, AUS, CAN, GBR and NZL//20291123", "genre": "slides", "programs": [ "XKEYSCORE", "PINWALE", "TRAFFICTHIEF", "MARINA", "AGILITY", "PILBEAM", "MAILORDER", "CADENCE", "NKB", "PSC", "SSO", "UIS" ], "countries": [ "Pakistan", "Russia", "Turkey", "Switzerland" ], "people": [] }, "publication": [ { "publisher": "The Intercept", "date_published": "2015-07-01", "title": "XKEYSCORE: NSA’s Google for the World’s Private Communications", "link": "https://theintercept.com/2015/07/01/nsas-google-worlds-private-communications/", "reporters": [ "Marquis-Boire, Morgan", "Greenwald, Glenn", "Lee, Micah" ], "contributors": [], "doc_source": "https://theintercept.com/document/2015/07/01/xks-sigdev-tool/", "doc_url": "https://s3.documentcloud.org/documents/2116339/xks-as-a-sigdev-tool.pdf" }, { "publisher": "Metropolitan Books", "date_published": "2014-05-13", "title": "No Place to Hide", "link": "https://www.glenngreenwald.net/#BookDocuments", "reporters": [ "Greenwald, Glenn" ], "contributors": [], "version": "no-place-to-hide", "note": "The pages reprinted in the document appendix of No Place to Hide (appendix pages 70,80)." }, { "publisher": "SVT Nyheter", "date_published": "2013-12-11", "title": "FRA has access to controversial surveillance system", "link": "https://www.svt.se/nyheter/granskning/ug/fra-has-access-to-controversial-surveillance-system", "reporters": [ "Bergman, Sven", "Dyfvermark, Joachim", "Gallagher, Ryan", "Greenwald, Glenn", "Laurin, Fredrik", "Struwe, Filip" ], "contributors": [], "doc_source": "http://www.svt.se/ug/read-the-snowden-documents-from-the-nsa", "version": "nsa-slides-xkeyscore", "note": "SVT Nyheter published two slides from this deck — \"Slowing down the Internet\" and \"What makes XKS so good at SIGDEV?\", pages 4 and 6 — on 11 December 2013, a year and a half before The Intercept released the deck itself on 1 July 2015. The two slides appeared inside a three-slide PDF accompanying SVT's reporting on NSA's cooperation with Sweden's FRA." } ], "aka_id": [ { "id": "nsa-slides-xkeyscore" }, { "id": "EFF_-_20131211-svt-nsa_slides_xkeyscore" } ], "pdf_sha256": "8038e5e49318f07ab109badd6e8c087968c308981145a4c13877cca77a0189b1", "pages": 44, "versions": { "no-place-to-hide": { "pdf_sha256": "0c480486f0202bb3470affa52506ba2259b377c94106ab642ed5b570d6a89056" }, "nsa-slides-xkeyscore": { "pdf_sha256": "05397b912612a684f937469e66c5a8290b58aa3569b8a2adea2e3a74f36236ff" } } }