snowchive-meta/data/add-sd-blackhole/metadata.json
2026-11-19 04:39:11 +02:00

135 lines
7.4 KiB
JSON

{
"title": "BLACK HOLE ANALYTICS, ADD/SD briefing, September 2009",
"type": "REPORTED",
"description": "This September 2009 GCHQ briefing describes the BLACK HOLE analytics infrastructure and Next Generation Events (NGE) project, which planned to scale from ~8 10-gigabit connections to ~200 10Gs by March 2011, storing intercepted data in massive flat-file repositories at Benhall and Bude before loading it into \"Question Focused Dataset\" (QFD) databases that power surveillance tools like MUTANT BROTH, KARMA POLICE, INFINITE MONKEYS, and others. The document outlines how \"probes\" extract Target Detection Identifiers (TDIs) from intercepted data feeds, store them in BLACK HOLE repositories, then enable analysts to profile targets' online activities, create social networks, investigate websites and forums, track web searches and Google Earth usage, and perform large-scale data mining across internet and telephony data.",
"long_description": "This classified GCHQ briefing from September 2009 provides a comprehensive technical overview of the BLACK HOLE analytics system and the ambitious Next Generation Events (NGE) roadmap designed to dramatically scale the agency's bulk surveillance infrastructure. The presentation describes a multi-phase expansion plan: \"Convergence 1\" with approximately 8 10-gigabit connections, followed by \"Internet Profiling 1\" (~80 10Gs serving ~260 experts), \"Cloud upscale\" (~50 10Gs), and ultimately \"Internet Profiling 2\" reaching ~200 10-gigabit connections by March 2011. The architecture centers on \"probes\" that tap into data feeds and extract various types of information—TDI (Target Detection Identifier) extraction, HTTP extraction, and VBulletin forum extraction—which then flows into BLACK HOLE flat-file storage systems located at GCHQ facilities in Benhall and Bude. The document explains that BLACK HOLE serves as intermediate storage \"after initial processing, and before being manipulated and correlated and loaded into the QFD database tables,\" with each specialized surveillance tool having its own Question Focused Dataset (QFD). The briefing proudly lists the analytical capabilities available \"on the desktop,\" showing logos of targeted companies and services including Inmarsat, Google, VISA, Facebook, Skype, and British Airways, organized around questions like \"Where?\", \"When?\", \"What?\", and \"With whom?\"\n\nThe document reveals how GCHQ's suite of surveillance tools integrates with the BLACK HOLE infrastructure to enable comprehensive monitoring of internet and telephony activities. Existing capabilities include MUTANT BROTH (creating profiles of targets' online activities), AUTO ASSOC (finding other identifiers for targets), SOCIAL ANIMAL (creating social networks), KARMA POLICE (investigating websites), INFINITE MONKEYS (analyzing web forums), MEMORY HOLE (tracking web searches), HRMAP (website investigation), and MARBLED GECKO (monitoring Google Earth usage). The briefing announces upcoming \"converged\" capabilities that will merge internet and telephony surveillance, including \"Evolved Mutant Broth\" for combined online and phone activity profiles, \"Hard Assoc\" for finding alternative identifiers across both domains, \"Evolved Social Animal\" for social networks including phone data, and \"Samuel Pepys\" for real-time monitoring. The presentation emphasizes the development of a \"Capability Dev Workspace\" where collection engineers, data miners, analytics developers, and analysts can collaboratively experiment with live operational data, with one slide humorously suggesting an analyst might exclaim \"Oh heck, I've caught Bin Laden\" while another character jokes about developing \"My FaceSpace events\" collection. The briefing highlights plans for \"data mining at scale\" using cloud computing to run large TNN (presumably Telephony Network Node) graphs, a system called Distillery to detect real-time anomalies like \"impossible travel,\" and the Joint Collaboration Environment (Innov8) for running large-scale analytics using both GCHQ and NSA data. The document concludes by stating future implications will enable analysts to \"easily monitor changes in our targets' profiles and networks,\" \"develop and trial new capabilities using real life analytical experiments,\" and \"respond quickly in a crisis\"—essentially describing an unprecedented capability for population-scale surveillance and profiling of internet users worldwide, all operating under the UK's permissive legal framework with minimal judicial oversight.",
"tags": [
"Next Generation Events",
"Applied Research",
"Benhall",
"Bude",
"GCHQ",
"structured storage",
"QFD",
"database",
"data feeds",
"HTTP interception",
"vBulletin",
"web forums",
"forums",
"TDI",
"GTDI",
"Generic Target Detection Identifier",
"Presence events",
"online identifiers",
"IP addresses",
"data processing",
"target correlation",
"data correlation",
"correlation",
"surveillance tools",
"analytic tools",
"internet profiling",
"profiling",
"JCE",
"Joint Collaboration Environment",
"INMARSAT",
"Google",
"VISA",
"Facebook",
"Skype",
"British Airways",
"targeted services",
"analysis",
"point in time",
"cookies",
"web cookies",
"browser cookies",
"telephony",
"GSM",
"surveillance architecture",
"direct access",
"fiber tapping",
"fiber optic cables",
"telecommunications",
"humor",
"real life experiments",
"web browsing",
"browsing history",
"probing",
"tapping",
"international cables",
"emails",
"instant messaging",
"social media monitoring",
"unique identifier",
"personal information",
"usernames",
"email addresses",
"monitor usage",
"de-anonymization",
"linking identifiers",
"web browsing history",
"single identifier",
"digital footprint",
"online activity",
"queries",
"search engines",
"MSN",
"Messenger",
"Windows Live Messenger",
"contact chaining",
"Google Maps",
"Google Earth"
],
"document": {
"creator": "GCHQ",
"date_created": "2009-09-01",
"classification": [
"TOP SECRET",
"STRAP 1"
],
"dissemination": [],
"banner": "TOP SECRET STRAP1",
"genre": "slides",
"programs": [
"BLACKHOLE",
"MUTANTBROTH",
"AUTOASSOC",
"INFINITEMONKEYS",
"KARMAPOLICE",
"SOCIALANIMAL",
"HRMAP",
"MEMORYHOLE",
"MARBLEDGECKO",
"HARDASSOC",
"SAMUELPEPYS",
"BLAZINGSADDLES",
"TELLURIAN",
"BUFFER",
"DISTILLERY",
"LOOKINGGLASS",
"NGE"
],
"countries": [],
"people": [
"bin Laden, Osama"
]
},
"publication": [
{
"publisher": "The Intercept",
"date_published": "2015-09-25",
"title": "Profiled: From Radio to Porn, British Spies Track Web Users' Online Identities",
"link": "https://theintercept.com/2015/09/25/gchq-radio-porn-spies-track-web-users-online-identities/",
"reporters": [
"Gallagher, Ryan"
],
"contributors": [],
"doc_source": "https://theintercept.com/document/2015/09/25/add-sd-black-hole/",
"doc_url": "https://s3.documentcloud.org/documents/2432226/add-sd-blackhole.pdf"
}
],
"pdf_sha256": "55203ffc7941d2a4cbbd49fbf5f7b7246df4b0e1b84f0b4affb9dda155e25f59",
"pages": 15
}