snowchive-meta/data/cloud-developers-exchange-july-2011/metadata.json
2026-11-19 04:39:11 +02:00

116 lines
11 KiB
JSON

{
"title": "ICTR Cloud Efforts",
"type": "REPORTED",
"description": "A GCHQ presentation details how the agency developed cloud-based big data analytics to achieve \"population scale\" surveillance—processing trillions of internet events to build geographic and behavioral profiles on every online identifier worldwide, enabling discovery of unknown targets through pattern analysis rather than traditional keyword searches.",
"long_description": "This GCHQ presentation from their ICT Research division describes a sophisticated suite of big data surveillance tools built on cloud computing and MapReduce frameworks to analyze internet activity at planetary scale. The system creates \"Geo-Time profiles for every Internet identifier\" the agency observes, condensing trillions of surveillance events into billions of individual profiles tracking when and where each email address, IP address, or phone number appears globally. Using techniques borrowed from commercial recommender systems—like Netflix's movie suggestions or Amazon's book recommendations—GCHQ applies the same algorithms to discover \"terrorists who like website X also like website Y,\" building association graphs that reveal communities and patterns invisible to traditional intelligence analysis.\n\nThe presentation reveals multiple codename programs working in concert: KARMA POLICE correlates IP addresses with website visits, SILVER LINING provides the bulk data infrastructure, and tools like AWKWARD TURTLE create website recommendation graphs based on target browsing behavior. GCHQ emphasizes this represents \"target discovery at population scale\" using \"bulk unselected events—all events for country or world,\" meaning they're analyzing complete internet traffic for entire nations without pre-selecting targets. The agency explicitly states they process data on individuals not suspected of wrongdoing, using graph analysis and machine learning to identify \"unknown targets\" by detecting unusual behavioral patterns in the noise of global communications.\n\nThe presentation demonstrates sophisticated use of graph theory and network analysis, treating the internet as a massive interconnected graph where nodes represent people, devices, websites, or search terms, and edges represent relationships or communications between them. Using open-source academic tools like Apache Spark's Bagel implementation and Carnegie Mellon's Pegasus graph mining system, GCHQ performs what they call \"graph componentization\"—breaking down massive networks into smaller communities and clusters. They explicitly reference research showing that large networks are \"dominated by Giant Connected Component\" with a \"loosely connected periphery,\" meaning most internet users form one massive interconnected web, but the interesting intelligence targets often exist in smaller, isolated components at the edges. By identifying these disconnected or loosely connected groups, analysts can focus on communities exhibiting unusual isolation or communication patterns that might indicate operational security by adversary groups.\n\nOne particularly revealing section describes building \"query term graphs\" from search engine data, where nodes represent search queries and weighted directed edges connect queries that the same user searched for within five minutes of each other. Using PageRank and Personalized PageRank algorithms (the same mathematics Google uses to rank web pages), GCHQ analyzes these graphs to understand how search behavior clusters around topics of interest. The presentation shows an example analysis seeded on the term \"GCHQ\" itself, revealing that users who search for GCHQ also frequently search for terms like \"Free People Check,\" \"MI6,\" \"MI5,\" \"Foreign & Commonwealth Office,\" \"KGB,\" and intriguingly, \"Wikileaks\" (ranking 110th). This meta-analysis of their own agency's search footprint demonstrates how the system can map the conceptual landscape around any topic by analyzing collective search behavior, potentially identifying radicalization pathways, research patterns, or investigative journalism into intelligence agencies.\n\nThe document is remarkably candid about the experimental nature of these techniques and their limitations. Unlike the confident certainty often projected in intelligence presentations, GCHQ acknowledges these are \"fragile\" methods requiring \"a critical mass of suitably skilled people\" and close collaboration between data scientists and operational analysts. They emphasize \"it's not about tool development but the development of new (and fragile) data mining techniques,\" suggesting these systems require constant human interpretation and refinement. The agency notes that automated target discovery \"can only be used to effect if it is tied in with analyst knowledge of other patterns of behaviour, possibly geo-related,\" meaning the algorithms identify candidates but human expertise remains essential for validation. They describe discovering \"phone groups of interest undiscoverable by traditional analysis\" through behavioral pattern matching, representing genuine operational successes in finding targets who successfully evaded keyword-based surveillance.\n\nThe presentation's discussion of \"Critical Success Factors\" reveals the organizational challenges of implementing mass surveillance at this scale. GCHQ identifies four requirements: technical expertise in data mining, analyst understanding of target modus operandi and ability to follow up leads, supporting IT infrastructure (provided by SILVER LINING), and crucially, \"bulk access to relevant data sets.\" This last requirement appears to be a point of internal friction—the repeated mentions of lacking access to CULT WEAVE and the inclusion of a Dilbert cartoon mocking cloud computing buzzwords suggest political and bureaucratic obstacles within the agency. The cartoon, showing a pointy-haired boss demanding cloud computing implementation \"so I have something to talk about at the executive meeting,\" hints at pressure to adopt trendy technologies without necessarily having clear operational justification, though the presentation then pivots to claiming genuine \"operational benefit\" from their cloud initiatives.\n\nThe final sections explore what GCHQ calls \"Exploratory Data Analysis of Internet/Cyber Events,\" asking fundamental questions about internet behavior at scale: What is statistically normal? Can they identify regional patterns in internet traffic? Can they fingerprint users based purely on browsing habits? Can they detect malicious sites by analyzing the graph structure of how users reach them? These research questions reveal an agency grappling with information overload, trying to develop mathematical frameworks for understanding global internet behavior so they can identify meaningful anomalies. They propose using techniques like \"BadRank\" (a play on PageRank) to identify malicious websites by analyzing link patterns from known-bad sites, and discuss analyzing the complete netflow dataset to understand IP-level traffic patterns. The document acknowledges a \"gap in understanding of events at Internet Scale,\" suggesting that even with access to vast surveillance feeds, making sense of the data remains a fundamental challenge.\n\nThe slides discuss \"trillions of events,\" \"population scale,\" \"bulk unselected events,\" and \"all events for country or world\" in clinical, technical language, treating mass surveillance as primarily a data engineering challenge. There's no discussion of legal frameworks, privacy implications, or the ethics of building profiles on every internet user globally. The agency frames this as target discovery—finding terrorists and adversaries—but the methodology requires first surveilling everyone to identify the few. The document's matter-of-fact tone about processing \"all the events we have for a country\" and building profiles for \"every Internet identifier we see\" reveals an intelligence culture of comprehensive population monitoring where everyone's data feeds the machine learning systems that might, eventually, identify someone worth investigating more closely.",
"tags": [
"Dilbert comic",
"pattern-of-life",
"complete surveillance",
"ICTR",
"ICT Research",
"big data",
"cloud computing",
"MapReduce",
"population-scale surveillance",
"bulk collection",
"unselected collection",
"target discovery",
"graph analysis",
"Geo-Time profiles",
"behavioral analysis",
"modus operandi",
"internet metadata",
"mass surveillance",
"data mining",
"machine learning",
"graph componentization",
"community detection",
"recommender systems",
"social network analysis",
"trillions of events",
"behavioral profiling",
"exploratory data analysis",
"cyber intelligence",
"geolocation",
"IP correlation",
"website profiling",
"search query monitoring",
"personalized PageRank",
"network analysis",
"contact chaining",
"association scoring",
"bi-partite graphs",
"weighted graphs",
"giant connected component",
"alternate identifiers",
"relationship mapping",
"query flow graphs",
"browser fingerprinting",
"malicious site detection",
"botnets",
"distributed attack detection",
"netflow analysis"
],
"document": {
"creator": "GCHQ",
"date_created": "2011-07-01",
"classification": [
"TOP SECRET",
"STRAP 1"
],
"dissemination": [
"5EYES"
],
"banner": "UK TOP SECRET STRAP1 5EYES",
"genre": "slides",
"programs": [
"EVERYASSOC",
"PROBABILITYCLOUD",
"SILVERLINING",
"CULTWEAVE",
"KARMAPOLICE",
"EVERYPOLICE",
"AWKWARDTURTLE",
"GORDIANKNOT",
"SANDIA",
"BLOODHOUND",
"FIVEALIVE",
"NGE",
"MAPREDUCE"
],
"countries": [],
"people": [
"Adams, Scott",
"Wright, Peter",
"Dean, Jeffrey",
"Ghemawat, Sanjay",
"Leskovec, Jure",
"Lang, Kevin J.",
"Dasgupta, Anirban",
"Mahoney, Michael W.",
"Boldi, Paolo",
"Bonchi, Francesco",
"Castillo, Carlos",
"Donato, Debora",
"Gionis, Aristides",
"Vigna, Sebastiano"
]
},
"publication": [
{
"publisher": "The Intercept",
"date_published": "2015-09-25",
"title": "Profiled: From Radio to Porn, British Spies Track Web Users' Online Identities",
"link": "https://theintercept.com/2015/09/25/gchq-radio-porn-spies-track-web-users-online-identities/",
"reporters": [
"Gallagher, Ryan"
],
"contributors": [],
"doc_source": "https://theintercept.com/document/2015/09/25/cloud-developers-exchange-july-2011/",
"doc_url": "https://s3.documentcloud.org/documents/2432252/cloud-developers-exchange-july-2011.pdf"
}
],
"pdf_sha256": "dc12ef88cb5c97cce978357e1c6bdcce4cd4a5768bb51d13dc0a7c095cf5b864",
"pages": 29
}