{ "title": "GCHQ Analytic Cloud Challenges", "type": "REPORTED", "description": "The 2012 GCHQ Analytic Cloud document outlines a sophisticated data processing infrastructure handling approximately 50 billion daily events across multiple sites, utilizing QFD databases and HADOOP clusters for various analytic capabilities targeting intelligence gathering and analysis. It describes architectural components, challenges in scaling and enabling analysts to cope with data complexity, and proposes future technology options including consolidation approaches and new cloud architectures to enhance capabilities.", "long_description": "The March 2012 GCHQ Analytic Cloud document reveals the staggering scale of the agency's surveillance capabilities, processing approximately 50 billion daily events from 250 high-capacity data streams, a volume expected to grow to 320 Bearers with a capacity for 100 billion events per day by May 2012. This data tsunami is funneled through a complex architecture of Query Focused Databases (QFDs) and HADOOP clusters, which power an arsenal of covert analytics tools with names like KARMA POLICE, SOCIAL ANTHROPOID, and EPIC FAIL—software designed to track individuals' communications, map social connections, and even detect careless use of privacy networks like TOR. The system collects and retains terabytes of metadata for six months on everything from web visits to phone calls, feeding intelligence products that GCHQ shares with foreign partners through web interfaces and data brokering services.\n\nDespite its immense capabilities, the document underscores significant challenges in managing this surveillance behemoth, highlighting concerns about the \"affordable, continuing scale\" of operations as power, cooling, and storage demands strain resources. GCHQ analysts are struggling to cope with the \"complexity and volume of data,\" prompting efforts to streamline workflows and develop \"push analytics\" like Rumour Mill, which automatically surfaces information about targets. The agency is actively exploring new architectures, including a \"skinny cloud\" and consolidated database systems, to increase efficiency and flexibility while wrestling with how to value and prioritize the deluge of information it collects—a constant internal debate about what to keep and what to discard in a surveillance system that is simultaneously omnipresent and overwhelmed.", "tags": [ "cloud analytics", "big data", "mass surveillance", "signals intelligence", "metadata", "event processing", "data retention", "Bude", "Cheltenham", "petabyte storage", "10G", "data volume", "events", "unselected collection", "bulk collection", "telephony", "web browsing", "TDI", "RADIUS", "IP addresses", "IMEI comms-events", "emails", "HDFS", "HBASE", "Oracle", "RHEL", "Red Hat", "Apache", "private cloud", "QFD", "Skinny Cloud", "GCHQ Core", "MVR", "data ingest", "data partitioning", "correlation", "data mining", "Tor", "interactive query", "batch processing", "Near Real-Time Intelligence", "alerting", "contact chaining", "geolocation", "pattern-of-life", "behavioral analysis", "anomaly detection", "target discovery", "predictive analysis", "web traffic analysis", "search query monitoring", "cloned phone detection", "user identification", "converged communications", "data sharing" ], "document": { "creator": "GCHQ", "date_created": "2012-05-14", "classification": [ "TOP SECRET", "STRAP 1" ], "dissemination": [], "banner": "UK TOP SECRET STRAP 1", "genre": "slides", "programs": [ "EVOLVEDMUTANTBROTH", "MUTANTBROTH", "HARDASSOC", "HRMAP", "KARMAPOLICE", "SOCIALANTHROPOID", "AUTOASSOC", "GEOFUSION", "MVR", "TERRAIN", "SALAMANCA", "BLACKHOLE", "SAMUELPEPYS", "BROKER", "VAIL", "GUIDINGLIGHT", "FIRSTCONTACT", "CLOUDYCOBRA", "RUMOURMILL", "EVERYASSOC", "EVERYPOLICE", "EVERYCREATURE", "EVERYEAD", "EVERYCIPHER", "GREYFOX", "SILVERFOX", "FIRSTCONTACT", "CLOUDYCOBRA", "GUIDINGLIGHT", "GOLDENAXE", "TRIBALCARNEM", "PUBLICANEMONE", "EPICFAIL", "STERLINGMOTH", "FOGHORN", "LOOKINGGLASS", "FIREENGINE", "NGGEO", "ALPHACENTAURI", "CLOUDBASE", "ACCUMULO", "ASEM", "SEM", "HAKIM", "STREAMING", "HBASE", "LAUREL", "SKINNYCLOUD", "HARDY", "GCHQCORE" ], "countries": [ "Spain", "Romania", "Pakistan", "United Kingdom" ], "people": [] }, "publication": [ { "publisher": "The Intercept", "date_published": "2015-09-25", "title": "Profiled: From Radio to Porn, British Spies Track Web Users' Online Identities", "link": "https://theintercept.com/2015/09/25/gchq-radio-porn-spies-track-web-users-online-identities/", "reporters": [ "Gallagher, Ryan" ], "contributors": [], "doc_source": "https://theintercept.com/document/2015/09/25/gchq-analytic-cloud-challenges/", "doc_url": "https://s3.documentcloud.org/documents/2432286/gchq-analytic-cloud-challenges.pdf" } ], "pdf_sha256": "d2bbcb9cfb27c409423fab9095346fc8c2a1b5016ff76c1a2ddad84ce8766143", "pages": 23 }