first commit

This commit is contained in:
libroot 2026-11-19 04:39:11 +02:00
commit 0d90363116
Signed by: libroot
GPG key ID: 2AC68C029EBE9ACE
6510 changed files with 679146 additions and 0 deletions

View file

@ -0,0 +1,103 @@
{
"title": "QFD BLACKHOLE Technology Behind INOC",
"type": "REPORTED",
"description": "A UK SECRET STRAP2 COMINT ORCON GCHQ Applied Research presentation from March 2009 documenting the BLACK HOLE bulk data repository's scale and structure, and the suite of analytical tools — MUTANT BROTH, KARMA POLICE, INFINITE MONKEYS and MARBLED GECKO — built on top of it to answer specific intelligence questions about online presence, browsing activity, forum usage and geolocation searches.",
"long_description": "This is a March 2009 GCHQ Applied Research presentation covering the technical architecture and analytical capabilities of the BLACK HOLE bulk data repository and its associated query tools, collectively known as INOC. The document provides the clearest contemporaneous statement of what BLACK HOLE was and how it related to the analytical tools built on top of it.\n\nBLACK HOLE is described not as a database but as a flat data store — a design choice that prioritised scale and ingestion speed over structured querying. By March 2009 it had accumulated over 1.1 trillion events since August 2007, representing 53.3 terabytes compressed or 217 terabytes uncompressed, with approximately 10 billion new events added daily. The cost figures are striking in their modesty: £1,000 per terabyte of storage, £20,000 per 10Gbps probe, with a full 10-probe prototype costing around £400,000. The data breakdown shows web browsing as the dominant content type at 41%, followed by web search (19%) and SMTP email (12%), with webmail, instant messaging, social networking, anonymiser traffic and CNE server logs making up the remainder.\n\nThe INOC section frames the analytical tools as answers to specific investigative questions: MUTANT BROTH answers who, where, when and what through online presence tracking via cookies; KARMA POLICE answers which websites a person has visited; INFINITE MONKEYS tracks bulletin board and forum usage; MARBLED GECKO tracks Google Maps and Google Earth searches. The presentation notes with some candour that the proliferation of specific tools was creating usability problems — \"too many specific questions, too much for analysts to learn\" — and that the goal was to make target discovery simple and intuitive enough that analysts only needed to drill into specialist systems when necessary.\n\nThe end product report statistics chart covering February 2008 to January 2009 shows MUTANT BROTH, INFINITE MONKEYS, and GOOB as the most heavily used tools by operational output. The user distribution chart breaking down active users by GCHQ operational division provides a rare glimpse into which parts of the organisation were making heaviest use of the bulk internet surveillance infrastructure. OPCTIT stands out as by far the dominant user with approximately 121 active analysts — more than four times the second-ranked division OPICP at around 25 — while the remaining divisions cluster below 20, most below 10.\n\nOne visually interesting element of the presentation is slide 6, which illustrates the usability aspiration with a modified Google logo where a black speech bubble containing \"b13\" obscures the \"gle\" portion of the word — rendering it \"Goob13.\" The black blob is likely a visual pun on BLACK HOLE itself, literally swallowing part of the Google logo — a self-aware joke about a system designed to ingest vast quantities of internet traffic, including Google searches and services, and make it searchable by analysts. The \"b13\" inside the bubble matches B13, one of the areas in the users-by-area chart on slide 9.",
"tags": [
"INOC",
"flat store",
"internet surveillance",
"web browsing",
"cookies",
"TDI",
"Applied Research",
"STRAP2",
"analytic tools",
"Computer Network Exploitation",
"anonymizers",
"probe infrastructure",
"data ingest",
"surveillance infrastructure",
"bulk collection",
"web browsing history",
"browsing history",
"online identifiers",
"cookie tracking",
"internet metadata",
"communications metadata",
"population-scale surveillance",
"passive collection",
"fiber tapping",
"cable tapping",
"data mining",
"pattern-of-life",
"online activity",
"search engines",
"geolocation",
"Google Maps",
"web forums",
"forum monitoring",
"social media monitoring",
"instant messaging",
"email interception",
"proxies",
"Tor",
"HTTP interception",
"target identification",
"data repository",
"petabyte storage",
"event logging",
"intelligence operations",
"intelligence analysis"
],
"document": {
"creator": "GCHQ",
"date_created": "2009-03",
"classification": [
"UK SECRET",
"STRAP 2",
"COMINT",
"ORCON"
],
"dissemination": [],
"banner": "UK SECRET STRAP2 COMINT ORCON",
"genre": "slides",
"programs": [
"BLACKHOLE",
"MESME",
"NETEZZA",
"CLOUDS",
"MUTANTBROTH",
"KARMAPOLICE",
"INFINITEMONKEYS",
"MARBLEDGECKO",
"GOOB",
"AUTOASSOC",
"PSOUP",
"SOCIALANIMAL",
"SAMUELPEPYS",
"MEMORYHOLE",
"HRMAP",
"PSOUPALERT",
"LAUGHINGHYENA"
],
"countries": [],
"people": []
},
"publication": [
{
"publisher": "The Intercept",
"date_published": "2015-09-25",
"title": "Profiled: From Radio to Porn, British Spies Track Web Users' Online Identities",
"link": "https://theintercept.com/2015/09/25/gchq-radio-porn-spies-track-web-users-online-identities/",
"reporters": [
"Gallagher, Ryan"
],
"contributors": [],
"doc_source": "https://theintercept.com/document/2015/09/25/qfd-blackhole-technology-behind-inoc/",
"doc_url": "https://s3.documentcloud.org/documents/2432297/qfd-blackhole-technology-behind-inoc.pdf"
}
],
"pdf_sha256": "ff3f63d4dee5a4ecbc49d8719d8e717d3460f9c023482c3a7139315ca09ba067",
"pages": 8
}