first commit

This commit is contained in:
libroot 2026-11-19 04:39:11 +02:00
commit 0d90363116
Signed by: libroot
GPG key ID: 2AC68C029EBE9ACE
6510 changed files with 679146 additions and 0 deletions

View file

@ -0,0 +1,103 @@
{
"title": "QFD BLACKHOLE Technology Behind INOC",
"type": "REPORTED",
"description": "A UK SECRET STRAP2 COMINT ORCON GCHQ Applied Research presentation from March 2009 documenting the BLACK HOLE bulk data repository's scale and structure, and the suite of analytical tools — MUTANT BROTH, KARMA POLICE, INFINITE MONKEYS and MARBLED GECKO — built on top of it to answer specific intelligence questions about online presence, browsing activity, forum usage and geolocation searches.",
"long_description": "This is a March 2009 GCHQ Applied Research presentation covering the technical architecture and analytical capabilities of the BLACK HOLE bulk data repository and its associated query tools, collectively known as INOC. The document provides the clearest contemporaneous statement of what BLACK HOLE was and how it related to the analytical tools built on top of it.\n\nBLACK HOLE is described not as a database but as a flat data store — a design choice that prioritised scale and ingestion speed over structured querying. By March 2009 it had accumulated over 1.1 trillion events since August 2007, representing 53.3 terabytes compressed or 217 terabytes uncompressed, with approximately 10 billion new events added daily. The cost figures are striking in their modesty: £1,000 per terabyte of storage, £20,000 per 10Gbps probe, with a full 10-probe prototype costing around £400,000. The data breakdown shows web browsing as the dominant content type at 41%, followed by web search (19%) and SMTP email (12%), with webmail, instant messaging, social networking, anonymiser traffic and CNE server logs making up the remainder.\n\nThe INOC section frames the analytical tools as answers to specific investigative questions: MUTANT BROTH answers who, where, when and what through online presence tracking via cookies; KARMA POLICE answers which websites a person has visited; INFINITE MONKEYS tracks bulletin board and forum usage; MARBLED GECKO tracks Google Maps and Google Earth searches. The presentation notes with some candour that the proliferation of specific tools was creating usability problems — \"too many specific questions, too much for analysts to learn\" — and that the goal was to make target discovery simple and intuitive enough that analysts only needed to drill into specialist systems when necessary.\n\nThe end product report statistics chart covering February 2008 to January 2009 shows MUTANT BROTH, INFINITE MONKEYS, and GOOB as the most heavily used tools by operational output. The user distribution chart breaking down active users by GCHQ operational division provides a rare glimpse into which parts of the organisation were making heaviest use of the bulk internet surveillance infrastructure. OPCTIT stands out as by far the dominant user with approximately 121 active analysts — more than four times the second-ranked division OPICP at around 25 — while the remaining divisions cluster below 20, most below 10.\n\nOne visually interesting element of the presentation is slide 6, which illustrates the usability aspiration with a modified Google logo where a black speech bubble containing \"b13\" obscures the \"gle\" portion of the word — rendering it \"Goob13.\" The black blob is likely a visual pun on BLACK HOLE itself, literally swallowing part of the Google logo — a self-aware joke about a system designed to ingest vast quantities of internet traffic, including Google searches and services, and make it searchable by analysts. The \"b13\" inside the bubble matches B13, one of the areas in the users-by-area chart on slide 9.",
"tags": [
"INOC",
"flat store",
"internet surveillance",
"web browsing",
"cookies",
"TDI",
"Applied Research",
"STRAP2",
"analytic tools",
"Computer Network Exploitation",
"anonymizers",
"probe infrastructure",
"data ingest",
"surveillance infrastructure",
"bulk collection",
"web browsing history",
"browsing history",
"online identifiers",
"cookie tracking",
"internet metadata",
"communications metadata",
"population-scale surveillance",
"passive collection",
"fiber tapping",
"cable tapping",
"data mining",
"pattern-of-life",
"online activity",
"search engines",
"geolocation",
"Google Maps",
"web forums",
"forum monitoring",
"social media monitoring",
"instant messaging",
"email interception",
"proxies",
"Tor",
"HTTP interception",
"target identification",
"data repository",
"petabyte storage",
"event logging",
"intelligence operations",
"intelligence analysis"
],
"document": {
"creator": "GCHQ",
"date_created": "2009-03",
"classification": [
"UK SECRET",
"STRAP 2",
"COMINT",
"ORCON"
],
"dissemination": [],
"banner": "UK SECRET STRAP2 COMINT ORCON",
"genre": "slides",
"programs": [
"BLACKHOLE",
"MESME",
"NETEZZA",
"CLOUDS",
"MUTANTBROTH",
"KARMAPOLICE",
"INFINITEMONKEYS",
"MARBLEDGECKO",
"GOOB",
"AUTOASSOC",
"PSOUP",
"SOCIALANIMAL",
"SAMUELPEPYS",
"MEMORYHOLE",
"HRMAP",
"PSOUPALERT",
"LAUGHINGHYENA"
],
"countries": [],
"people": []
},
"publication": [
{
"publisher": "The Intercept",
"date_published": "2015-09-25",
"title": "Profiled: From Radio to Porn, British Spies Track Web Users' Online Identities",
"link": "https://theintercept.com/2015/09/25/gchq-radio-porn-spies-track-web-users-online-identities/",
"reporters": [
"Gallagher, Ryan"
],
"contributors": [],
"doc_source": "https://theintercept.com/document/2015/09/25/qfd-blackhole-technology-behind-inoc/",
"doc_url": "https://s3.documentcloud.org/documents/2432297/qfd-blackhole-technology-behind-inoc.pdf"
}
],
"pdf_sha256": "ff3f63d4dee5a4ecbc49d8719d8e717d3460f9c023482c3a7139315ca09ba067",
"pages": 8
}

View file

@ -0,0 +1,260 @@
APPLIED RESEARCH
# QFDs and BLACHOLE Technology behind GCHQ/INOC
[REDACTED]
March 2009
© This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation. Refer any FOIA queries to GCHQ on [REDACTED].
APPLIED RESEARCH
Slide 1
---
UK SECRET STRAP2 COMINT ORCON
## 'BLACK HOLE'
Probes
Flat Store
Analytic engines
10G
Multi
<10G
Question specific DBs (INOC today)
Data-mart extraction
MESME/Netezza etc.
Clouds
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.
APPLIED RESEARCH
Slide 2
UK SECRET STRAP2 COMINT ORCON
---
UK SECRET STRAP2 COMINT ORCON
## BLACK HOLE
* Flat data store – not a database
* ~ 1,100,000,000,000 events since 6th August 2007
* 53.3TB compressed
* 217TB uncompressed
* 47% of data is from last 3 months
* ~10 Billion events per day
* Cheap system:
* £1k per TB of storage
* £20k per 10Gbps probe
* ~£400k for 10x10Gbps prototype
HTTP
Websearch 19%
SMTP 12%
Webmail 3%
CNE Server Log 5%
Social networking 3%
Instant Messenger 4%
Routing Protocols 3%
Anonymiser 2%
Other 8%
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Info e subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.
APPLIED RESEARCH
Slide 3
UK SECRET STRAP2 COMINT ORCON
---
UK SECRET STRAP2 COMINT ORCON
## Questions – INOC Today
Who, Where, When, What? – Online presence (TDI)
MUTANT BROTH
Who, which website(s)? – Online browsing activity
KARMA POLICE
Who, which fora?– Bulletin board usage
INFINITE MONKEYS
Where? – Google maps/earth usage
MARBLED GECKO
© Crown Copyright reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under information legislation. Refer disclosure requests to GCHQ on [REDACTED]
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.
APPLIED RESEARCH
Slide 4
UK SECRET STRAP2 COMINT ORCON
---
UK SECRET STRAP2 COMINT ORCON
## Questions – INOC Today
Who, Where, When, What? – Online presence (TDI)
MUTANT BROTH
Who, which website(s)? – Online browsing activity
KARMA POLICE
Who, which fora?– Bulletin board usage
INFINITE MONKEYS
Where? – Google maps/earth usage
MARBLED GECKO
... And many more
© Crown Copyright reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under information legislation. Refer disclosure requests to GCHQ on [REDACTED]
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.
APPLIED RESEARCH
Slide 5
UK SECRET STRAP2 COMINT ORCON
---
UK SECRET STRAP2 COMINT ORCON
## Questions – INOC Today
Too many specific questions... Too much for analysts to learn.
Need analyst systems to be simple and intuitive.
Goo b13
Let analysts find & simply understand their targets easily...
drill in to systems only when they need to
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.
APPLIED RESEARCH
Slide 6
UK SECRET STRAP2 COMINT ORCON
---
UK SECRET STRAP2 COMINT ORCON
AR Prototypes - EPR Stats - Feb 2008 - Jan 2009
End Product Reports contributed by:
Grade C
Grade B
Grade A
70
60
50
40
30
20
10
0
MUTANT_BROTH
INFINITE_MONKEYS
GOOB
1- AUTOASSOC
KARMA_POLICE
PSOUP
SOCIAL_ANIMAL
SAMUEL_PEPYS
MEMORY_HOLE
HRMAP
PSOUP_ALERT
LAUGHING_HYENA
ARBLED_GECKO
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.
APPLIED RESEARCH
Slide 8
UK SECRET STRAP2 COMINT ORCON
---
UK SECRET STRAP2 COMINT ORCON
Number of users who executed > 10 queries, by area
Number of users
140
120
100
80
60
40
20
0
OPCTIT
OPICP
OPIAP
OPIMENA
OPCCNE
OPABUDE
OPDGTAC
OPDGNE
OPASCAR
OPDGTE
OPITSI
OPISC
B13
OPIGSOC
OPIRCIT
OPINDIST
B17
OPIBROK
OPIDIG
OPIW
B13
B16
B18
OPAMHS
OPCTJTR
DIG
OPCCDP
OPCTDSD
OPDCRFC
OPDSDHQ
OPDSMO
OPICKX
PICFABS
Area
© Cr other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.
APPLIED RESEARCH
Slide 9
UK SECRET STRAP2 COMINT ORCON