first commit

This commit is contained in:
libroot 2026-11-19 04:39:11 +02:00
commit 0d90363116
Signed by: libroot
GPG key ID: 2AC68C029EBE9ACE
6510 changed files with 679146 additions and 0 deletions

View file

@ -0,0 +1,303 @@
TOP SECRET STRAP1
GCHQ
GTAC
# BLACK HOLE ANALYTICS
[REDACTED]!
ADD/SD briefing, September 2009
TOP SECRET STRAP1
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ on [REDACTED]
---
TOP SECRET STRAP1
## Contents
* Describe the new breed of C2C tools developed in Applied Research (TR)
* Scaling them up through the Next Generation Events (NGE) project
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ O[REDACTED]
---
TOP SECRET STRAP1
## Next Generation Events Roadmap
* Convergence 1
* ~8 x 10g
* Cloud upscale
* ~50 x 10g
* Internet Profiling 2
* ~200 x 10g
* July 2009
* Looking Glass deliveries
* Looking Glass deliveries
* Mar 2011
* Internet Profiling 1
* ~80 x 10g
* ~260 experts
* Capability development (including eA)
* Large-scale contact chaining
* Data mining and profiling
New technologies (particularly from JCE) incorporated into solution as they are de-risked / proven
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]
---
TOP SECRET STRAP1
## NGE Analytical Capabilities
Currently available on the desktop
* Where?
* When?
* What?
* With whom?
Future desktop capability
* Who is similar?
* Who is suspicious?
* Who does this thing?
* When should I worry?
* inmarsat
* Google
* VISA
* facebook
* skype
* Google
* BRITISH AIRWAYS
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ O[REDACTED]
---
TOP SECRET STRAP1
## Some useful definitions
* Presence Event
* Describes that an identifier was online, on an IP address, at a point in time.
* Simple, atomic element.
* Can be very easily developed so are very diverse
* Very useful!
* TDI = Target Description Identifier
* Has a type, eg Yahoo-Y-Cookie
* And a value, eg tom123@yahoo
* GTDI = Generic Target Detection Identifier
* Expands the TDI concept to telephony e.g. A GSM location update message
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ on [REDACTED]
---
TOP SECRET STRAP1
## Some more definitions
* BLACK HOLE: The large flat file storage where all the data sits
* After initial processing, and before being manipulated and correlated and loaded into the QFD database tables
* QFD: Question Focused Dataset (Database)
* One for each tool: MUTANT BROTH, AUTO ASSOC etc
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ O[REDACTED]
---
TOP SECRET STRAP1
### Probes
* DATA FEED
* TDI Extraction
* HTTP Extraction
* VBulletin Extraction
### Flat Storage BLACK HOLE
* TDIs
* GETs
* VBulletin
### Structured Storage QFDs
* MUTANT BROTH
* KARMA POLICE
* INFINITE MONKEYS
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]
---
TOP SECRET STRAP1
## QFD desktop...so far...
* Enables analysts to:
* Create a profile of a target's online activities (Mutant Broth)
* Find other identifiers for a target (Auto Assoc)
* Create a social network (Social Animal)
* Investigate websites or web forums of interest (Karma Police, Infinite Monkeys, HRMAP)
* Find out who has been searching the web and for what (Memory Hole)
* Find out who has been looking at what on Google Earth (Marbled Gecko)
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ on [REDACTED]
---
TOP SECRET STRAP1
## Converged QFD desktop...by Dec...
* Enables analysts to:
* Create a profile of a target's online activities alongside telephony (Evolved Mutant Broth)
* Find alternative identifiers across telephony and the internet (Hard Assoc)
* Create a social network including telephony (Evolved Social Animal)
* Find out what has been happening in real time (Samuel Pepys)
* All available on the desktop via Looking Glass plug-ins and Web GUIs
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]
---
TOP SECRET STRAP1
## Success story – TSS2 Nocon
* Slides removed – contact [REDACTED] for full version
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ on [REDACTED]
---
TOP SECRET STRAP1
```
TR Lab, Benhall
10g Bearers x 10-14
Data Gateway
TR BLACK HOLE
Test new QFDs
Processing Analytic Execution Environment (TR Lab, Benhall)
NGE BLACK HOLE (Benhall)
10g Bearers x 50
IP Prob
TELLURIAN
SSE
Other feeds
BUFFER (Benhall)
NGE BLACK HOLE (Benhall)
Bulk extract
NGE BLACK HOLE (Bude)
10g Bearers x 34
IP Prob
Bulk extract
TELLURIAN
Direct Access
SSE
Other feeds
BUFFER (Bude)
NGE BLACK HOLE (Bude)
Bulk extract
Processing Analytic Execution Environment (Ops Community, Benhall)
Operations
Bulk extract
Direct Access
BLACK HOLE User
Blazing Saddles
```
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]
---
TOP SECRET STRAP1
## Capability Dev Workspace
* I've just developed collection of My FaceSpace events!
* Is that better?
* I'll put that processing on a few more bearers...
* Collection engineer/research team
* This data might be best viewed like this; take a look
* Is that better?
* I'll just add in some of those results...
* User-interface specialist
* Easy access to data, however it is collected
* Use live data to work on real problems
* Put the latest results in front of analysts
* Processing power ready to use
* Rapid feedback and improvement
* Analyst with an operational problem
* Oh heck, I've caught Bin Laden
* Hmm, new data... I wonder what happens if I mine it like this...
* Will this NSA algorithm help?
* Oh, this looks interesting – what does an analyst think?
* Data miner
* Analytics developer
* Or this?
* OK, but could you just...
* Good, but how about...
* Could you tweak it to...
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCH[REDACTED]
---
TOP SECRET STRAP1
## Data Mining at scale
* Experiments with Cloud ongoing to enable analysts to request and run MOAGs (large TNN graphs) from the desktop
* Distillery will enable analysts to spot real time changes to the data at scale (e.g. detection of impossible travel)
* Joint Collaboration Environment (Innov8) – trialling running of large scale analytics using both GCHQ and NSA data
```
Phone 1
Phone 2
Phone 3
Phone 4
16 14 12 10 8 6 4 2 0
0 10 15 20 25 30 35
Time (weeks)
```
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCH[REDACTED]
---
TOP SECRET STRAP1
## Future Implications
* We shall be able to:
* easily monitor changes in our targets' profiles and networks
* develop and trial new capabilities using real life analytical experiments
* respond quickly in a crisis
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]
---
TOP SECRET STRAP1
## Questions?
[REDACTED]/BLAZING_SADDLES
[REDACTED]/NGE_BLACK_HOLE
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]

View file

@ -0,0 +1,135 @@
{
"title": "BLACK HOLE ANALYTICS, ADD/SD briefing, September 2009",
"type": "REPORTED",
"description": "This September 2009 GCHQ briefing describes the BLACK HOLE analytics infrastructure and Next Generation Events (NGE) project, which planned to scale from ~8 10-gigabit connections to ~200 10Gs by March 2011, storing intercepted data in massive flat-file repositories at Benhall and Bude before loading it into \"Question Focused Dataset\" (QFD) databases that power surveillance tools like MUTANT BROTH, KARMA POLICE, INFINITE MONKEYS, and others. The document outlines how \"probes\" extract Target Detection Identifiers (TDIs) from intercepted data feeds, store them in BLACK HOLE repositories, then enable analysts to profile targets' online activities, create social networks, investigate websites and forums, track web searches and Google Earth usage, and perform large-scale data mining across internet and telephony data.",
"long_description": "This classified GCHQ briefing from September 2009 provides a comprehensive technical overview of the BLACK HOLE analytics system and the ambitious Next Generation Events (NGE) roadmap designed to dramatically scale the agency's bulk surveillance infrastructure. The presentation describes a multi-phase expansion plan: \"Convergence 1\" with approximately 8 10-gigabit connections, followed by \"Internet Profiling 1\" (~80 10Gs serving ~260 experts), \"Cloud upscale\" (~50 10Gs), and ultimately \"Internet Profiling 2\" reaching ~200 10-gigabit connections by March 2011. The architecture centers on \"probes\" that tap into data feeds and extract various types of information—TDI (Target Detection Identifier) extraction, HTTP extraction, and VBulletin forum extraction—which then flows into BLACK HOLE flat-file storage systems located at GCHQ facilities in Benhall and Bude. The document explains that BLACK HOLE serves as intermediate storage \"after initial processing, and before being manipulated and correlated and loaded into the QFD database tables,\" with each specialized surveillance tool having its own Question Focused Dataset (QFD). The briefing proudly lists the analytical capabilities available \"on the desktop,\" showing logos of targeted companies and services including Inmarsat, Google, VISA, Facebook, Skype, and British Airways, organized around questions like \"Where?\", \"When?\", \"What?\", and \"With whom?\"\n\nThe document reveals how GCHQ's suite of surveillance tools integrates with the BLACK HOLE infrastructure to enable comprehensive monitoring of internet and telephony activities. Existing capabilities include MUTANT BROTH (creating profiles of targets' online activities), AUTO ASSOC (finding other identifiers for targets), SOCIAL ANIMAL (creating social networks), KARMA POLICE (investigating websites), INFINITE MONKEYS (analyzing web forums), MEMORY HOLE (tracking web searches), HRMAP (website investigation), and MARBLED GECKO (monitoring Google Earth usage). The briefing announces upcoming \"converged\" capabilities that will merge internet and telephony surveillance, including \"Evolved Mutant Broth\" for combined online and phone activity profiles, \"Hard Assoc\" for finding alternative identifiers across both domains, \"Evolved Social Animal\" for social networks including phone data, and \"Samuel Pepys\" for real-time monitoring. The presentation emphasizes the development of a \"Capability Dev Workspace\" where collection engineers, data miners, analytics developers, and analysts can collaboratively experiment with live operational data, with one slide humorously suggesting an analyst might exclaim \"Oh heck, I've caught Bin Laden\" while another character jokes about developing \"My FaceSpace events\" collection. The briefing highlights plans for \"data mining at scale\" using cloud computing to run large TNN (presumably Telephony Network Node) graphs, a system called Distillery to detect real-time anomalies like \"impossible travel,\" and the Joint Collaboration Environment (Innov8) for running large-scale analytics using both GCHQ and NSA data. The document concludes by stating future implications will enable analysts to \"easily monitor changes in our targets' profiles and networks,\" \"develop and trial new capabilities using real life analytical experiments,\" and \"respond quickly in a crisis\"—essentially describing an unprecedented capability for population-scale surveillance and profiling of internet users worldwide, all operating under the UK's permissive legal framework with minimal judicial oversight.",
"tags": [
"Next Generation Events",
"Applied Research",
"Benhall",
"Bude",
"GCHQ",
"structured storage",
"QFD",
"database",
"data feeds",
"HTTP interception",
"vBulletin",
"web forums",
"forums",
"TDI",
"GTDI",
"Generic Target Detection Identifier",
"Presence events",
"online identifiers",
"IP addresses",
"data processing",
"target correlation",
"data correlation",
"correlation",
"surveillance tools",
"analytic tools",
"internet profiling",
"profiling",
"JCE",
"Joint Collaboration Environment",
"INMARSAT",
"Google",
"VISA",
"Facebook",
"Skype",
"British Airways",
"targeted services",
"analysis",
"point in time",
"cookies",
"web cookies",
"browser cookies",
"telephony",
"GSM",
"surveillance architecture",
"direct access",
"fiber tapping",
"fiber optic cables",
"telecommunications",
"humor",
"real life experiments",
"web browsing",
"browsing history",
"probing",
"tapping",
"international cables",
"emails",
"instant messaging",
"social media monitoring",
"unique identifier",
"personal information",
"usernames",
"email addresses",
"monitor usage",
"de-anonymization",
"linking identifiers",
"web browsing history",
"single identifier",
"digital footprint",
"online activity",
"queries",
"search engines",
"MSN",
"Messenger",
"Windows Live Messenger",
"contact chaining",
"Google Maps",
"Google Earth"
],
"document": {
"creator": "GCHQ",
"date_created": "2009-09-01",
"classification": [
"TOP SECRET",
"STRAP 1"
],
"dissemination": [],
"banner": "TOP SECRET STRAP1",
"genre": "slides",
"programs": [
"BLACKHOLE",
"MUTANTBROTH",
"AUTOASSOC",
"INFINITEMONKEYS",
"KARMAPOLICE",
"SOCIALANIMAL",
"HRMAP",
"MEMORYHOLE",
"MARBLEDGECKO",
"HARDASSOC",
"SAMUELPEPYS",
"BLAZINGSADDLES",
"TELLURIAN",
"BUFFER",
"DISTILLERY",
"LOOKINGGLASS",
"NGE"
],
"countries": [],
"people": [
"bin Laden, Osama"
]
},
"publication": [
{
"publisher": "The Intercept",
"date_published": "2015-09-25",
"title": "Profiled: From Radio to Porn, British Spies Track Web Users' Online Identities",
"link": "https://theintercept.com/2015/09/25/gchq-radio-porn-spies-track-web-users-online-identities/",
"reporters": [
"Gallagher, Ryan"
],
"contributors": [],
"doc_source": "https://theintercept.com/document/2015/09/25/add-sd-black-hole/",
"doc_url": "https://s3.documentcloud.org/documents/2432226/add-sd-blackhole.pdf"
}
],
"pdf_sha256": "55203ffc7941d2a4cbbd49fbf5f7b7246df4b0e1b84f0b4affb9dda155e25f59",
"pages": 15
}