303 lines
No EOL
8.2 KiB
Markdown
303 lines
No EOL
8.2 KiB
Markdown
TOP SECRET STRAP1
|
||
|
||
GCHQ
|
||
GTAC
|
||
|
||
# BLACK HOLE ANALYTICS
|
||
|
||
[REDACTED]!
|
||
|
||
ADD/SD briefing, September 2009
|
||
|
||
TOP SECRET STRAP1
|
||
|
||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ on [REDACTED]
|
||
|
||
---
|
||
|
||
TOP SECRET STRAP1
|
||
|
||
## Contents
|
||
|
||
* Describe the new breed of C2C tools developed in Applied Research (TR)
|
||
* Scaling them up through the Next Generation Events (NGE) project
|
||
|
||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ O[REDACTED]
|
||
|
||
---
|
||
|
||
TOP SECRET STRAP1
|
||
|
||
## Next Generation Events Roadmap
|
||
|
||
* Convergence 1
|
||
* ~8 x 10g
|
||
* Cloud upscale
|
||
* ~50 x 10g
|
||
* Internet Profiling 2
|
||
* ~200 x 10g
|
||
|
||
* July 2009
|
||
* Looking Glass deliveries
|
||
* Looking Glass deliveries
|
||
* Mar 2011
|
||
|
||
* Internet Profiling 1
|
||
* ~80 x 10g
|
||
* ~260 experts
|
||
* Capability development (including eA)
|
||
* Large-scale contact chaining
|
||
* Data mining and profiling
|
||
|
||
New technologies (particularly from JCE) incorporated into solution as they are de-risked / proven
|
||
|
||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]
|
||
|
||
---
|
||
|
||
TOP SECRET STRAP1
|
||
|
||
## NGE Analytical Capabilities
|
||
|
||
Currently available on the desktop
|
||
* Where?
|
||
* When?
|
||
* What?
|
||
* With whom?
|
||
|
||
Future desktop capability
|
||
* Who is similar?
|
||
* Who is suspicious?
|
||
* Who does this thing?
|
||
* When should I worry?
|
||
|
||
* inmarsat
|
||
* Google
|
||
* VISA
|
||
* facebook
|
||
* skype
|
||
* Google
|
||
* BRITISH AIRWAYS
|
||
|
||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ O[REDACTED]
|
||
|
||
---
|
||
|
||
TOP SECRET STRAP1
|
||
|
||
## Some useful definitions
|
||
|
||
* Presence Event
|
||
* Describes that an identifier was online, on an IP address, at a point in time.
|
||
* Simple, atomic element.
|
||
* Can be very easily developed so are very diverse
|
||
* Very useful!
|
||
|
||
* TDI = Target Description Identifier
|
||
* Has a type, eg Yahoo-Y-Cookie
|
||
* And a value, eg tom123@yahoo
|
||
|
||
* GTDI = Generic Target Detection Identifier
|
||
* Expands the TDI concept to telephony e.g. A GSM location update message
|
||
|
||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ on [REDACTED]
|
||
|
||
---
|
||
|
||
TOP SECRET STRAP1
|
||
|
||
## Some more definitions
|
||
|
||
* BLACK HOLE: The large flat file storage where all the data sits
|
||
* After initial processing, and before being manipulated and correlated and loaded into the QFD database tables
|
||
* QFD: Question Focused Dataset (Database)
|
||
* One for each tool: MUTANT BROTH, AUTO ASSOC etc
|
||
|
||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ O[REDACTED]
|
||
|
||
---
|
||
|
||
TOP SECRET STRAP1
|
||
|
||
### Probes
|
||
|
||
* DATA FEED
|
||
* TDI Extraction
|
||
* HTTP Extraction
|
||
* VBulletin Extraction
|
||
|
||
### Flat Storage BLACK HOLE
|
||
|
||
* TDIs
|
||
* GETs
|
||
* VBulletin
|
||
|
||
### Structured Storage QFDs
|
||
|
||
* MUTANT BROTH
|
||
* KARMA POLICE
|
||
* INFINITE MONKEYS
|
||
|
||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]
|
||
|
||
---
|
||
|
||
TOP SECRET STRAP1
|
||
|
||
## QFD desktop...so far...
|
||
|
||
* Enables analysts to:
|
||
* Create a profile of a target's online activities (Mutant Broth)
|
||
* Find other identifiers for a target (Auto Assoc)
|
||
* Create a social network (Social Animal)
|
||
* Investigate websites or web forums of interest (Karma Police, Infinite Monkeys, HRMAP)
|
||
* Find out who has been searching the web and for what (Memory Hole)
|
||
* Find out who has been looking at what on Google Earth (Marbled Gecko)
|
||
|
||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ on [REDACTED]
|
||
|
||
---
|
||
|
||
TOP SECRET STRAP1
|
||
|
||
## Converged QFD desktop...by Dec...
|
||
|
||
* Enables analysts to:
|
||
* Create a profile of a target's online activities alongside telephony (Evolved Mutant Broth)
|
||
* Find alternative identifiers across telephony and the internet (Hard Assoc)
|
||
* Create a social network including telephony (Evolved Social Animal)
|
||
* Find out what has been happening in real time (Samuel Pepys)
|
||
* All available on the desktop via Looking Glass plug-ins and Web GUIs
|
||
|
||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]
|
||
|
||
---
|
||
|
||
TOP SECRET STRAP1
|
||
|
||
## Success story – TSS2 Nocon
|
||
|
||
* Slides removed – contact [REDACTED] for full version
|
||
|
||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ on [REDACTED]
|
||
|
||
---
|
||
|
||
TOP SECRET STRAP1
|
||
|
||
```
|
||
TR Lab, Benhall
|
||
10g Bearers x 10-14
|
||
Data Gateway
|
||
TR BLACK HOLE
|
||
Test new QFDs
|
||
Processing Analytic Execution Environment (TR Lab, Benhall)
|
||
NGE BLACK HOLE (Benhall)
|
||
10g Bearers x 50
|
||
IP Prob
|
||
TELLURIAN
|
||
SSE
|
||
Other feeds
|
||
BUFFER (Benhall)
|
||
NGE BLACK HOLE (Benhall)
|
||
Bulk extract
|
||
NGE BLACK HOLE (Bude)
|
||
10g Bearers x 34
|
||
IP Prob
|
||
Bulk extract
|
||
TELLURIAN
|
||
Direct Access
|
||
SSE
|
||
Other feeds
|
||
BUFFER (Bude)
|
||
NGE BLACK HOLE (Bude)
|
||
Bulk extract
|
||
Processing Analytic Execution Environment (Ops Community, Benhall)
|
||
Operations
|
||
Bulk extract
|
||
Direct Access
|
||
BLACK HOLE User
|
||
Blazing Saddles
|
||
```
|
||
|
||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]
|
||
|
||
---
|
||
|
||
TOP SECRET STRAP1
|
||
|
||
## Capability Dev Workspace
|
||
|
||
* I've just developed collection of My FaceSpace events!
|
||
* Is that better?
|
||
* I'll put that processing on a few more bearers...
|
||
* Collection engineer/research team
|
||
|
||
* This data might be best viewed like this; take a look
|
||
* Is that better?
|
||
* I'll just add in some of those results...
|
||
* User-interface specialist
|
||
* Easy access to data, however it is collected
|
||
* Use live data to work on real problems
|
||
* Put the latest results in front of analysts
|
||
* Processing power ready to use
|
||
* Rapid feedback and improvement
|
||
* Analyst with an operational problem
|
||
* Oh heck, I've caught Bin Laden
|
||
* Hmm, new data... I wonder what happens if I mine it like this...
|
||
* Will this NSA algorithm help?
|
||
* Oh, this looks interesting – what does an analyst think?
|
||
* Data miner
|
||
* Analytics developer
|
||
* Or this?
|
||
* OK, but could you just...
|
||
* Good, but how about...
|
||
* Could you tweak it to...
|
||
|
||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCH[REDACTED]
|
||
|
||
---
|
||
|
||
TOP SECRET STRAP1
|
||
|
||
## Data Mining at scale
|
||
|
||
* Experiments with Cloud ongoing to enable analysts to request and run MOAGs (large TNN graphs) from the desktop
|
||
* Distillery will enable analysts to spot real time changes to the data at scale (e.g. detection of impossible travel)
|
||
* Joint Collaboration Environment (Innov8) – trialling running of large scale analytics using both GCHQ and NSA data
|
||
|
||
```
|
||
Phone 1
|
||
Phone 2
|
||
Phone 3
|
||
Phone 4
|
||
16 14 12 10 8 6 4 2 0
|
||
0 10 15 20 25 30 35
|
||
Time (weeks)
|
||
```
|
||
|
||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCH[REDACTED]
|
||
|
||
---
|
||
|
||
TOP SECRET STRAP1
|
||
|
||
## Future Implications
|
||
|
||
* We shall be able to:
|
||
* easily monitor changes in our targets' profiles and networks
|
||
* develop and trial new capabilities using real life analytical experiments
|
||
* respond quickly in a crisis
|
||
|
||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]
|
||
|
||
---
|
||
|
||
TOP SECRET STRAP1
|
||
|
||
## Questions?
|
||
|
||
[REDACTED]/BLAZING_SADDLES
|
||
[REDACTED]/NGE_BLACK_HOLE
|
||
|
||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED] |