first commit
This commit is contained in:
commit
0d90363116
6510 changed files with 679146 additions and 0 deletions
303
data/add-sd-blackhole/add-sd-blackhole.md
Normal file
303
data/add-sd-blackhole/add-sd-blackhole.md
Normal file
|
|
@ -0,0 +1,303 @@
|
|||
TOP SECRET STRAP1
|
||||
|
||||
GCHQ
|
||||
GTAC
|
||||
|
||||
# BLACK HOLE ANALYTICS
|
||||
|
||||
[REDACTED]!
|
||||
|
||||
ADD/SD briefing, September 2009
|
||||
|
||||
TOP SECRET STRAP1
|
||||
|
||||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ on [REDACTED]
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET STRAP1
|
||||
|
||||
## Contents
|
||||
|
||||
* Describe the new breed of C2C tools developed in Applied Research (TR)
|
||||
* Scaling them up through the Next Generation Events (NGE) project
|
||||
|
||||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ O[REDACTED]
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET STRAP1
|
||||
|
||||
## Next Generation Events Roadmap
|
||||
|
||||
* Convergence 1
|
||||
* ~8 x 10g
|
||||
* Cloud upscale
|
||||
* ~50 x 10g
|
||||
* Internet Profiling 2
|
||||
* ~200 x 10g
|
||||
|
||||
* July 2009
|
||||
* Looking Glass deliveries
|
||||
* Looking Glass deliveries
|
||||
* Mar 2011
|
||||
|
||||
* Internet Profiling 1
|
||||
* ~80 x 10g
|
||||
* ~260 experts
|
||||
* Capability development (including eA)
|
||||
* Large-scale contact chaining
|
||||
* Data mining and profiling
|
||||
|
||||
New technologies (particularly from JCE) incorporated into solution as they are de-risked / proven
|
||||
|
||||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET STRAP1
|
||||
|
||||
## NGE Analytical Capabilities
|
||||
|
||||
Currently available on the desktop
|
||||
* Where?
|
||||
* When?
|
||||
* What?
|
||||
* With whom?
|
||||
|
||||
Future desktop capability
|
||||
* Who is similar?
|
||||
* Who is suspicious?
|
||||
* Who does this thing?
|
||||
* When should I worry?
|
||||
|
||||
* inmarsat
|
||||
* Google
|
||||
* VISA
|
||||
* facebook
|
||||
* skype
|
||||
* Google
|
||||
* BRITISH AIRWAYS
|
||||
|
||||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ O[REDACTED]
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET STRAP1
|
||||
|
||||
## Some useful definitions
|
||||
|
||||
* Presence Event
|
||||
* Describes that an identifier was online, on an IP address, at a point in time.
|
||||
* Simple, atomic element.
|
||||
* Can be very easily developed so are very diverse
|
||||
* Very useful!
|
||||
|
||||
* TDI = Target Description Identifier
|
||||
* Has a type, eg Yahoo-Y-Cookie
|
||||
* And a value, eg tom123@yahoo
|
||||
|
||||
* GTDI = Generic Target Detection Identifier
|
||||
* Expands the TDI concept to telephony e.g. A GSM location update message
|
||||
|
||||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ on [REDACTED]
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET STRAP1
|
||||
|
||||
## Some more definitions
|
||||
|
||||
* BLACK HOLE: The large flat file storage where all the data sits
|
||||
* After initial processing, and before being manipulated and correlated and loaded into the QFD database tables
|
||||
* QFD: Question Focused Dataset (Database)
|
||||
* One for each tool: MUTANT BROTH, AUTO ASSOC etc
|
||||
|
||||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ O[REDACTED]
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET STRAP1
|
||||
|
||||
### Probes
|
||||
|
||||
* DATA FEED
|
||||
* TDI Extraction
|
||||
* HTTP Extraction
|
||||
* VBulletin Extraction
|
||||
|
||||
### Flat Storage BLACK HOLE
|
||||
|
||||
* TDIs
|
||||
* GETs
|
||||
* VBulletin
|
||||
|
||||
### Structured Storage QFDs
|
||||
|
||||
* MUTANT BROTH
|
||||
* KARMA POLICE
|
||||
* INFINITE MONKEYS
|
||||
|
||||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET STRAP1
|
||||
|
||||
## QFD desktop...so far...
|
||||
|
||||
* Enables analysts to:
|
||||
* Create a profile of a target's online activities (Mutant Broth)
|
||||
* Find other identifiers for a target (Auto Assoc)
|
||||
* Create a social network (Social Animal)
|
||||
* Investigate websites or web forums of interest (Karma Police, Infinite Monkeys, HRMAP)
|
||||
* Find out who has been searching the web and for what (Memory Hole)
|
||||
* Find out who has been looking at what on Google Earth (Marbled Gecko)
|
||||
|
||||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ on [REDACTED]
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET STRAP1
|
||||
|
||||
## Converged QFD desktop...by Dec...
|
||||
|
||||
* Enables analysts to:
|
||||
* Create a profile of a target's online activities alongside telephony (Evolved Mutant Broth)
|
||||
* Find alternative identifiers across telephony and the internet (Hard Assoc)
|
||||
* Create a social network including telephony (Evolved Social Animal)
|
||||
* Find out what has been happening in real time (Samuel Pepys)
|
||||
* All available on the desktop via Looking Glass plug-ins and Web GUIs
|
||||
|
||||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET STRAP1
|
||||
|
||||
## Success story – TSS2 Nocon
|
||||
|
||||
* Slides removed – contact [REDACTED] for full version
|
||||
|
||||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ on [REDACTED]
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET STRAP1
|
||||
|
||||
```
|
||||
TR Lab, Benhall
|
||||
10g Bearers x 10-14
|
||||
Data Gateway
|
||||
TR BLACK HOLE
|
||||
Test new QFDs
|
||||
Processing Analytic Execution Environment (TR Lab, Benhall)
|
||||
NGE BLACK HOLE (Benhall)
|
||||
10g Bearers x 50
|
||||
IP Prob
|
||||
TELLURIAN
|
||||
SSE
|
||||
Other feeds
|
||||
BUFFER (Benhall)
|
||||
NGE BLACK HOLE (Benhall)
|
||||
Bulk extract
|
||||
NGE BLACK HOLE (Bude)
|
||||
10g Bearers x 34
|
||||
IP Prob
|
||||
Bulk extract
|
||||
TELLURIAN
|
||||
Direct Access
|
||||
SSE
|
||||
Other feeds
|
||||
BUFFER (Bude)
|
||||
NGE BLACK HOLE (Bude)
|
||||
Bulk extract
|
||||
Processing Analytic Execution Environment (Ops Community, Benhall)
|
||||
Operations
|
||||
Bulk extract
|
||||
Direct Access
|
||||
BLACK HOLE User
|
||||
Blazing Saddles
|
||||
```
|
||||
|
||||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET STRAP1
|
||||
|
||||
## Capability Dev Workspace
|
||||
|
||||
* I've just developed collection of My FaceSpace events!
|
||||
* Is that better?
|
||||
* I'll put that processing on a few more bearers...
|
||||
* Collection engineer/research team
|
||||
|
||||
* This data might be best viewed like this; take a look
|
||||
* Is that better?
|
||||
* I'll just add in some of those results...
|
||||
* User-interface specialist
|
||||
* Easy access to data, however it is collected
|
||||
* Use live data to work on real problems
|
||||
* Put the latest results in front of analysts
|
||||
* Processing power ready to use
|
||||
* Rapid feedback and improvement
|
||||
* Analyst with an operational problem
|
||||
* Oh heck, I've caught Bin Laden
|
||||
* Hmm, new data... I wonder what happens if I mine it like this...
|
||||
* Will this NSA algorithm help?
|
||||
* Oh, this looks interesting – what does an analyst think?
|
||||
* Data miner
|
||||
* Analytics developer
|
||||
* Or this?
|
||||
* OK, but could you just...
|
||||
* Good, but how about...
|
||||
* Could you tweak it to...
|
||||
|
||||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCH[REDACTED]
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET STRAP1
|
||||
|
||||
## Data Mining at scale
|
||||
|
||||
* Experiments with Cloud ongoing to enable analysts to request and run MOAGs (large TNN graphs) from the desktop
|
||||
* Distillery will enable analysts to spot real time changes to the data at scale (e.g. detection of impossible travel)
|
||||
* Joint Collaboration Environment (Innov8) – trialling running of large scale analytics using both GCHQ and NSA data
|
||||
|
||||
```
|
||||
Phone 1
|
||||
Phone 2
|
||||
Phone 3
|
||||
Phone 4
|
||||
16 14 12 10 8 6 4 2 0
|
||||
0 10 15 20 25 30 35
|
||||
Time (weeks)
|
||||
```
|
||||
|
||||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCH[REDACTED]
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET STRAP1
|
||||
|
||||
## Future Implications
|
||||
|
||||
* We shall be able to:
|
||||
* easily monitor changes in our targets' profiles and networks
|
||||
* develop and trial new capabilities using real life analytical experiments
|
||||
* respond quickly in a crisis
|
||||
|
||||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET STRAP1
|
||||
|
||||
## Questions?
|
||||
|
||||
[REDACTED]/BLAZING_SADDLES
|
||||
[REDACTED]/NGE_BLACK_HOLE
|
||||
|
||||
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]
|
||||
Loading…
Add table
Add a link
Reference in a new issue