snowchive-meta/data/add-sd-blackhole/add-sd-blackhole.md
2026-11-19 04:39:11 +02:00

8.2 KiB
Raw Permalink Blame History

TOP SECRET STRAP1

GCHQ GTAC

BLACK HOLE ANALYTICS

[REDACTED]!

ADD/SD briefing, September 2009

TOP SECRET STRAP1

This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ on [REDACTED]


TOP SECRET STRAP1

Contents

  • Describe the new breed of C2C tools developed in Applied Research (TR)
  • Scaling them up through the Next Generation Events (NGE) project

This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ O[REDACTED]


TOP SECRET STRAP1

Next Generation Events Roadmap

  • Convergence 1

    • ~8 x 10g
  • Cloud upscale

    • ~50 x 10g
  • Internet Profiling 2

    • ~200 x 10g
  • July 2009

  • Looking Glass deliveries

  • Looking Glass deliveries

  • Mar 2011

  • Internet Profiling 1

    • ~80 x 10g
    • ~260 experts
  • Capability development (including eA)

  • Large-scale contact chaining

  • Data mining and profiling

New technologies (particularly from JCE) incorporated into solution as they are de-risked / proven

This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]


TOP SECRET STRAP1

NGE Analytical Capabilities

Currently available on the desktop

  • Where?
  • When?
  • What?
  • With whom?

Future desktop capability

  • Who is similar?

  • Who is suspicious?

  • Who does this thing?

  • When should I worry?

  • inmarsat

  • Google

  • VISA

  • facebook

  • skype

  • Google

  • BRITISH AIRWAYS

This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ O[REDACTED]


TOP SECRET STRAP1

Some useful definitions

  • Presence Event

    • Describes that an identifier was online, on an IP address, at a point in time.
    • Simple, atomic element.
    • Can be very easily developed so are very diverse
    • Very useful!
  • TDI = Target Description Identifier

    • Has a type, eg Yahoo-Y-Cookie
    • And a value, eg tom123@yahoo
  • GTDI = Generic Target Detection Identifier

    • Expands the TDI concept to telephony e.g. A GSM location update message

This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ on [REDACTED]


TOP SECRET STRAP1

Some more definitions

  • BLACK HOLE: The large flat file storage where all the data sits
    • After initial processing, and before being manipulated and correlated and loaded into the QFD database tables
  • QFD: Question Focused Dataset (Database)
    • One for each tool: MUTANT BROTH, AUTO ASSOC etc

This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ O[REDACTED]


TOP SECRET STRAP1

Probes

  • DATA FEED
  • TDI Extraction
  • HTTP Extraction
  • VBulletin Extraction

Flat Storage BLACK HOLE

  • TDIs
  • GETs
  • VBulletin

Structured Storage QFDs

  • MUTANT BROTH
  • KARMA POLICE
  • INFINITE MONKEYS

This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]


TOP SECRET STRAP1

QFD desktop...so far...

  • Enables analysts to:
    • Create a profile of a target's online activities (Mutant Broth)
    • Find other identifiers for a target (Auto Assoc)
    • Create a social network (Social Animal)
    • Investigate websites or web forums of interest (Karma Police, Infinite Monkeys, HRMAP)
    • Find out who has been searching the web and for what (Memory Hole)
    • Find out who has been looking at what on Google Earth (Marbled Gecko)

This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ on [REDACTED]


TOP SECRET STRAP1

Converged QFD desktop...by Dec...

  • Enables analysts to:
    • Create a profile of a target's online activities alongside telephony (Evolved Mutant Broth)
    • Find alternative identifiers across telephony and the internet (Hard Assoc)
    • Create a social network including telephony (Evolved Social Animal)
    • Find out what has been happening in real time (Samuel Pepys)
  • All available on the desktop via Looking Glass plug-ins and Web GUIs

This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]


TOP SECRET STRAP1

Success story – TSS2 Nocon

  • Slides removed – contact [REDACTED] for full version

This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ on [REDACTED]


TOP SECRET STRAP1

TR Lab, Benhall
10g Bearers x 10-14
Data Gateway
TR BLACK HOLE
Test new QFDs
Processing Analytic Execution Environment (TR Lab, Benhall)
NGE BLACK HOLE (Benhall)
10g Bearers x 50
IP Prob
TELLURIAN
SSE
Other feeds
BUFFER (Benhall)
NGE BLACK HOLE (Benhall)
Bulk extract
NGE BLACK HOLE (Bude)
10g Bearers x 34
IP Prob
Bulk extract
TELLURIAN
Direct Access
SSE
Other feeds
BUFFER (Bude)
NGE BLACK HOLE (Bude)
Bulk extract
Processing Analytic Execution Environment (Ops Community, Benhall)
Operations
Bulk extract
Direct Access
BLACK HOLE User
Blazing Saddles

This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]


TOP SECRET STRAP1

Capability Dev Workspace

  • I've just developed collection of My FaceSpace events!

  • Is that better?

  • I'll put that processing on a few more bearers...

  • Collection engineer/research team

  • This data might be best viewed like this; take a look

  • Is that better?

  • I'll just add in some of those results...

  • User-interface specialist

  • Easy access to data, however it is collected

  • Use live data to work on real problems

  • Put the latest results in front of analysts

  • Processing power ready to use

  • Rapid feedback and improvement

  • Analyst with an operational problem

  • Oh heck, I've caught Bin Laden

  • Hmm, new data... I wonder what happens if I mine it like this...

  • Will this NSA algorithm help?

  • Oh, this looks interesting – what does an analyst think?

  • Data miner

  • Analytics developer

  • Or this?

  • OK, but could you just...

  • Good, but how about...

  • Could you tweak it to...

This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCH[REDACTED]


TOP SECRET STRAP1

Data Mining at scale

  • Experiments with Cloud ongoing to enable analysts to request and run MOAGs (large TNN graphs) from the desktop
  • Distillery will enable analysts to spot real time changes to the data at scale (e.g. detection of impossible travel)
  • Joint Collaboration Environment (Innov8) – trialling running of large scale analytics using both GCHQ and NSA data
Phone 1
Phone 2
Phone 3
Phone 4
16 14 12 10 8 6 4 2 0
0 10 15 20 25 30 35
Time (weeks)

This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCH[REDACTED]


TOP SECRET STRAP1

Future Implications

  • We shall be able to:
    • easily monitor changes in our targets' profiles and networks
    • develop and trial new capabilities using real life analytical experiments
    • respond quickly in a crisis

This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]


TOP SECRET STRAP1

Questions?

[REDACTED]/BLAZING_SADDLES [REDACTED]/NGE_BLACK_HOLE

This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation Refer any FOIA queries to GCHQ [REDACTED]