snowchive-meta/data/qfd-blackhole-technology-behind-inoc/qfd-blackhole-technology-behind-inoc.md
2026-11-19 04:39:11 +02:00

5.9 KiB
Raw Permalink Blame History

APPLIED RESEARCH

QFDs and BLACHOLE Technology behind GCHQ/INOC

[REDACTED] March 2009

© This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation. Refer any FOIA queries to GCHQ on [REDACTED].

APPLIED RESEARCH

Slide 1


UK SECRET STRAP2 COMINT ORCON

'BLACK HOLE'

Probes Flat Store Analytic engines

10G

Multi <10G

Question specific DBs (INOC today)

Data-mart extraction MESME/Netezza etc.

Clouds

© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.

APPLIED RESEARCH

Slide 2

UK SECRET STRAP2 COMINT ORCON


UK SECRET STRAP2 COMINT ORCON

BLACK HOLE

  • Flat data store – not a database
  • ~ 1,100,000,000,000 events since 6th August 2007
    • 53.3TB compressed
    • 217TB uncompressed
    • 47% of data is from last 3 months
  • ~10 Billion events per day
  • Cheap system:
    • £1k per TB of storage
    • £20k per 10Gbps probe
    • ~£400k for 10x10Gbps prototype

HTTP Websearch 19% SMTP 12% Webmail 3% CNE Server Log 5% Social networking 3% Instant Messenger 4% Routing Protocols 3% Anonymiser 2% Other 8%

© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Info e subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.

APPLIED RESEARCH

Slide 3

UK SECRET STRAP2 COMINT ORCON


UK SECRET STRAP2 COMINT ORCON

Questions – INOC Today

Who, Where, When, What? – Online presence (TDI) MUTANT BROTH

Who, which website(s)? – Online browsing activity KARMA POLICE

Who, which fora?– Bulletin board usage INFINITE MONKEYS

Where? – Google maps/earth usage MARBLED GECKO

© Crown Copyright reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under information legislation. Refer disclosure requests to GCHQ on [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.

APPLIED RESEARCH

Slide 4

UK SECRET STRAP2 COMINT ORCON


UK SECRET STRAP2 COMINT ORCON

Questions – INOC Today

Who, Where, When, What? – Online presence (TDI) MUTANT BROTH

Who, which website(s)? – Online browsing activity KARMA POLICE

Who, which fora?– Bulletin board usage INFINITE MONKEYS

Where? – Google maps/earth usage MARBLED GECKO

... And many more

© Crown Copyright reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under information legislation. Refer disclosure requests to GCHQ on [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.

APPLIED RESEARCH

Slide 5

UK SECRET STRAP2 COMINT ORCON


UK SECRET STRAP2 COMINT ORCON

Questions – INOC Today

Too many specific questions... Too much for analysts to learn.

Need analyst systems to be simple and intuitive.

Goo b13

Let analysts find & simply understand their targets easily...

drill in to systems only when they need to

© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.

APPLIED RESEARCH

Slide 6

UK SECRET STRAP2 COMINT ORCON


UK SECRET STRAP2 COMINT ORCON

AR Prototypes - EPR Stats - Feb 2008 - Jan 2009

End Product Reports contributed by: Grade C Grade B Grade A 70 60 50 40 30 20 10 0 MUTANT_BROTH INFINITE_MONKEYS GOOB 1- AUTOASSOC KARMA_POLICE PSOUP SOCIAL_ANIMAL SAMUEL_PEPYS MEMORY_HOLE HRMAP PSOUP_ALERT LAUGHING_HYENA ARBLED_GECKO

© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.

APPLIED RESEARCH

Slide 8

UK SECRET STRAP2 COMINT ORCON


UK SECRET STRAP2 COMINT ORCON

Number of users who executed > 10 queries, by area Number of users 140 120 100 80 60 40 20 0 OPCTIT OPICP OPIAP OPIMENA OPCCNE OPABUDE OPDGTAC OPDGNE OPASCAR OPDGTE OPITSI OPISC B13 OPIGSOC OPIRCIT OPINDIST B17 OPIBROK OPIDIG OPIW B13 B16 B18 OPAMHS OPCTJTR DIG OPCCDP OPCTDSD OPDCRFC OPDSDHQ OPDSMO OPICKX PICFABS Area

© Cr other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.

APPLIED RESEARCH

Slide 9

UK SECRET STRAP2 COMINT ORCON