snowchive-meta/data/qfd-blackhole-technology-behind-inoc/qfd-blackhole-technology-behind-inoc.md
2026-11-19 04:39:11 +02:00

260 lines
No EOL
5.9 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

APPLIED RESEARCH
# QFDs and BLACHOLE Technology behind GCHQ/INOC
[REDACTED]
March 2009
© This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation. Refer any FOIA queries to GCHQ on [REDACTED].
APPLIED RESEARCH
Slide 1
---
UK SECRET STRAP2 COMINT ORCON
## 'BLACK HOLE'
Probes
Flat Store
Analytic engines
10G
Multi
<10G
Question specific DBs (INOC today)
Data-mart extraction
MESME/Netezza etc.
Clouds
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.
APPLIED RESEARCH
Slide 2
UK SECRET STRAP2 COMINT ORCON
---
UK SECRET STRAP2 COMINT ORCON
## BLACK HOLE
* Flat data store – not a database
* ~ 1,100,000,000,000 events since 6th August 2007
* 53.3TB compressed
* 217TB uncompressed
* 47% of data is from last 3 months
* ~10 Billion events per day
* Cheap system:
* £1k per TB of storage
* £20k per 10Gbps probe
* ~£400k for 10x10Gbps prototype
HTTP
Websearch 19%
SMTP 12%
Webmail 3%
CNE Server Log 5%
Social networking 3%
Instant Messenger 4%
Routing Protocols 3%
Anonymiser 2%
Other 8%
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Info e subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.
APPLIED RESEARCH
Slide 3
UK SECRET STRAP2 COMINT ORCON
---
UK SECRET STRAP2 COMINT ORCON
## Questions – INOC Today
Who, Where, When, What? – Online presence (TDI)
MUTANT BROTH
Who, which website(s)? – Online browsing activity
KARMA POLICE
Who, which fora?– Bulletin board usage
INFINITE MONKEYS
Where? – Google maps/earth usage
MARBLED GECKO
© Crown Copyright reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under information legislation. Refer disclosure requests to GCHQ on [REDACTED]
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.
APPLIED RESEARCH
Slide 4
UK SECRET STRAP2 COMINT ORCON
---
UK SECRET STRAP2 COMINT ORCON
## Questions – INOC Today
Who, Where, When, What? – Online presence (TDI)
MUTANT BROTH
Who, which website(s)? – Online browsing activity
KARMA POLICE
Who, which fora?– Bulletin board usage
INFINITE MONKEYS
Where? – Google maps/earth usage
MARBLED GECKO
... And many more
© Crown Copyright reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under information legislation. Refer disclosure requests to GCHQ on [REDACTED]
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.
APPLIED RESEARCH
Slide 5
UK SECRET STRAP2 COMINT ORCON
---
UK SECRET STRAP2 COMINT ORCON
## Questions – INOC Today
Too many specific questions... Too much for analysts to learn.
Need analyst systems to be simple and intuitive.
Goo b13
Let analysts find & simply understand their targets easily...
drill in to systems only when they need to
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.
APPLIED RESEARCH
Slide 6
UK SECRET STRAP2 COMINT ORCON
---
UK SECRET STRAP2 COMINT ORCON
AR Prototypes - EPR Stats - Feb 2008 - Jan 2009
End Product Reports contributed by:
Grade C
Grade B
Grade A
70
60
50
40
30
20
10
0
MUTANT_BROTH
INFINITE_MONKEYS
GOOB
1- AUTOASSOC
KARMA_POLICE
PSOUP
SOCIAL_ANIMAL
SAMUEL_PEPYS
MEMORY_HOLE
HRMAP
PSOUP_ALERT
LAUGHING_HYENA
ARBLED_GECKO
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.
APPLIED RESEARCH
Slide 8
UK SECRET STRAP2 COMINT ORCON
---
UK SECRET STRAP2 COMINT ORCON
Number of users who executed > 10 queries, by area
Number of users
140
120
100
80
60
40
20
0
OPCTIT
OPICP
OPIAP
OPIMENA
OPCCNE
OPABUDE
OPDGTAC
OPDGNE
OPASCAR
OPDGTE
OPITSI
OPISC
B13
OPIGSOC
OPIRCIT
OPINDIST
B17
OPIBROK
OPIDIG
OPIW
B13
B16
B18
OPAMHS
OPCTJTR
DIG
OPCCDP
OPCTDSD
OPDCRFC
OPDSDHQ
OPDSMO
OPICKX
PICFABS
Area
© Cr other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.
APPLIED RESEARCH
Slide 9
UK SECRET STRAP2 COMINT ORCON