APPLIED RESEARCH # QFDs and BLACHOLE Technology behind GCHQ/INOC [REDACTED] March 2009 © This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation. Refer any FOIA queries to GCHQ on [REDACTED]. APPLIED RESEARCH Slide 1 --- UK SECRET STRAP2 COMINT ORCON ## 'BLACK HOLE' Probes Flat Store Analytic engines 10G Multi <10G Question specific DBs (INOC today) Data-mart extraction MESME/Netezza etc. Clouds © Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation. APPLIED RESEARCH Slide 2 UK SECRET STRAP2 COMINT ORCON --- UK SECRET STRAP2 COMINT ORCON ## BLACK HOLE * Flat data store – not a database * ~ 1,100,000,000,000 events since 6th August 2007 * 53.3TB compressed * 217TB uncompressed * 47% of data is from last 3 months * ~10 Billion events per day * Cheap system: * £1k per TB of storage * £20k per 10Gbps probe * ~£400k for 10x10Gbps prototype HTTP Websearch 19% SMTP 12% Webmail 3% CNE Server Log 5% Social networking 3% Instant Messenger 4% Routing Protocols 3% Anonymiser 2% Other 8% © Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Info e subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation. APPLIED RESEARCH Slide 3 UK SECRET STRAP2 COMINT ORCON --- UK SECRET STRAP2 COMINT ORCON ## Questions – INOC Today Who, Where, When, What? – Online presence (TDI) MUTANT BROTH Who, which website(s)? – Online browsing activity KARMA POLICE Who, which fora?– Bulletin board usage INFINITE MONKEYS Where? – Google maps/earth usage MARBLED GECKO © Crown Copyright reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under information legislation. Refer disclosure requests to GCHQ on [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation. APPLIED RESEARCH Slide 4 UK SECRET STRAP2 COMINT ORCON --- UK SECRET STRAP2 COMINT ORCON ## Questions – INOC Today Who, Where, When, What? – Online presence (TDI) MUTANT BROTH Who, which website(s)? – Online browsing activity KARMA POLICE Who, which fora?– Bulletin board usage INFINITE MONKEYS Where? – Google maps/earth usage MARBLED GECKO ... And many more © Crown Copyright reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under information legislation. Refer disclosure requests to GCHQ on [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation. APPLIED RESEARCH Slide 5 UK SECRET STRAP2 COMINT ORCON --- UK SECRET STRAP2 COMINT ORCON ## Questions – INOC Today Too many specific questions... Too much for analysts to learn. Need analyst systems to be simple and intuitive. Goo b13 Let analysts find & simply understand their targets easily... drill in to systems only when they need to © Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation. APPLIED RESEARCH Slide 6 UK SECRET STRAP2 COMINT ORCON --- UK SECRET STRAP2 COMINT ORCON AR Prototypes - EPR Stats - Feb 2008 - Jan 2009 End Product Reports contributed by: Grade C Grade B Grade A 70 60 50 40 30 20 10 0 MUTANT_BROTH INFINITE_MONKEYS GOOB 1- AUTOASSOC KARMA_POLICE PSOUP SOCIAL_ANIMAL SAMUEL_PEPYS MEMORY_HOLE HRMAP PSOUP_ALERT LAUGHING_HYENA ARBLED_GECKO © Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation. APPLIED RESEARCH Slide 8 UK SECRET STRAP2 COMINT ORCON --- UK SECRET STRAP2 COMINT ORCON Number of users who executed > 10 queries, by area Number of users 140 120 100 80 60 40 20 0 OPCTIT OPICP OPIAP OPIMENA OPCCNE OPABUDE OPDGTAC OPDGNE OPASCAR OPDGTE OPITSI OPISC B13 OPIGSOC OPIRCIT OPINDIST B17 OPIBROK OPIDIG OPIW B13 B16 B18 OPAMHS OPCTJTR DIG OPCCDP OPCTDSD OPDCRFC OPDSDHQ OPDSMO OPICKX PICFABS Area © Cr other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation. APPLIED RESEARCH Slide 9 UK SECRET STRAP2 COMINT ORCON