ROCKRIDGE SECRET STRAP1 NEXT GENERATION events --- ## Scope and Aims Ingest more events feeds as new accesses come online Increase maturity and availability of QFDs Pull through more QFDs based on Ops priority Deliver QFDs capable of holding 'Convergence' data and wider event types ROUGH DIAMOND Provide a data mining and collaborative QFD development facility (BLACK HOLE - part of ROUGH DIAMOND) Enable sharing of QFD data with 2nd and 3rd Parties Interface with visualisation services in FIRE STORM SECRET STRAP1 NEXT GENERATION events --- ## What is a QFD? Designed to answer single analytic question (e.g. 'where is my target?') Simple table structure compared to traditional multi-function databases (e.g. HAUSTORIUM) Pioneered by ICTR, now developed by a community including Next Gen Events, ICTR, SD, GTE, ... Question Focused Database No specialised database technologies so simpler to develop and maintain Additional instances can easily be deployed at new locations or to increase capacity Smaller size and lower complexity means easier and quicker to develop and change SECRET STRAP1 NEXT GENERATION events --- ## What does each QFD answer? Mutant Broth * When was my target on line? * Where was my target on line? HRMap * What web pages was my target looking at before going to this dodgy website? Karma Police * What websites has my target visited? * Who's been visiting this dodgy websites? Social Animal * Who is my target interacting with on social networking sites? * What files have my target been uploading/downloading? GooBzs (QFD Query Federator) AutoAssoc * What alternative identifiers can I use to search for my target? Marbled Gecko * Who's been looking at this suspicious part of the world? * What part of the world has my target been looking at? Samuel Pepys (Coming soon!) * What is my target doing on-line right now?! Infinite Monkeys * Who's been posting (vBulletin boards) to this forum? * What posting (vBulletin boards) activity has my target been up to? Memory Hole * Who's been searching for these suspicious things on-line? * What has my target been searching for on-line? SECRET STRAP1 NEXT GENERATION events --- ## Ingest roadmap Internet Presence Events from 10G bearers eAD events SALAMANCA Telephony Events Initial Converged Events from TERRAINS MONOPOLY Special Source Events Broadband RADIUS events CDMA2000 test events Further 10G bearers at RPC1 (Bude) Feb 10 Mar 10 Apr 10 May 10 Jun 10 Jul 10 Aug 10 Trial part 1 - MUTANT BROTH, INFINITE MONKEYS, HRMAP, MEMORY HOLE from mobile tunnels Experiment Explore Deployed across CPC and RPC1 Trial part 2 - MMS, Blackberry, Google Maps, mobile Hotmail, mobile Gmail from mobile tunnels Explore Deployed across CPC and RPC1 TPS are working with the NGE Project and SMO Mobile theme to produce internet presence and application usage events from within mobile phone 'tunnels' in internet bearers. These will be trialed before full operational rollout Trial part 3 - Hotmail, Gmail, mail RU, Yahoo webmail from internet bearers Experiment Explore Deployed across CPC and RPC1 Trial part 4 - Windows Live IM, Yahoo Mail, SIP from internet bearers 'QFD style' events will also be produced for types of event traditionally fed into the older HAUSTORIUM and HARBOUR PILOT databases Explore Deployed across CPC and RPC1 SECRET STRAP1 NEXT GENERATION events --- ## Convergence QFDs Evolved MUTANT BROTH Search by geographical area Search Results Google Screenshots from evolved MUTANT BROTH web interface, and an export of it's data to Google Earth This major thread of work will: * Store events where internet applications are accessed from a mobile device * Allow analysts to relate mobile device identifiers to internet identifiers such as email addresses * Enable QFDs to store other more diverse event types, such as telephony events (currently SALAMANCA), and email events (currently HAUSTORIUM / HARBOUR PILOT) * Interface to LOOKING GLASS visualisation coming soon (in FIRE STORM work package) SECRET STRAP1 NEXT GENERATION events --- ## SAMUEL PEPYS QFD Purpose: Provide a near-real-time diarisation of any IP address Results GeoFusion reports IP address 90.237.[REDACTED] as MALMO (low confidence), SE (medlum confidence). | Date | Time (UTC) | Source | Destination | Type | Description | | :------- | :--------- | :---------------- | :------------- | :-------- | :---------- | | 02/02/10 | 00:08:01 | 90.237.[REDACTED] | 205.178.145.65 | Websearch | Visited cryptome.org/eyeball/gchq-eyeball.htm (after se | ``` Bearer GWUSC503 Connection TCP: 90.237[REDACTED] port 51475 to 205.178.145.65 port 80 Normalised query gchq cotswold Search Term gchq cotswold Search Host www.google.se Search-Clicked-Host cryptome.org Search-Clicked-URI /eyeball/gchq-eyeball.htm Accept-Language SV-SC User-Agent Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; WOW64; Trident/1.0; GTB6.1; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; eSobiSubscriber 2.0.4.16) Cookie nginx cached-1 Geo-IP-Src 55.60;13.00; MALMO;SE;BIVV Geo-IP-Dst 39.0062;-//-77.4288;STERLING;US;MMM ``` | Date | Time (UTC) | Source | Destination | Type | Description | | :------- | :--------- | :---------------- | :------------- | :-------- | :---------- | | 02/02/10 | 00:07:59 | 90.237.[REDACTED] | 205.178.145.65 | HTTP | GET cryptome.org/eyeball/gchq-eyeball.htm | | 01/02/10 | 23:41:12 | 90.237.[REDACTED] | 205.178.145.65 | HTTP | GET cryptome.org/site-r/site-r.htm | | 01/02/10 | 23:41:08 | 90.237.[REDACTED] | 205.178.145.65 | HTTP | GET eyeball-series.org/site-r/site-r.htm | | 01/02/10 | 23:38:28 | 90.237.[REDACTED] | 205.178.145.65 | Websearch | Visited cryptome.org/eyeball/siter-birdseye/siter-birdsey | | 01/02/10 | 23:38:26 | 90.237.[REDACTED] | 205.178.145.65 | HTTP | GET cryptome.org/eyeball/siter-birdseye/siter-birdeye.h | Expand all Collapse all Export CSV Export raw 1/1 6 Row(s) Prototyped by ICTR – Currently being pulled through by ROCK RIDGE, will be scaled to full 10G volumes by May 2010 SECRET STRAP1 NEXT GENERATION events --- ## BLACK HOLE What is BLACK HOLE? * A flat file store housing all data from a wide range of feeds (events and content) * Provides a set of tools for accessing that data. * Intended to be the source of events (and limited content) for the development of new QFDs and analytics. * Contains a rolling 6 months retention Part of ROUGH DIAMOND What does it enable? * New QFDs to be rapidly prototyped, then to be added to the operational QFD suite * Trialling of new bulk analysis ideas * New sources of data to be introduced quickly into existing QFDs. * Users to look for particular patterns and behaviours (target discovery) * TR, GTAC and GTE access to more data for research purposes, which may not be QFD related. SECRET STRAP1 NEXT GENERATION events --- ## User Feedback 'Absolutely FABULOUS well done ' (Iain Lobban, ref SUPERDRAKE reporting) 'its amazing to see how the pace of delivery in TDB has increased and I have been impressed by your responsiveness to customer needs. ' ([REDACTED], Senior User) 'Almost exactly a year ago I set you the challenge of delivering an upscaled massive events capability in order to support Internet Operations being conducted by GCHQ. Through your stripy team working on BLAZING SADDLES, BLUESHIFT and SUPPORTING INO you successfully met this challenge and delivered us a significant new capability in July.' ([REDACTED], Deputy Director Cyber Operations) 'It's working flawlessly' (analyst, ref BLACK HOLE) 'Bloody awesome' (analyst, ref SUPERDRAKE QFD) SECRET STRAP1 NEXT GENERATION events