snowchive-meta/data/demystifying-nge-rock-ridge/demystifying-nge-rock-ridge.md
2026-11-19 04:39:11 +02:00

7.6 KiB
Raw Blame History

ROCKRIDGE

SECRET STRAP1

NEXT GENERATION events


Scope and Aims

Ingest more events feeds as new accesses come online

Increase maturity and availability of QFDs

Pull through more QFDs based on Ops priority

Deliver QFDs capable of holding 'Convergence' data and wider event types

ROUGH DIAMOND

Provide a data mining and collaborative QFD development facility (BLACK HOLE - part of ROUGH DIAMOND)

Enable sharing of QFD data with 2nd and 3rd Parties

Interface with visualisation services in FIRE STORM

SECRET STRAP1 NEXT GENERATION events


What is a QFD?

Designed to answer single analytic question (e.g. 'where is my target?')

Simple table structure compared to traditional multi-function databases (e.g. HAUSTORIUM)

Pioneered by ICTR, now developed by a community including Next Gen Events, ICTR, SD, GTE, ...

Question Focused Database

No specialised database technologies so simpler to develop and maintain

Additional instances can easily be deployed at new locations or to increase capacity

Smaller size and lower complexity means easier and quicker to develop and change

SECRET STRAP1 NEXT GENERATION events


What does each QFD answer?

Mutant Broth

  • When was my target on line?
  • Where was my target on line?

HRMap

  • What web pages was my target looking at before going to this dodgy website?

Karma Police

  • What websites has my target visited?
  • Who's been visiting this dodgy websites?

Social Animal

  • Who is my target interacting with on social networking sites?
  • What files have my target been uploading/downloading?

GooBzs (QFD Query Federator)

AutoAssoc

  • What alternative identifiers can I use to search for my target?

Marbled Gecko

  • Who's been looking at this suspicious part of the world?
  • What part of the world has my target been looking at?

Samuel Pepys (Coming soon!)

  • What is my target doing on-line right now?!

Infinite Monkeys

  • Who's been posting (vBulletin boards) to this forum?
  • What posting (vBulletin boards) activity has my target been up to?

Memory Hole

  • Who's been searching for these suspicious things on-line?
  • What has my target been searching for on-line?

SECRET STRAP1 NEXT GENERATION events


Ingest roadmap

Internet Presence Events from 10G bearers eAD events

SALAMANCA Telephony Events

Initial Converged Events from TERRAINS

MONOPOLY Special Source Events

Broadband RADIUS events

CDMA2000 test events

Further 10G bearers at RPC1 (Bude)

Feb 10 Mar 10 Apr 10 May 10 Jun 10 Jul 10 Aug 10

Trial part 1 - MUTANT BROTH, INFINITE MONKEYS, HRMAP, MEMORY HOLE from mobile tunnels

Experiment Explore Deployed across CPC and RPC1

Trial part 2 - MMS, Blackberry, Google Maps, mobile Hotmail, mobile Gmail from mobile tunnels

Explore Deployed across CPC and RPC1

TPS are working with the NGE Project and SMO Mobile theme to produce internet presence and application usage events from within mobile phone 'tunnels' in internet bearers. These will be trialed before full operational rollout

Trial part 3 - Hotmail, Gmail, mail RU, Yahoo webmail from internet bearers

Experiment Explore Deployed across CPC and RPC1

Trial part 4 - Windows Live IM, Yahoo Mail, SIP from internet bearers

'QFD style' events will also be produced for types of event traditionally fed into the older HAUSTORIUM and HARBOUR PILOT databases

Explore Deployed across CPC and RPC1

SECRET STRAP1

NEXT GENERATION events


Convergence QFDs

Evolved MUTANT BROTH Search by geographical area Search Results

Google

Screenshots from evolved MUTANT BROTH web interface, and an export of it's data to Google Earth

This major thread of work will:

  • Store events where internet applications are accessed from a mobile device
  • Allow analysts to relate mobile device identifiers to internet identifiers such as email addresses
  • Enable QFDs to store other more diverse event types, such as telephony events (currently SALAMANCA), and email events (currently HAUSTORIUM / HARBOUR PILOT)
  • Interface to LOOKING GLASS visualisation coming soon (in FIRE STORM work package)

SECRET STRAP1 NEXT GENERATION events


SAMUEL PEPYS QFD

Purpose: Provide a near-real-time diarisation of any IP address

Results GeoFusion reports IP address 90.237.[REDACTED] as MALMO (low confidence), SE (medlum confidence).

Date Time (UTC) Source Destination Type Description
02/02/10 00:08:01 90.237.[REDACTED] 205.178.145.65 Websearch Visited cryptome.org/eyeball/gchq-eyeball.htm (after se
Bearer              GWUSC503
Connection          TCP: 90.237[REDACTED] port 51475 to 205.178.145.65 port 80
Normalised query    gchq cotswold
Search Term         gchq cotswold
Search Host         www.google.se
Search-Clicked-Host cryptome.org
Search-Clicked-URI  /eyeball/gchq-eyeball.htm
Accept-Language     SV-SC
User-Agent          Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; WOW64; Trident/1.0; GTB6.1; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; eSobiSubscriber 2.0.4.16)
Cookie              nginx cached-1
Geo-IP-Src          55.60;13.00; MALMO;SE;BIVV
Geo-IP-Dst          39.0062;-//-77.4288;STERLING;US;MMM
Date Time (UTC) Source Destination Type Description
02/02/10 00:07:59 90.237.[REDACTED] 205.178.145.65 HTTP GET cryptome.org/eyeball/gchq-eyeball.htm
01/02/10 23:41:12 90.237.[REDACTED] 205.178.145.65 HTTP GET cryptome.org/site-r/site-r.htm
01/02/10 23:41:08 90.237.[REDACTED] 205.178.145.65 HTTP GET eyeball-series.org/site-r/site-r.htm
01/02/10 23:38:28 90.237.[REDACTED] 205.178.145.65 Websearch Visited cryptome.org/eyeball/siter-birdseye/siter-birdsey
01/02/10 23:38:26 90.237.[REDACTED] 205.178.145.65 HTTP GET cryptome.org/eyeball/siter-birdseye/siter-birdeye.h

Expand all Collapse all Export CSV Export raw 1/1 6 Row(s)

Prototyped by ICTR – Currently being pulled through by ROCK RIDGE, will be scaled to full 10G volumes by May 2010

SECRET STRAP1 NEXT GENERATION events


BLACK HOLE

What is BLACK HOLE?

  • A flat file store housing all data from a wide range of feeds (events and content)
  • Provides a set of tools for accessing that data.
  • Intended to be the source of events (and limited content) for the development of new QFDs and analytics.
  • Contains a rolling 6 months retention Part of ROUGH DIAMOND

What does it enable?

  • New QFDs to be rapidly prototyped, then to be added to the operational QFD suite
  • Trialling of new bulk analysis ideas
  • New sources of data to be introduced quickly into existing QFDs.
  • Users to look for particular patterns and behaviours (target discovery)
  • TR, GTAC and GTE access to more data for research purposes, which may not be QFD related.

SECRET STRAP1 NEXT GENERATION events


User Feedback

'Absolutely FABULOUS well done ' (Iain Lobban, ref SUPERDRAKE reporting)

'its amazing to see how the pace of delivery in TDB has increased and I have been impressed by your responsiveness to customer needs. ' ([REDACTED], Senior User)

'Almost exactly a year ago I set you the challenge of delivering an upscaled massive events capability in order to support Internet Operations being conducted by GCHQ. Through your stripy team working on BLAZING SADDLES, BLUESHIFT and SUPPORTING INO you successfully met this challenge and delivered us a significant new capability in July.' ([REDACTED], Deputy Director Cyber Operations)

'It's working flawlessly' (analyst, ref BLACK HOLE)

'Bloody awesome' (analyst, ref SUPERDRAKE QFD)

SECRET STRAP1 NEXT GENERATION events