1143 lines
No EOL
29 KiB
Markdown
1143 lines
No EOL
29 KiB
Markdown
# events
|
||
|
||
# Events Product Centre
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## Agenda
|
||
|
||
* Welcome
|
||
* Immingle
|
||
* Salamanca
|
||
* QFDs
|
||
* Guiding Light
|
||
* Questions
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## IMMINGLE
|
||
|
||
Key changes July 2010 to present:
|
||
|
||
* Inferred data from B3M now flagged
|
||
* Updates to handle identifiers from HARD ASSOC and B3M correctly
|
||
* MAINWAY: MSRNs now grouped and flagged in same way as SALAMANCA
|
||
* MAINWAY: direct access to event details provided
|
||
* GPRS flagging – THUGGEE rules applied to SALAMANCA events
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## IMMINGLE
|
||
|
||
```
|
||
Event Summary (First Circle) Query
|
||
|
||
Query Name IMM 14-10-2010 103234
|
||
|
||
Legalities Justification
|
||
MIRANDA 20140
|
||
HRA Justification Dev testing
|
||
JIC Priority 7NS
|
||
|
||
Date Range
|
||
Start Date 1 Month Ago 14/09/2010 00:00:00
|
||
End Date Today 14/10/2010 23:59:59
|
||
|
||
Event Sources
|
||
SALAMANCA
|
||
B3M
|
||
HARD ASSOC
|
||
FASCIA
|
||
SAMDYCE
|
||
HAUSTORIUM
|
||
CULTWEAVE
|
||
MAINWAY II
|
||
Event details
|
||
Precomputed Pairs
|
||
Used to find external SEED identifiers alternatives will be the supplied seed
|
||
|
||
MAINWAY options and Help pages
|
||
|
||
FREEFORM MERLOT Add Term
|
||
Data File Location | Source | Custom Template
|
||
Add file | Remove Selected | Remove All
|
||
|
||
Selector Parameters
|
||
Before using C2C for event analysis, please read this section of the user guide
|
||
|
||
Seed Entry
|
||
Add All ->
|
||
<- Return Type
|
||
<- Return Selected
|
||
<- Return All
|
||
List of seeds to be queried
|
||
[REDACTED] phone
|
||
[REDACTED] phone
|
||
[REDACTED] phone
|
||
[REDACTED] phone
|
||
[REDACTED] phone
|
||
|
||
PHONE IMSI IMEI FHEX
|
||
C2C Machine
|
||
Remove All
|
||
|
||
Enrichment Sources
|
||
BROAD OAK
|
||
CONTRAOCTAVE
|
||
CRYING FOWL
|
||
Select All | Deselect All
|
||
|
||
Visualisation Options
|
||
Display Associations
|
||
Perform Dedupe
|
||
Flag Seed identifier
|
||
Include Unknown Identifiers
|
||
Include inferred identifiers
|
||
Summary Count for High Rates Report 1000
|
||
|
||
C2C Only
|
||
These options apply to HAUSTORIUM only
|
||
Relationships: Strong All
|
||
DO ACTIVITY SUMMARY
|
||
Maximum Event Participant Count 200
|
||
|
||
Query Cancel
|
||
```
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## IMMINGLE
|
||
|
||
GPRS flag and count
|
||
|
||
```
|
||
MALAYSIA :26 MAINWAY
|
||
AKABIA
|
||
MALAYSIA
|
||
07/10/2010 06:57 5
|
||
11
|
||
07/11/2010 05-14/09/2010 07:51 14
|
||
19/09/2010-08:20
|
||
MAINW 02/10/2010 03:37 SALAMANCA
|
||
29/09/2010 01:21
|
||
MAINWAY II: 8, FASCIA: 151 FASCIA: 3WAY II: 10
|
||
FASCIA: 24
|
||
ASCIA: 21
|
||
30123/09/2010 14:19:39
|
||
MAINWAY II: 1, FASCIA: 3
|
||
FASCIA: 11 1/4, FASCIA. 111
|
||
60122674959. 26 MAINWAY II: 4, FASCIA:
|
||
FASCIA: 0 0 11:06
|
||
18/09/2010 07:2012
|
||
FASCIA: 19
|
||
02/10/2010 11:263/2010 03:34 6
|
||
MAINWAY II: 17, FASCIA: 33
|
||
|
||
GPRS Events: 8
|
||
SAUDI ARABIA
|
||
SEED IDENTIFIER
|
||
|
||
MAINWAY II, FASCIA: 9
|
||
01/10/2010 10:3710 12:52
|
||
01/10/2010 10:40 15,4ASCIA: 1
|
||
MAINWAY II: 8, SALAMANCA: B, F
|
||
19/09/20 10 12:15
|
||
MAINWAY II: 3, SALAMANCA: 3
|
||
```
|
||
|
||
| | | |
|
||
|---------------------- |------|-------------------------------------|
|
||
| Datasource | Text | MAINWAY II, SALAMANCA, OCT... |
|
||
| BROAD OAK Is N... | Text | No |
|
||
| GPRS Last Seen | Text | 26-09-2010 07:43:02 |
|
||
| GPRS First Seen | Text | 24-09-2010 06:31:07 |
|
||
| BROAD OAK Desi... | Text | [REDACTED] |
|
||
| BROAD OAK Sher... | Text | 1 |
|
||
| BROAD ОАК Неа... | Text | Saudi Aramco SVP Special Assign... |
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## IMMINGLE
|
||
|
||
What next?
|
||
|
||
* FASCIA GPRS flagging
|
||
* HAUSTORIUM decommissioning
|
||
* Next Gen Contact Chaining trial....
|
||
|
||
NEXT GENERATION
|
||
events
|
||
|
||
FIRE STORM
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## BRIO and SALAMANCA
|
||
|
||
Key changes since July:
|
||
|
||
* NRT (Near Real Time) Storage = 3 days
|
||
* Extra feeds from TERRAINs at BUDE and SOUNDER
|
||
* 2nd Party usage of SALAMANCA: SHAREOWN replaces ESCHAR
|
||
* CallAnsweredState and CallEndState added to TERRAIN-SALAMANCA feed
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## BRIO/SALAMANCA (cont.)
|
||
|
||
Pakistan NGN inferencing errors
|
||
|
||
```
|
||
GCQuery (Operational Server) - Microsoft Internet Explorer
|
||
File Edit View Favorites Tools Help
|
||
Back Search Favorites
|
||
Address: [REDACTED]
|
||
|
||
GCQuery (Operational Server)
|
||
HELP
|
||
|
||
HPSu 8.3
|
||
|
||
Outage Status
|
||
There are no planned outages.
|
||
|
||
WARNING - Incorrect data in SALAMANCA!
|
||
Inferred events on Source FDDG-QY, SIGAD UKC-3071 between 18 June 2010 and 19 October 2010 should not be used.
|
||
More Information
|
||
|
||
Problems?
|
||
Please report all problems or queries via the ITS2.
|
||
|
||
Need an account?
|
||
You can get a GCQuery account via the ITSO's Online Forms:
|
||
* SALAMANCA ARTEMIS GREEN
|
||
* All Others
|
||
|
||
re-update. The Presidential Britain above Feb 2000 and may be
|
||
rmation Regulation. Refer-updates are requests to GCHQ at
|
||
|
||
[REDACTED] Last Updated: 7th Mar 2010
|
||
```
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## QFDs
|
||
|
||
Blazing Saddles
|
||
|
||
[REDACTED]
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## Scaling
|
||
|
||
Blazing Saddles
|
||
|
||
Current scale
|
||
|
||
* There are 100 unique bearers feeding the BzS tools.
|
||
* Consistently averaging over 30 billion events per-day into the input buffer.
|
||
* MB is loading over 10.5 billion
|
||
* 6 months data retention for MB = 1,890,000,000,000 records and requires 400 TB.
|
||
* Total storage of over one petabyte.
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## Scaling
|
||
|
||
Blazing Saddles
|
||
|
||
Future Scale
|
||
|
||
* Further 58 bearers by end of 2010
|
||
* An additional 40 bearers in Q1 2011.
|
||
* MB will ingest over 20 billion events per day requiring one petabyte of storage.
|
||
* Overall storage will increase to 2.5 petabytes.
|
||
* Scope scaling to 400 bearers.
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## TDI listing
|
||
|
||
```
|
||
HR Ma
|
||
INFINITE MON
|
||
KARMA POLIC
|
||
MARBLED GEC
|
||
MEMORY HOL
|
||
MUTANT BROT
|
||
SAMUEL PEPY
|
||
SOCIAL ANIMAL
|
||
SOCIAL ANTHROPOID user guide
|
||
TDI Database login using Corporate Directory password
|
||
|
||
TDI listing
|
||
|
||
Browse by technology
|
||
Rows 15 Go
|
||
View Chart
|
||
|
||
Or, type in the TDI type you are interested in
|
||
TDI type Yahoo-Y-Cookie Go
|
||
|
||
Technologies associated with Yahoo-Y-Cookie
|
||
Tech Description
|
||
Yahoo Yahoo provides various free web services including webmail, a web portal, a web directory, news, and mapping.
|
||
1-1
|
||
|
||
Yahoo-Y-Cookie details
|
||
|
||
TDI Name Yahoo-Y-Cookie
|
||
Type TDI
|
||
Scope User
|
||
Description This is the username of the Yahoo! user who is logged in. The username Is the first bit of the Yahoo! e-mail address (everything before the "@" if the user is on a primary domain-or the full email address if the user is on a secondary domain (see <a href=[REDACTED]>here</a> for more info on primary/secondary domains). In raw traffic, the username appears as part of the "Y" cookie string, with "I" before it and "/o" after it, obfuscated using a simple substitution code called ROT13. The QFDs automatically break out the relevant part of the cookie and turn it back into plain text for use as a TDI. If a Yahoo-Y-Cookie is seen with user agent Mozilla/4.0 (compatible; MSIE 5.5...) then that is probably the Yahoo! instant messaging client rather than a web browser.
|
||
Further info If the information above is blank, incomplete, or unhelpful, please email OTE Tech Tracking and demand answers.
|
||
|
||
BEGAL rules generating Yahoo-Y-Cookie events
|
||
Name ID
|
||
EXP_Yahoo-Y-Cookie_0 11113
|
||
1-1
|
||
|
||
[Full Feed]
|
||
Subscribe to the blog feed on the GCWeb dynamic home page.
|
||
```
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## Samuel Pepys
|
||
|
||
Blazing Saddles
|
||
|
||
Pull through and upscaling of TR SPs.
|
||
|
||
* Currently 43 bearers.
|
||
* 14 from TR SP
|
||
* 29 additional bearers from TPS (generating HTTP, TDI, Websearch, FTP and Squeal).
|
||
* Circa 40 additional bearers just generating Squeal.
|
||
* Approval to increase aperture to 100 bearers for all data-types.
|
||
* Approval to increase user numbers to 200.
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
Blazing Saddles
|
||
|
||
## SOCIAL ANTHROPOID
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
Blazing Saddles
|
||
|
||
What is Social Anthropoid?
|
||
|
||
* SOCIAL ANTHROPOID is a converged comms database. It will allow you to see when your targets have communicated via phone, over the internet, or using converged channels (e.g., sending e-mails from a phone or making voice calls over the internet).
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
Blazing Saddles
|
||
|
||
What about the existing comms data-bases?
|
||
|
||
* When SOCIAL ANTHROPOID contains all the necessary data and has all the core functionality of the legacy tools Social animal, HAUSTORIUM and SALAMANCA will be de-commissioned.
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
Blazing Saddles
|
||
|
||
What data is in Social Anthropoid??
|
||
|
||
* All of Salamanca data (telephony)
|
||
* Social animal data.
|
||
* Instant Messenger.
|
||
* Webmail. - SIP & H323 VOIP
|
||
* Yahoo Voice
|
||
* Blackberry
|
||
* MMS
|
||
* SMS (from Salamanca and other sources)
|
||
* GTP (GPRS session set-ups)
|
||
* And more..
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
Blazing Saddles
|
||
|
||
What about SMTP, POP3 and IMAP?
|
||
|
||
* Starting to receive these data types now.
|
||
* Capability deployed as part of HeartBeat 11.
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
Blazing Saddles
|
||
|
||
```
|
||
Query
|
||
Saved queries
|
||
You have 1 saved queries.
|
||
Test:
|
||
|
||
* Queries will be automatically submitted to all instances of SOCIAL ANTHROPOID, SOCIAL ANIMAL and Converge SOCIAL ANIMAL.
|
||
* For bulk queries, enter multiple selectors (one per line).
|
||
* If allow wildcards is ticked, it is treated as a multi-character wildcard (e.g., pauls* ill match paul123, paul456, paul*@yahoo.com will match paul123@yahoo.com but not paul123@hotmail.com). Unlike other QFDs, * and \ have no special meaning (to query for a literal string, uncheck allow wildcards rather than 'escaping' the wildcard.
|
||
* By default, results will be returned in which your input selector appears in either the User A or User B column (in SOCIAL ANIMAL terms is the 'actor' or the 'subject' with in the event). To return results in which your selector appears only as the active user, tick the Query active users only checkbox.
|
||
* Front-end processing normalises C2C selectors in various ways, including the removal of dots from the usernames of Email addresses. To get Gmail results, you will need to normalise your queries in the same way (e.g., search for badguy@gmail.com instead of bad.guy@gmail.com). Gmail itself ignores the dots so there is no danger of getting events for the wrong account. If in doubt, consult your local C2C tech ex.
|
||
|
||
Miranda 2014C
|
||
JIC Priority & Purpose 1NS
|
||
HRA Justification testing
|
||
|
||
Search period (optional): to
|
||
Filter results by matched selectors prior to display
|
||
Allow wildcards
|
||
Query active users only
|
||
|
||
Save Query
|
||
Submit query
|
||
```
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
C2C
|
||
|
||
Blazing Saddles
|
||
|
||
```
|
||
Query Input | Results [ REDACTED] x | Results [REDACTED] | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x
|
||
|
||
User A role
|
||
User A type
|
||
User A
|
||
User A row value
|
||
User A display name
|
||
User B type
|
||
User B
|
||
User B row value
|
||
User B display name
|
||
|
||
Filter criteria
|
||
To filter the result set choose some criteria
|
||
|
||
28Oct-
|
||
Active e
|
||
Action:
|
||
:ha:
|
||
message
|
||
Locato
|
||
Source
|
||
More
|
||
24-Oct-
|
||
Active e
|
||
Action:
|
||
:ha:
|
||
message
|
||
Locat
|
||
Source
|
||
More
|
||
24-Oct-
|
||
Active e
|
||
Action:
|
||
:ha:
|
||
message
|
||
Locat
|
||
Source
|
||
more
|
||
09-Oct-2010 10:26:12 – chat event (windowslive), 2 selectors
|
||
Page 1 of 1 First Previous 1 Next Last (1) Change Sort Order Display Summary CSV Export
|
||
Temporal filtering
|
||
TDI filtering
|
||
Participant count filtering
|
||
```
|
||
|
||
Results summary
|
||
| Action | Action Type | User A Role | User A | User A Type | User B Role | User B | User B Type | First Seen | Last Seen | Count |
|
||
|--------|-------------|-------------|--------|-------------|-------------|--------|-------------|------------|-----------|-------|
|
||
| chat | message | chat message | [REDACTED]@hotmail.com | email | | [REDACTED]@linuxmail.org | email | 02-Jul-2010 14:01:19 | 02-Jul-2010 14:01:19 | 1 |
|
||
| chat | message | chat message | [REDACTED]@hotmail.com | Sender | Recipient | [REDACTED]@linuxmail.org | Unknown | 23-Aug-2010 18:21:49 | 23-Aug-2010 18:23:13 | 2 |
|
||
| chat | message | chat message | [REDACTED]@linuxmail.org | Sender | Recipient | [REDACTED]@hotmail.com | Unknown | 24-Oct-2010 15:34:48 | 24-Oct-2010 15:34:48 | 1 |
|
||
| chat | message | chat message | [REDACTED]@linuxmail.org | Sender | Recipient | [REDACTED]@hotmail.com | Unknown | 24-Oct-2010 15:57:43 | 28-Oct-2010 04:00:23 | 2 |
|
||
| chat | message | chat message | [REDACTED]@hotmail.fr | email | | [REDACTED]@linuxmail.org | email | 12-Jun-2010 18:16:14 | 12-Jun-2010 18:16:33 | 2 |
|
||
| chat | message | chat message | [REDACTED]@hotmail.it | Sender | Recipient | [REDACTED]@linuxmail.org | Unknown | 09-Oct-2010 18:26:08 | 09-Oct-2010 18:26:12 | 2 |
|
||
| chat | message | chat message | [REDACTED]@linuxmail.org | email | | [REDACTED]@linuxmail.org | email | 23-Aug-2010 18:40:56 | 23-Aug-2010 18:41:08 | 6 |
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
Telephony in Santhropoid
|
||
|
||
Blazing Saddles
|
||
|
||
```
|
||
Query Input | Results [ REDACTED] x | Results [REDACTED] | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x
|
||
|
||
User A role
|
||
User A type
|
||
User A
|
||
User A row value
|
||
User A display name
|
||
User B type
|
||
User B
|
||
User B row value
|
||
User B display name
|
||
|
||
03-Nov-2010 17:34:20 - telephony event (global), 2 selectors, duration: 00:00:06
|
||
Active user: [REDACTED](tel_number)
|
||
Action: unknown Action type: call
|
||
unknown call tel_number [REDACTED] dialed tel_number [REDACTED]
|
||
Locators:
|
||
Source Point-Code: 60741 Destination Point-Code: 20082
|
||
More
|
||
|
||
03-Nov-2010 17:34:19 - telephony event (global), 2 selectors, duration: 00:00:06
|
||
Active user: [REDACTED]el_number)
|
||
Action: unknown Action type: call
|
||
unknown call tel_number [REDACTED] dialed tel_number [REDACTED]
|
||
Locators:
|
||
Source Point-Code: 60741 Destination Point-Code: 20082
|
||
More
|
||
|
||
01-Nov-2010 17:17:39 telephony event (global), 2 selectors, duration: 00:00:05
|
||
Active user: [REDACTED](tel_number)
|
||
Action: unknown Action type: call
|
||
unknown call tel_number [REDACTED] dialed tel_number [REDACTED]
|
||
Locators:
|
||
Source Point Code: 60741 Destination Point Code: 20082
|
||
More
|
||
```
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
Convergence - GTP tunnel
|
||
|
||
Blazing Saddles
|
||
|
||
```
|
||
Query Input | Results [ REDACTED] x | Results [REDACTED] | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x
|
||
|
||
User A role
|
||
User A type
|
||
User A
|
||
User A row value
|
||
User A display name
|
||
User B type
|
||
User B
|
||
User B row value
|
||
User B display name
|
||
|
||
01-Nov-2010 07:16:06 - tunnel event (gtp), 3 selectors
|
||
Active user: [REDACTED] (imsi) [REDACTED] (tel_number) [REDACTED] (blackberry_mid_pin)
|
||
Action: create Action type: tunnel
|
||
create tunnel imsi [REDACTED] not_available not_available
|
||
create tunnel tel_number [REDACTED] not_available not_available
|
||
create tunnel blackberry_mid_pin [REDACTED] not_available not_available
|
||
Locators:
|
||
Source IPV4: [REDACTED] Source SgsnAddress: [REDACTED] Destination IPV4: [REDACTED]
|
||
GTP Convergence Data:
|
||
This event represents the creation of GTP tunnel [REDACTED]
|
||
View all events from this GTP tunnel
|
||
More
|
||
|
||
01-Nov-2010 04:20:10 - tunnel event (gtp), 3 selectors
|
||
Active user: [REDACTED] imsi) [REDACTED] tel_number) [REDACTED] blackberry_mid_pin)
|
||
Action: create Action type: tunnel
|
||
create tunnel imsi [REDACTED] not_available not_available
|
||
create tunnel tel_number [REDACTED] not_available not_available
|
||
create tunnel blackberry_mid_pin [REDACTED] not_available not_available
|
||
Locators:
|
||
Source IPV4: [REDACTED] Source SgsnAddress: [REDACTED] Destination IPV4: [REDACTED]
|
||
GTP Convergence Data:
|
||
This event represents the creation of GTP tunnel [REDACTED]
|
||
View all events from this GTP tunnel
|
||
```
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
Convergence - Leaky Gateways
|
||
|
||
Blazing Saddles
|
||
|
||
```
|
||
Query Input | Results [ REDACTED] x | Results [REDACTED] | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x
|
||
|
||
User A role
|
||
User A type
|
||
User A
|
||
User A row value
|
||
User A display name
|
||
User B type
|
||
User B
|
||
User B row value
|
||
User B display name
|
||
|
||
24-Oct-2010 06:11:57 - webmail event (yahoo), 5 selectors
|
||
Active user: [REDACTED] (UserId) [REDACTED] AccountOwner [REDACTED] (MachineId) [REDACTED] (X-huawei-IMSI)
|
||
Action: send Action type: message
|
||
From Unknown [REDACTED]@yahoo.com From [REDACTED] Unknown [REDACTED]@yahoo.com
|
||
Locators:
|
||
Source IPV4: [REDACTED] Destination IPV4: [REDACTED]
|
||
More
|
||
|
||
24-Oct-2010 06:11:57 webmail event (yahoo), 5 selectors
|
||
Active user: [REDACTED] (UserId) [REDACTED] AccountOwner [REDACTED] (MachineId) [REDACTED] (X-huawei-IMSI)
|
||
Action: send Action type: message
|
||
From Unknown [REDACTED]@yahoo.com From Unknown [REDACTED]@yahoo.com
|
||
Locators:
|
||
Source IPV4: [REDACTED] Destination IPV4: [REDACTED]
|
||
More
|
||
|
||
24-Oct-2010 06:10:16 - webmail event (yahoo), 5 selectors
|
||
Active user: [REDACTED] (UserId) [REDACTED] AccountOwner [REDACTED] (MachineId) [REDACTED] (X-huawei-IMSI)
|
||
Action: send Action type: message
|
||
From Unknown [REDACTED]@yahoo.com From Unknown [REDACTED]
|
||
Locators:
|
||
Source IPV4: [REDACTED] Destination IPV4: [REDACTED]
|
||
More
|
||
|
||
24-Oct-2010 06:10:16 webmail event (yahoo), 5 selectors
|
||
Active user: [REDACTED] (UserId) [REDACTED] AccountOwner [REDACTED] (MachineId) [REDACTED] (X-huawei-IMSI)
|
||
```
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
OSN
|
||
|
||
Blazing Saddles
|
||
|
||
```
|
||
Query Input | Results [ REDACTED] x | Results [REDACTED] | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x
|
||
|
||
User A role
|
||
User A type
|
||
User A
|
||
User A row value
|
||
User A display name
|
||
User B type
|
||
User B
|
||
User B row value
|
||
User B display name
|
||
|
||
15-Jul-2010 21:41:42 - SOCIAL ANIMAL event (Facebook), 2 selectors
|
||
Active user: [REDACTED] (Facebook-uid)
|
||
Action: chat Action type: message
|
||
chat message Facebook-uid [REDACTED] Facebook-uid [REDACTED]
|
||
Locators:
|
||
Source IPV4: [REDACTED]
|
||
More
|
||
|
||
15-Jul-2010 21:41:42 - SOCIAL ANIMAL event (Facebook), 2 selectors
|
||
Active user: [REDACTED] (Facebook-uid)
|
||
Action: alias Action type: user
|
||
alias user Facebook-uid [REDACTED] email [REDACTED]
|
||
Locators:
|
||
More
|
||
|
||
15-Jul-2010 21:41:37 - SOCIAL ANIMAL event (Facebook), 2 selectors
|
||
Active user: [REDACTED] (Facebook-uid)
|
||
Action: chat Action type: message
|
||
chat message Facebook-uid [REDACTED] Facebook-uid [REDACTED]
|
||
Locators:
|
||
Source IPV4: [REDACTED]
|
||
More
|
||
|
||
15-Jul-2010 21:41:33 - SOCIAL ANIMAL event (Facebook), 2 selectors
|
||
Active user: [REDACTED] (Facebook-uid)
|
||
Action: chat Action type: message
|
||
```
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
Blazing Saddles
|
||
|
||
Looks good, When can I have an account?
|
||
|
||
* Santhropoid is currently in the second stage of UAT.
|
||
* We currently have 200 users representing all areas of the business.
|
||
* Aiming to be in a position to release Santhropoid to the masses in early January.
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## New data sources
|
||
|
||
Blazing Saddles
|
||
|
||
* LUSTRE – new data-source available in MB. Good for North Africa.
|
||
* Source field – This will enable new non-routine data-sources to be added to the QFD's.
|
||
* CNE
|
||
* JTRIG – GLASSBACK data used for test case.
|
||
* COLLATERAL
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## New Loaders
|
||
|
||
Blazing Saddles
|
||
|
||
New loaders deployed to MB and HR Map, improvements to KP.
|
||
|
||
* Latency of the data in the QFDs has been greatly reduced, now around 12 hours.
|
||
* Each instance of MB can now ingest 8 billion events per-day (total 32 billion)
|
||
* Some QFDs were previously 1-5 days behind.
|
||
* Query performance during loading has also been improved.
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## GUIDING LIGHT QFD
|
||
|
||
Presented by [REDACTED]
|
||
(Guiding Light SU)
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## What is GUIDING LIGHT?
|
||
|
||
* New QFD developed in August 2010 by TDB-Events.
|
||
* Primary objective:
|
||
|
||
"To understand the traffic seen on the Next Gen Events bearers."
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## What can it do for me?
|
||
|
||
General Questions:
|
||
|
||
* Given a case notation, what are the TDI types that are found on it?
|
||
* Given a TDI type/subset, which bearers produce the highest number of events?
|
||
* What type of traffic is on which bearers and where is it coming from?
|
||
* Which bearers provide the most amount of traffic type x from place y?
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## Front End Interface
|
||
|
||
```
|
||
GUIDING LIGHT
|
||
|
||
From Date To Date
|
||
Custom 01-NOV-2010 Custom 01-NOV-2010
|
||
|
||
Bearer e.g. GWUKC151%
|
||
Event Type e.g. %facebook%
|
||
|
||
Country Digraphs (using ISO standard)
|
||
|
||
Query Type
|
||
From A to B
|
||
|
||
Country A
|
||
Country B
|
||
Min Event Count
|
||
|
||
Note: The % wildcard character represents 0 or more characters.
|
||
|
||
Daily Counts Source Types Bearers Countries Event Types Full Profile
|
||
```
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## Results - Full Profile Query
|
||
|
||
```
|
||
Results
|
||
|
||
Sections
|
||
Parameters
|
||
Results
|
||
Event Types
|
||
Bearers
|
||
Countries (From)
|
||
Countries (To)
|
||
Event Type, Country
|
||
Country, Bearer
|
||
Bearer, Event Type
|
||
Daily Counts
|
||
Bearer Counts
|
||
Bearer Pivot
|
||
EventType Count
|
||
EventType Pivot
|
||
Country Counts
|
||
Country Pivot
|
||
SourceType Count
|
||
Source Type Pivot
|
||
```
|
||
|
||
| # | Source Type | Source Site | Descriptor | Event Type | Origination Country | Destination Country | Event Count |
|
||
|---|-------------|-------------|------------|------------|---------------------|---------------------|-------------|
|
||
| 1 | vol | Cheltenham | EREPO | WEB_CSS | | | 2,613,571 |
|
||
| 2 | s3c_presence | Cheltenham | EREPO | Yahoo-B-Cookie | US | FK | 1,817,599 |
|
||
| 3 | s3c_presence | Cheltenham | EREPO | Yahoo-Y-Cookie | US | FK | 1,002,176 |
|
||
| 4 | s3c_presence | Cheltenham | EREPO | Yahoo-B-Cookie | US | IR | 1,281,351 |
|
||
| 5 | s3c_presence | Cheltenham | EREPO | Yahoo-Y-Cookie | US | IR | 82,175 |
|
||
| 6 | host_referer | Cheltenham | LADPO | HOST_AL_URL_S | | FK | 25,236 |
|
||
| 7 | s3c_presence | Cheltenham | EREPO | SIP-INVITE-From | PK | FK | 212,000 |
|
||
| 8 | s3c_presence | Cheltenham | FSTDO | SIP-INVITE-From | US | FK | 197,752 |
|
||
| 9 | | Cheltenham | EREPO | YdJS | PK | US | 167,713 |
|
||
| 10 | s3c_presence | Cheltenham | EREPO | Yahoo Messenger | US | FK | 167,520 |
|
||
| 11 | | Cheltenham | EREPO | YdJS | US | FK | 162,533 |
|
||
| 12 | s3c_presence | Cheltenham | EREPO | SIP-INVITE-From | GB | FK | 145,339 |
|
||
| 13 | host_referer | Cheltenham | EREPO | HOST_REFERER | | IR | 145,000 |
|
||
| 14 | s3c_presence | Cheltenham | EREPO | Yahoo-Messenger | PK | US | 132,238 |
|
||
| 15 | s3c_s | Cheltenham | EREPO | s p | US | FK | 112,434 |
|
||
| 16 | s3c_presence | Cheltenham | LADPO | SIP-INVITE-From | PK | US | 91,239 |
|
||
| 17 | s3c_presence | Cheltenham | EREPO | Yahoo-D-Cookie | US | XX | 92,217 |
|
||
| 18 | s3c_s | Cheltenham | FSTDO | s p | C | FK | 73,711 |
|
||
| 19 | s3c_presence | Cheltenham | EREPO | SIP-INVITE-From | PK | DE | 91,772 |
|
||
| 20 | s3c_presence | Cheltenham | EREPO | SIP-INVITE-From | CH | FK | 82,173 |
|
||
| 21 | s3c_presence | Cheltenham | EREPO | Yahoo-B-Cookie | US | II | 76,272 |
|
||
| 22 | s3c_presence | Cheltenham | LADPO | Yahoo-Y-Cookie | US | XX | 67,226 |
|
||
| 23 | s3c_s | Cheltenham | EREPO | s p | PK | FK | 52,330 |
|
||
| 24 | s3c_presence | Cheltenham | FSTDO | Google-DPFC-Cookie | PK | FK | 54,716 |
|
||
| 25 | s3c_presence | Cheltenham | EREPO | Yahoo Y C | US | II | 43,336 |
|
||
| 26 | s3c_presence | Cheltenham | EREPO | Yahoo B Cookie | US | GB | 43,771 |
|
||
| 27 | s3c_presence | Cheltenham | EREPO | Yahoo-Y-Cookie | US | GB | 41,333 |
|
||
| 28 | search | Cheltenham | EREPO | www.google.com.pk | PK | FK | 62,333 |
|
||
| 29 | s3c_presence | Cheltenham | EREPO | Simbar-SIMBAR-User-Agent | US | IR | 32,252 |
|
||
| 30 | s3c_presence | Cheltenham | EREPO | SIP-INVITE-From | M | FK | 34,232 |
|
||
| 31 | search | Cheltenham | LADPO | www.google.com.pk | | FK | 24,234 |
|
||
| 32 | s3c_presence | Cheltenham | EREPO | DXT_Shopreports-CRC_T-User-Agent | US | FK | 21,100 |
|
||
| 33 | s3c_s | Cheltenham | FSTDO | s p | PK | US | 25,715 |
|
||
| 34 | search | Cheltenham | EREPO | www.jungle.com | US | IP | 25,127 |
|
||
| 35 | | Cheltenham | EREPO | s l | CH | FK | 94,434 |
|
||
| 36 | s3c_s | Cheltenham | EREPO | s l | PK | DE | 31,257 |
|
||
| 37 | s3c_presence | Cheltenham | EREPO | SIP-INVITE-From | FR | FK | 12,376 |
|
||
| 38 | s3c_presence | Cheltenham | EREPO | Facebook User-Cookie | US | FK | 12,246 |
|
||
| 39 | host_referer | Cheltenham | EREPO | HOST_REFERER | | XX | 12,449 |
|
||
| 40 | s3c_presence | Cheltenham | LADPO | Simbar-SIMBAR-User-Agent | US | FK | 14,432 |
|
||
| 41 | host_referer | Cheltenham | EREPO | HOST_REFERER | | II | 12,104 |
|
||
| 42 | s3c_s | Cheltenham | FSTDO | s p | NY | FK | 11,740 |
|
||
| 43 | s3c_presence | Cheltenham | EREPO | Yahoo S P REGISTER From | US | II | 11,510 |
|
||
| 44 | search | Cheltenham | EREPO | clinilab.udum.or.uk | PK | FK | 11,558 |
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## Results Pivot: Event Types
|
||
|
||
```
|
||
Sections
|
||
Parameters
|
||
Results
|
||
Event Types
|
||
Bearers
|
||
Countries (From)
|
||
Countries (To)
|
||
Event Type, Country
|
||
Country, Bearer
|
||
Bearer, Event Type
|
||
Daily Counts
|
||
Bearer Counts
|
||
Bearer Pivot
|
||
EventType Count
|
||
EventType Pivot
|
||
Country Counts
|
||
Country Pivot
|
||
SourceType Count
|
||
Source Type Pivot
|
||
```
|
||
|
||
| Event Type | Event Count |
|
||
|-------------------------------------|-------------|
|
||
| Yahoo-B-Cookie | 3,370,673 |
|
||
| Yahoo-Y-Cookie | 2,682,201 |
|
||
| WEBLOG | 2,519,571 |
|
||
| SIP-INVITE-From | 874,568 |
|
||
| HOST_REFERER | 439,609 |
|
||
| YMSG | 364,400 |
|
||
| sip | 345,876 |
|
||
| Yahoo-Messenger | 337,035 |
|
||
| www.google.com.pk | 72,419 |
|
||
| Google-PREFID-Cookie | 71,337 |
|
||
| Simbar-SIMBAR-User-Agent | 58,653 |
|
||
| www.google.com | 38,747 |
|
||
| EXP_Shopperreports-SRS_IT-User-Agent| 33,847 |
|
||
| Facebook-c_user-Cookie | 21,664 |
|
||
| Yahoo-SIP-REGISTER-From | 16,444 |
|
||
| www.bing.com | 14,485 |
|
||
| clients1.google.com.pk | 14,093 |
|
||
| SIP-REGISTER-From | 14,020 |
|
||
| Yahoo-B-Set-Cookie | 11,720 |
|
||
| DoubleClick-id-Cookie | 10,107 |
|
||
| Yahoo-Y-Set-Cookie | 8,631 |
|
||
| MS-MUID-Cookie | 7,222 |
|
||
| Yahoo-login-Method-Body | 6,964 |
|
||
| Google-Earth-Tile | 6,905 |
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## Results Pivot: Countries (From)
|
||
|
||
```
|
||
Sections
|
||
Parameters
|
||
Results
|
||
Event Types
|
||
Bearers
|
||
Countries (From)
|
||
Countries (To)
|
||
Event Type, Country
|
||
Country, Bearer
|
||
Bearer, Event Type
|
||
Daily Counts
|
||
Bearer Counts
|
||
Bearer Pivot
|
||
EventType Count
|
||
EventType Pivot
|
||
Country Counts
|
||
Country Pivot
|
||
SourceType Count
|
||
Source Type Pivot
|
||
```
|
||
|
||
| Origination Country | Destination Country | Bearer | Event Count |
|
||
| ------------------- | ------------------- | ------ | ----------- |
|
||
| PK | US | EREPO | 450,341 |
|
||
| | FK | EREPO | 341,767 |
|
||
| | DE | EREPO | 103,742 |
|
||
| | MY | EREPO | 3,482 |
|
||
| | KS | EREPO | 2,102 |
|
||
| | IN | EREPO | 1,704 |
|
||
| | CN | EREPO | 779 |
|
||
| | CH | EREPO | 668 |
|
||
| | IL | EREPO | 245 |
|
||
| | GF | EREPO | 133 |
|
||
| | GB | EREPO | 128 |
|
||
| | HK | EREPO | 73 |
|
||
| | FR | EREPO | 47 |
|
||
| | IP | EREPO | 18 |
|
||
| | EU | EREPO | 14 |
|
||
| | JF | EREPO | 12 |
|
||
| | NO | EREPO | 8 |
|
||
| | FS | EREPO | 8 |
|
||
| | DE | EREPO | 6 |
|
||
| | CA | EREPO | 4 |
|
||
| | FD | EREPO | 2 |
|
||
| | NK | EREPO | 2 |
|
||
| | II | EREPO | 1 |
|
||
| | IP | EREPO | 1 |
|
||
| | XX | EREPO | 1 |
|
||
| GB | FK | EREPO | 257,239 |
|
||
| | IR | EREPO | 2,242 |
|
||
| | II | EREPO | 179 |
|
||
| | XX | EREPO | 56 |
|
||
| | US | EREPO | 13 |
|
||
| | DE | EREPO | 1 |
|
||
| CH | FK | EREPO | 128,305 |
|
||
| | IR | EREPO | 7,9 |
|
||
| | II | EREPO | 17 |
|
||
| | XX | EREPO | 12 |
|
||
| MY | FK | EREPO | 48,190 |
|
||
| | XX | EREPO | 27 |
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## Recent Enhancements
|
||
|
||
Data from Bude (RPC)
|
||
|
||
* Including data from SWORDPLAY
|
||
|
||
New fields
|
||
|
||
* PDDG
|
||
* SIGAD
|
||
* SSDG
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## Future Enhancements
|
||
|
||
Near future:
|
||
|
||
* Adding BROAD OAK Targeting data
|
||
* Incorporating MI functionality from REFORMER (where appropriate!)
|
||
* Adding more feeds. (Ongoing)
|
||
|
||
Longer term:
|
||
|
||
* Adding Cipher and eAD MI information
|
||
* Linkage into ARTEMIS (or its successor)
|
||
|
||
SECRET STRAP1
|
||
GCHQ
|
||
|
||
---
|
||
|
||
events
|
||
|
||
## Any Questions
|
||
|
||
?
|
||
|
||
SECRET STRAP1
|
||
GCHQ |