snowchive-meta/data/tdi-introduction/tdi-introduction.md
2026-11-19 04:39:11 +02:00

20 KiB
Raw Blame History

APPLIED RESEARCH

Target Detection Identifiers

[REDACTED]

March 2009

© This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation. Refer any FOIA queries to [REDACTED].

Slide 1

APPLIED RESEARCH


UK SECRET STRAP2 COMINT ORCON

High-Speed Internet Processing

TCP SYN

GET /
User-Agent: Mozilla 4.1, IE5
Host:www.google.com
Cookie:ik=xzxsrzczccZ

TCP FIN

...
09:28:01 2008-10-13 [REDACTED] 7776 80 GET / Cookie: ik= qyzwww.....
09:28:13 2008-10-13 [REDACTED] 3456 80 GET / Cookie:ik=xzxsrzczccz
...

Google™

Event data sent to bulk store

© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.

UK SECRET STRAP2 COMINT ORCON Slide 2


UK SECRET STRAP2 COMINT ORCON

High-Speed Internet Processing

  • Bulk events key to SIGINT success on Internet

  • Event types that are valuable for Intelligence change (quickly)

    • 2000 SMTP/POP3
    • 2001 Webmail
    • ...
    • 2007 vBulletin
    • 2008 Social Networks,...,?
  • GCHQ's Applied Research are pioneering ways of dealing with this:

    • Presence Events (TDI)
    • Very large scale high speed flat file storage to bulk store TDIs
    • Just enough data marts

© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.

UK SECRET STRAP2 COMINT ORCON Slide 3


UK SECRET STRAP2 COMINT ORCON

IP Packet Information

  • Many possible types of information

  • Many techniques available

  • HTTP Get requests dominate cutting edge techniques

  • To get Intelligence value Information must relate to a person or device... a TDI

© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.

UK SECRET STRAP2 COMINT ORCON Slide 4


UK SECRET STRAP2 COMINT ORCON

TDI ...?

;

© Crown Copyright. A [REDACTED] other UK information legislation. Refer disclosure requests [REDACTED] under Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.

UK SECRET STRAP2 COMINT ORCON Slide 5


UK SECRET STRAP2 COMINT ORCON

TDI ...?

;

© Crown Copyright. [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.

UK SECRET STRAP2 COMINT ORCON Slide 6


UK SECRET STRAP2 COMINT ORCON

TDI

Target Detection Identifier

© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.

UK SECRET STRAP2 COMINT ORCON Slide 7


UK SECRET STRAP2 COMINT ORCON

TDI

Target Detection Identifier

Who When Where (doing) What

© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.

UK SECRET STRAP2 COMINT ORCON Slide 8


UK SECRET STRAP2 COMINT ORCON

TDI

Target Detection Identifier

Who When Where (doing) What

Fundamental atom of the Internet age.

© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.

UK SECRET STRAP2 COMINT ORCON Slide 9


UK SECRET STRAP2 COMINT ORCON

Target Detection Identifiers

  • DEFINITION

    • TDIs are definite indicators of presence, that are unique and persistent for a user/machine.
  • Built on the familiar

    • Telephony +44 – international phone code
    • Signalling tells us this phone user is 'online'
  • Target Detection Identifiers

    • Started with the Internet, mobile networks too.
    • TDI is a 'SIGINT standardised code'.
    • Not a standard managed by the ITU/ETSI.
    • Extraction from packets much more complex.

© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.

UK SECRET STRAP2 COMINT ORCON Slide 10


UK SECRET STRAP2 COMINT ORCON

TDI sources

hi5
BBC
YAHOO!
msn Hotmail
YouTube
reddit
Windows Live
MAKTOOb
First Arabic /English e-mail
facebook
Ask
Google
Windows
amazon.com
yellow
vBulletin
WORDPRESS
Google Earth
CNN

© Crown Copyright. All rights re [REDACTED] disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.

UK SECRET STRAP2 COMINT ORCON Slide 11


UK SECRET STRAP2 COMINT ORCON

Target Detection Identifiers

  • 70 distinct TDI types discovered.

  • 2500 TDIs/sec (GET, de-duplicated)

  • => 200 Million per day per 10Gbps

  • De-dupe rate ???

  • Cost – 250 hours per TDI

  • Automated discovery prototype

TDI Type TDI Location User/Machine
Yahoo-Y-Cookie Cookie User
Yahoo-B-Cookie Coookie Machine
Google-IK Request-URI User
Paltalk-Nickname Request-URI User
MS-MUID-Cookie Cookie Machine
Google-SID-Cookie Cookie Machine
Maktoob-MEUser-Cookie Cookie User
Orkut-PREFID-Cookie Cookie User
Cloob-Username Cookie User

© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Fr [REDACTED] other UK information legislation. Refer disclosure requests to [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.

UK SECRET STRAP2 COMINT ORCON Slide 12


SECRET

[REDACTED] Search GCHQ
CHQ Homepage | B AutoTDI
GET data

Report is for the 7-day period from 04/10/08 (16:20) to 11/10/08 (16:20).
A cross-subnet threshold of ≤ 10% has been applied.
A mean user transmission threshold of ≥ 10 has been applied.
Putative selectors http://, photos/, images/, files/ are blacklisted from this plot.
Click to sort by a given column.
Domain Context Technology Selector Example value Bearer count Observation count Mean user transmission frequency Cross-/16 percentage
facebook Cookie datr= [REDACTED] 8 671 12.98 3.51
facebook Cookie c_user= [REDACTED] 8 651 12.09 3.51
facebook Cookie __utmz= [REDACTED] 7 609 12.14 4.25
facebook Cookie __utma= [REDACTED] 7 609 12.44 3.56
facebook Cookie h_user= [REDACTED] 7 601 12.37 3.74
facebook Cookie __qca= [REDACTED] 6 364 10.38 4.97
reuters Cookie lv= [REDACTED] 6 336 10.67 0.24
facebook Cookie next_path= [REDACTED] 6 323 18.63 9.18
live Cookie MUID= [REDACTED] 7 321 10.81 3.24
reuters Cookie id= [REDACTED] 6 312 21.59 0.45
google URI q= [REDACTED] 7 311 15.02 5.81
reuters Cookie ss= [REDACTED] 6 309 16.83 0.39
yahoo Cookie B= [REDACTED] 7 307 10.76 2.76
yahoo Cookie d= [REDACTED] 6 306 10.60 7.79
youporn Cookie sid= [REDACTED] 5 290 24.90 1.96
youporn Cookie __utma= [REDACTED] 5 282 24.23 1.65
youporn Cookie __utmz= [REDACTED] 5 281 22.92 4.60
reuters Cookie anonId= [REDACTED] 6 279 16.22 0.46
youporn Cookie __qca= [REDACTED] 5 277 24.40 1.69
yahoo URI p= [REDACTED] 7 277 31.18 6.89
bebo Cookie bdaysession= [REDACTED] 7 275 27.19 2.06
google Cookie LM= [REDACTED] 7 272 16.85 7.31
google Cookie ID= [REDACTED] 7 271 16.80 3.73
google Cookie TM= [REDACTED] 7 270 17.07 6.57
bebo Cookie Username= [REDACTED] 7 268 27.18 2.21
bebo Cookie Email= [REDACTED] 7 268 27.67 2.24
yahoo Cookie I_s= [REDACTED] 4 264 78.61 3.00
google Cookie S= [REDACTED] 6 264 39.35 3.82
yahoo Cookie ip= [REDACTED] 3 253 14.24 2.54
yieldmanager Cookie uid= [REDACTED] 7 251 66.03 1.01
reuters Cookie RaptTracker= [REDACTED] 6 242 14.24 0.48
yahoo Referer p= [REDACTED] 4 242 17.85 7.19

UK SECRET STRAP2 COMINT ORCON

TDI Applications

  • Bulk store of all TDIs seen in last 6 months [MUTANT BROTH]

  • Bulk store TDI correlations (6 months) [AUTO ASSOC]

  • Bulk store TDI <-> website correlations (6 months) [KARMA POLICE]

  • Bulk store TDI vBulletin activity [INFINITE MONKEYS]

  • Bulk store TDI Social Networking Site activity [SOCIAL ANIMAL]

  • Bulk store web search requests [MEMORY HOLE]

  • Bulk store Google Earth requests [MARBLED GECKO]

  • Bulk store of Host.Referer references [HRMAP]

© Crown [REDACTED] Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.

UK SECRET STRAP2 COMINT ORCON Slide 14


SECRET

Edit View History Bookmarks Tools Help
B [REDACTED] Search GCHQ
s wiki 10gR2 fsrf PE ct blog /em/ im AR b13sw eci wiki ft wikipedia

MUTANT BROTH | 401 Authorization Required

UTANT BROTH

Identifier Search | IP Address Search | Password Search

Welcome

Logged in as [REDACTED] all queries logged for audit.
Database currently contains identifiers from the period Tue Dec 25 16:26:40 2007 to Fri Jun 20 22:13:19 2008 (18.41 billion rows of as 07-JUN-08).
Warning: data for period(s):-
*   Fri Jun 20 22:13:21 2008 - Tue Jun 24 09:33:20 2008
is loaded, but currently unavailable for query due to index building. The rest of the database can be queried as usual during the rebuild.

Search for Identifiers
*   If allow wildcards is ticked, % and _ are multi-character (bob%) and single-character (b_b@hotmail.com) wildcards.
*   Queries are always case-sensitive (bob@hotmail.com ≠ BOB@hotmail.com ≠ BOB@HOTMAIL.COM). There is an option to automatically convert to lowercase.
*   For bulk queries, paste in a list of identifiers separated by newlines (one per line).
*   You can enter a minimum/maximum date for the search: default is to search all available selectors

[REDACTED]

MIRANDA 20135
JIC 1
Purpose NS
Reason demo
Execute

Allow wildcards
Convert to lowercase before searching

Matching Identifiers

The following identifiers have been found in the MUTANT BROTH database.
Select those that match your target(s) to generate a summary of target activity.
TDI type TDI value
Chat-MS-Messenger [REDACTED]@hotmail.fr

SECRET

Edit View History Bookmarks Tools Help
B [REDACTED] Search GCHQ
s wiki 10gR2 fsrf PE ct blog /em/ im AR b13sw eci wiki ft wikipedia

MUTANT BROTH | 401 Authorization Required

UTANT BROTH

Identifier Search | IP Address Search | Password Search

Welcome

Logged in as [REDACTED] all queries logged for audit.
Database currently contains identifiers from the period Tue Dec 25 16:26:40 2007 to Fri Jun 20 22:13:19 2008 (18.41 billion rows of as 07-JUN-08).

Warning: data for period(s):-
*   Fri Jun 20 22:1
is loaded, but curre rebuild.

(18.41 billion rows of as 07-JUN-08).

Search for Ident
*   If allow wildcards is
*   Queries are always case-sensitive (bob@hotmail.com ≠ BOB@hotmail.com ≠ BOB@HOTMAIL.COM). There is an option to automatically convert to lowercase.
*   For bulk queries, paste in a list of identifiers separated by newlines (one per line).
*   You can enter a minimum/maximum date for the search: default is to search all available selectors

[REDACTED]

MIRANDA 20135
JIC 1
Purpose NS
Reason demo
Execute

Allow wildcards
Convert to lowercase before searching

Matching Identifiers

The following identifiers have been found in the MUTANT BROTH database.
Select those that match your target(s) to generate a summary of target activity.
TDI type TDI value
Chat-MS-Messenger [REDACTED]@hotmail.fr

SECRET

WHEN WHERE WHAT WHO

Edit View History Bookmarks Tools Help
B [REDACTED] Search GCHQ
wiki 10gR2 fsrf PE ct blog /em/ im AR b13sw eci wiki ft wikipedia

MUTANT BROTH | 401 Authorization Required

[REDACTED] Chat Identifiers), the source IP Geo-Location identifies the other end of the communications link to the

Save as CSV
Date Time Source IP HHFP Source IP Geo Identifier Type Identifier Value Passw
17/06/2008 17:08:44 [REDACTED] 6de32bb0 41.02;28.96;ISTANBUL; TR; 5MMM Hi5-Email-Cookie [REDACTED]@hotmail.com
17/08 [REDACTED] 6de32bb0 41.02[REDACTED] M Hi5-Email-Cookie [REDACTED]@hotmail.com
17/06/2008 16:55:21 6de32bb0 41.9022;-87.6726; CHICAGO; US;5 [REDACTED] [REDACTED]@hotmail.com
17/06/2008 16:55:16 [REDACTED] 6de32bb0 41.02;28.96; ISTANBUL; TR; 5MMM Hi5-Email-Cookie [REDACTED]@hotmail.com
17/06/2008 16:54:47 6de32bb0 41.9022;-87.6726; CHICAGO; US;5MMM Hi5-Email-Cookie [REDACTED]@hotmail.com
17/06/2008 16:52:13 [REDACTED] 6de32bb0 39.94;32.86; ANKARA; TR; 5MMM Hi5-Email-Cookie [REDACTED]@hotmail.com
15/06/2008 19:20:33 [REDACTED] de8bdc48 33.5;36.3; DIMASHQ;SY; 5MLV Hi5-Email-Cookie [REDACTED]@hotmail.com

UK SECRET STRAP2 COMINT ORCON

Other Bulk Event Applications

  • Most events that can be associated back to TDIs:

  • File Transfer Signature (eg proof of life videos)

  • Detection by Internet profile – eg 'Dead Letter Drop'.

  • Yahoo webcam images

  • Airline reservation confirmation emails

© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to [REDACTED] Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must be obtained for dissemination outside the organisation.

UK SECRET STRAP2 COMINT ORCON Slide 18