snowchive-meta/data/skynet-courier-detection-via-machine-learning/skynet-courier-detection-via-machine-learning.md
2026-11-19 04:39:11 +02:00

13 KiB

TOP SECRET//COMINT//REL TO USA, FVEY

SKYNET:

Courier Detection via Machine Learning

  • [REDACTED], R66F/JHU
  • [REDACTED], R66F
  • [REDACTED], R66F
  • [REDACTED], T1211
  • [REDACTED], T1211
  • [REDACTED], S2I5
  • [REDACTED], S2I5/TD

June 5, 2012

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Given a handful of courier selectors, can we find others that “behave similarly” by analyzing GSM metadata?

OPR249(I)
SAIDU SHARIF
NDB
JALALABAD
15600
NDB
ARDEZ
640-2°E
Khost
BANNU
Miram Shah, North Waziristan
11770
Wana, South Waziristan
DERA ISMAIL KHAN
13000
OPR217
OPR203
NDB
CHARSADDA
PESHAWAR
OPR205
TARBELA
DAM
HARIPUR
NHAS
OPR210
OPR202
KOHAT
MARI INDUS EAS
OPR217
NDB
DARWALI
OPR203
4992
OPR216
SARGODHA
OPR230
DERA ISMAIL KHAN
OPR218
JHANG SADR
OPR253
7980
OPR251
RANQUI
MULTAN
VEHARI
Abbottabad
Islamabad
Rawalpindi
OPR277
RAWALPINDI
TAXILA
CHAKIALA
NDB
BANIHA
OPR235
ABANDONED
RAHWALI
NDB
SHEIKRUF
Faisalabad
DME
LAHORE
NDB
FAISALABAD
OKARA
Lahore
Sahiwal
Miles 20  40  60  80
Km         100
CEASE

Paths Legend

It's worth noting that:
* we are looking for different people using phones in similar ways
* without using any call chaining techniques from known selectors
* by scanning through all selectors seen in Pakistan that have not left Af/Pak (~55M)

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

From GSM metadata, we can measure aspects of each selector's pattern-of-life, social network, and travel behavior

Path Legend
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Sunday
Monday
Tuesday
Wednesday
Thursday
Friday
Saturday

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

This presentation describes our search for AQSL couriers using behavioral profiling

  • Behavioral Feature Extraction
  • Cross Validation Experiment on AQSL Couriers
  • Preliminary SIGINT Findings

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Counting unique UCELLIDs shows that couriers travel more often than typical Pakistani selectors

density
0.06
0.05
0.04
0.03
0.02
0.01
0.00

0 20 40 60 80 100 120
Number of Unique CellIDs

Group
AQSL Local Comms
AQSL Remote Comms
Seen in Pakistan

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

By examining multiple features at once, we can see some indicative behaviors of our courier selectors

Average Daily Outgoing Voice
10
8
6
4
2
0

Average Daily Incoming Voice
0 2 4 6 8 10

numCell
0
20
40
60
80
100
120

Group
Seen in Pakistan
AQSL Local Comms
AQSL Remote Comms

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Looking at a hierarchical clustering derived from all 80 features, the AQSL groups mostly stay together

Gen Pop : msi
Gen Pop : imsi
Gen Pop : imsi
AQSL Remote Comms : imsi
AQSL Remote Comms : imsi
AQSL Remote Comms : imsi
AQSL Remote Comms : imsi
Gen Pop: msi
AQSL Local Comms : imsi
AQSL Local Comms : imsi
Gen Pop : imsi
Gen Pop : msi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : msi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : msi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : msi
Gen Pop : msi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : imsi
4 days of collect   AQSL Local Comms : imsi
Gen Pop : imsi

1.0
0.8
0.6
0.4
0.2
0.0

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Now, we'll describe a cross validation experiment on the AQSL selectors that we were provided

  • Behavioral Feature Extraction
  • Cross Validation Experiment on AQSL Couriers
  • Preliminary SIGINT Findings

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Our initial detector uses the centroid of the AQSL couriers to “find other selectors like these”

AQSL Cross-Validation Experiment

  • 7 MSISDN/IMSI pairs
  • Hold each pair out and score them when training the centroid on the rest
  • Assume that random draws of Pakistani selectors are nontargets
  • How well do we do?
miss probability (%)
99
95
80
60
40
20
5
1
0.1
0.01
1e-04
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
false alarm probability (%)

* Centroid(All Raw Features)
* Centroid(All Normalized Features)
* Centroid(Outgoing Raw Features)
* Centroid(Outgoing Normalized Features)

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Our initial detector uses the centroid of the AQSL couriers to “find other selectors like these”

AQSL Cross-Validation Experiment

  • Initial experiments showed EER in 10-20% range
  • Here, performance is much worse against these nontargets:
    • Seen in Pakistan
    • Not seen outside of Af/Pak
    • Not FVEY selectors
miss probability (%)
99
95
80
60
40
20
5
1
0.1
0.01
1e-04
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
false alarm probability (%)

* Centroid(All Raw Features)
* Centroid(All Normalized Features)
* Centroid(Outgoing Raw Features)
* Centroid(Outgoing Normalized Features)

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Statistical algorithms are able to find the couriers at very low false alarm rates, if we're allowed to miss half of them

Random Forest Classifier

  • 7 MSISDN/IMSI pairs
  • Hold each pair out and then try to find them after learning how to distinguish remaining couriers from other Pakistanis (using 100k random selectors here)
  • Assume that random draws of Pakistani selectors are nontargets
  • 0.18% False Alarm Rate at 50% Miss Rate
miss probability (%)
99
95
80
60
40
20
5
1
0.1
0.01
1e-04
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
false alarm probability (%)

better

* Centroid(All Raw Features)
* Centroid(All Normalized Features)
* Centroid(Outgoing Raw Features)
* Centroid(Outgoing Normalized Features)
* Random Forest(All Raw Features)
* Random Forest (Outgoing Raw Features)

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

We've been experimenting with several error metrics on both small and large test sets

100k Test Selectors 55M Test Selectors
Training Data Classifier Features False Alarm Rate at 50% Miss Rate Mean Reciprocal Rank Tasked Selectors in Top 500 Tasked Selectors in Top 100
None Random None 50% 1/23k (simulated) 0.64 (active/Pak) 0.13 (active/Pak)
Known Couriers Centroid All 20% 1/18k
Outgoing 43% 1/27k
Random Forest Outgoing 0.18% 1/9.9 5 1
+ Anchory Selectors Random Forest Outgoing

Random Forest:

  • 0.18% false alarm rate at 50% miss rate
  • 7x improvement over random performance when evaluating its tasked precision at 100

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

To get more training data we scraped selectors from S2I11 Anchory reports containing keyword "courier”

Anchory Selectors

  • Searched for reports containing "S2I11” AND "courier"
  • Filtered out non-mobile numbers and kept selectors with "interesting” travel patterns seen in SmartTracker
Quetta
bad
Borth V tan
ahore

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Adding selectors from Anchory reports to the training data reduced the false alarm rates even further

Anchory Selectors

  • Searched for reports containing "S2I11” AND "courier"
  • Filtered out non-mobile numbers and kept selectors with "interesting” travel patterns seen in SmartTracker
miss probability (%)
99
95
80
60
40
20
5
1
0.1
0.01
1e-04
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
false alarm probability (%)

* Centroid(All Raw Features)
* Centroid(All Normalized Features)
* Centroid(Outgoing Raw Features)
* Centroid(Outgoing Normalized Features)
* Random Forest(All Raw Features)
* Random Forest (Outgoing Raw Features)

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

We've been experimenting with several error metrics on both small and large test sets

100k Test Selectors 55M Test Selectors
Training Data Classifier Features False Alarm Rate at 50% Miss Rate Mean Reciprocal Rank Tasked Selectors in Top 500 Tasked Selectors in Top 100
None Random None 50% 1/23k (simulated) 0.64 (active/Pak) 0.13 (active/Pak)
Known Couriers Centroid All 20% 1/18k
Outgoing 43% 1/27k
Random Forest Outgoing 0.18% 1/9.9 5 1
+ Anchory Selectors Random Forest Outgoing 0.008% 1/14 21 6

Random Forest trained on Known Couriers + Anchory Selectors:

  • 0.008% false alarm rate at 50% miss rate
  • 46x improvement over random performance when evaluating its tasked precision at 100

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Now, we'll investigate some findings after running these classifiers on +55M Pakistani selectors via MapReduce

  • Behavioral Feature Extraction
  • Cross Validation Experiment on AQSL Couriers
  • Preliminary SIGINT Findings

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

The highest scoring selector that traveled to Peshawar and Lahore is PROB AHMED ZAIDAN

amabad
walpind
Khost
Miram Shah, North Wazirist
Wana, South Waziristan
Faisalaba
hore

Miles 100
Km 100 200 300

Path Legend
PROB AHMED MUWAFAK ZAIDAN
TIDE Person Number: [REDACTED]
* MEMBER OF AL-QA'IDA
* MEMBER OF MUSLIM BROTHERHOOD
* WORKS FOR AL JAZEERA

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

In the top 500 scoring selectors, 21 are tasked leading us to believe that we're on the right track

amabad
walpind
Shah North
Wana, South Waz n
hore
uetta

Miles 100
Km 100 200 300

Path Legend
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

We have also discovered many untasked selectors with interesting travel patterns

ost
iram Shah, North Waziristan
Wana, South Waziristan
ahore
Quetta
Karachi

Path Legend
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Preliminary results indicate that we're on the right track, but much remains to be done

Cross Validation Experiment:

  • Random Forest classifier operating at 0.18% false alarm rate at 50% miss
  • Enhancing training data with Anchory selectors reduced that to 0.008%
  • Mean Reciprocal Rank is ~1/10

Preliminary SIGINT Findings:

  • Behavioral features helped discover similar selectors with “courier-like" travel patterns
  • High number of tasked selectors at the top is hopefully indicative of the detector performing well “in the wild”
miss probability (%)
99
95
80
60
40
20
5
1
0.1
0.01
1e-04
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
false alarm probability (%)

* Centroid(All Raw Features)
* Centroid(All Normalized Features)
* Centroid(Outgoing Raw Features)
* Centroid(Outgoing Normalized Features)
* Random Forest(All Raw Features)
* Random Forest (Outgoing Raw Features)
amabad
Wana, South Waz
quetta
Miles 100
Km 100 200 300

TOP SECRET//COMINT//REL TO USA, FVEY