620 lines
No EOL
13 KiB
Markdown
620 lines
No EOL
13 KiB
Markdown
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
# SKYNET:
|
|
## Courier Detection via Machine Learning
|
|
|
|
* [REDACTED], R66F/JHU
|
|
* [REDACTED], R66F
|
|
* [REDACTED], R66F
|
|
* [REDACTED], T1211
|
|
* [REDACTED], T1211
|
|
* [REDACTED], S2I5
|
|
* [REDACTED], S2I5/TD
|
|
|
|
June 5, 2012
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Given a handful of courier selectors, can we find others that “behave similarly” by analyzing GSM metadata?
|
|
|
|
```
|
|
OPR249(I)
|
|
SAIDU SHARIF
|
|
NDB
|
|
JALALABAD
|
|
15600
|
|
NDB
|
|
ARDEZ
|
|
640-2°E
|
|
Khost
|
|
BANNU
|
|
Miram Shah, North Waziristan
|
|
11770
|
|
Wana, South Waziristan
|
|
DERA ISMAIL KHAN
|
|
13000
|
|
OPR217
|
|
OPR203
|
|
NDB
|
|
CHARSADDA
|
|
PESHAWAR
|
|
OPR205
|
|
TARBELA
|
|
DAM
|
|
HARIPUR
|
|
NHAS
|
|
OPR210
|
|
OPR202
|
|
KOHAT
|
|
MARI INDUS EAS
|
|
OPR217
|
|
NDB
|
|
DARWALI
|
|
OPR203
|
|
4992
|
|
OPR216
|
|
SARGODHA
|
|
OPR230
|
|
DERA ISMAIL KHAN
|
|
OPR218
|
|
JHANG SADR
|
|
OPR253
|
|
7980
|
|
OPR251
|
|
RANQUI
|
|
MULTAN
|
|
VEHARI
|
|
Abbottabad
|
|
Islamabad
|
|
Rawalpindi
|
|
OPR277
|
|
RAWALPINDI
|
|
TAXILA
|
|
CHAKIALA
|
|
NDB
|
|
BANIHA
|
|
OPR235
|
|
ABANDONED
|
|
RAHWALI
|
|
NDB
|
|
SHEIKRUF
|
|
Faisalabad
|
|
DME
|
|
LAHORE
|
|
NDB
|
|
FAISALABAD
|
|
OKARA
|
|
Lahore
|
|
Sahiwal
|
|
Miles 20 40 60 80
|
|
Km 100
|
|
CEASE
|
|
|
|
Paths Legend
|
|
|
|
It's worth noting that:
|
|
* we are looking for different people using phones in similar ways
|
|
* without using any call chaining techniques from known selectors
|
|
* by scanning through all selectors seen in Pakistan that have not left Af/Pak (~55M)
|
|
```
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## From GSM metadata, we can measure aspects of each selector's pattern-of-life, social network, and travel behavior
|
|
|
|
```
|
|
Path Legend
|
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
|
|
Sunday
|
|
Monday
|
|
Tuesday
|
|
Wednesday
|
|
Thursday
|
|
Friday
|
|
Saturday
|
|
```
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## This presentation describes our search for AQSL couriers using behavioral profiling
|
|
|
|
* Behavioral Feature Extraction
|
|
* Cross Validation Experiment on AQSL Couriers
|
|
* Preliminary SIGINT Findings
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Counting unique UCELLIDs shows that couriers travel more often than typical Pakistani selectors
|
|
|
|
```
|
|
density
|
|
0.06
|
|
0.05
|
|
0.04
|
|
0.03
|
|
0.02
|
|
0.01
|
|
0.00
|
|
|
|
0 20 40 60 80 100 120
|
|
Number of Unique CellIDs
|
|
|
|
Group
|
|
AQSL Local Comms
|
|
AQSL Remote Comms
|
|
Seen in Pakistan
|
|
```
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## By examining multiple features at once, we can see some indicative behaviors of our courier selectors
|
|
|
|
```
|
|
Average Daily Outgoing Voice
|
|
10
|
|
8
|
|
6
|
|
4
|
|
2
|
|
0
|
|
|
|
Average Daily Incoming Voice
|
|
0 2 4 6 8 10
|
|
|
|
numCell
|
|
0
|
|
20
|
|
40
|
|
60
|
|
80
|
|
100
|
|
120
|
|
|
|
Group
|
|
Seen in Pakistan
|
|
AQSL Local Comms
|
|
AQSL Remote Comms
|
|
```
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Looking at a hierarchical clustering derived from all 80 features, the AQSL groups mostly stay together
|
|
|
|
```
|
|
Gen Pop : msi
|
|
Gen Pop : imsi
|
|
Gen Pop : imsi
|
|
AQSL Remote Comms : imsi
|
|
AQSL Remote Comms : imsi
|
|
AQSL Remote Comms : imsi
|
|
AQSL Remote Comms : imsi
|
|
Gen Pop: msi
|
|
AQSL Local Comms : imsi
|
|
AQSL Local Comms : imsi
|
|
Gen Pop : imsi
|
|
Gen Pop : msi
|
|
Gen Pop : imsi
|
|
Gen Pop : imsi
|
|
Gen Pop : imsi
|
|
Gen Pop : msi
|
|
Gen Pop : imsi
|
|
Gen Pop : imsi
|
|
Gen Pop : imsi
|
|
Gen Pop : msi
|
|
Gen Pop : imsi
|
|
Gen Pop : imsi
|
|
Gen Pop : imsi
|
|
Gen Pop : imsi
|
|
Gen Pop : msi
|
|
Gen Pop : msi
|
|
Gen Pop : imsi
|
|
Gen Pop : imsi
|
|
Gen Pop : imsi
|
|
Gen Pop : imsi
|
|
4 days of collect AQSL Local Comms : imsi
|
|
Gen Pop : imsi
|
|
|
|
1.0
|
|
0.8
|
|
0.6
|
|
0.4
|
|
0.2
|
|
0.0
|
|
```
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Now, we'll describe a cross validation experiment on the AQSL selectors that we were provided
|
|
|
|
* Behavioral Feature Extraction
|
|
* **Cross Validation Experiment on AQSL Couriers**
|
|
* Preliminary SIGINT Findings
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Our initial detector uses the centroid of the AQSL couriers to “find other selectors like these”
|
|
|
|
AQSL Cross-Validation Experiment
|
|
|
|
* 7 MSISDN/IMSI pairs
|
|
* Hold each pair out and score them when training the centroid on the rest
|
|
* Assume that random draws of Pakistani selectors are nontargets
|
|
* How well do we do?
|
|
|
|
```
|
|
miss probability (%)
|
|
99
|
|
95
|
|
80
|
|
60
|
|
40
|
|
20
|
|
5
|
|
1
|
|
0.1
|
|
0.01
|
|
1e-04
|
|
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
|
|
false alarm probability (%)
|
|
|
|
* Centroid(All Raw Features)
|
|
* Centroid(All Normalized Features)
|
|
* Centroid(Outgoing Raw Features)
|
|
* Centroid(Outgoing Normalized Features)
|
|
```
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Our initial detector uses the centroid of the AQSL couriers to “find other selectors like these”
|
|
|
|
AQSL Cross-Validation Experiment
|
|
|
|
* Initial experiments showed EER in 10-20% range
|
|
* Here, performance is much worse against these nontargets:
|
|
* Seen in Pakistan
|
|
* Not seen outside of Af/Pak
|
|
* Not FVEY selectors
|
|
|
|
```
|
|
miss probability (%)
|
|
99
|
|
95
|
|
80
|
|
60
|
|
40
|
|
20
|
|
5
|
|
1
|
|
0.1
|
|
0.01
|
|
1e-04
|
|
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
|
|
false alarm probability (%)
|
|
|
|
* Centroid(All Raw Features)
|
|
* Centroid(All Normalized Features)
|
|
* Centroid(Outgoing Raw Features)
|
|
* Centroid(Outgoing Normalized Features)
|
|
```
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Statistical algorithms are able to find the couriers at very low false alarm rates, if we're allowed to miss half of them
|
|
|
|
Random Forest Classifier
|
|
|
|
* 7 MSISDN/IMSI pairs
|
|
* Hold each pair out and then try to find them after learning how to distinguish remaining couriers from other Pakistanis (using 100k random selectors here)
|
|
* Assume that random draws of Pakistani selectors are nontargets
|
|
* 0.18% False Alarm Rate at 50% Miss Rate
|
|
|
|
```
|
|
miss probability (%)
|
|
99
|
|
95
|
|
80
|
|
60
|
|
40
|
|
20
|
|
5
|
|
1
|
|
0.1
|
|
0.01
|
|
1e-04
|
|
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
|
|
false alarm probability (%)
|
|
|
|
better
|
|
|
|
* Centroid(All Raw Features)
|
|
* Centroid(All Normalized Features)
|
|
* Centroid(Outgoing Raw Features)
|
|
* Centroid(Outgoing Normalized Features)
|
|
* Random Forest(All Raw Features)
|
|
* Random Forest (Outgoing Raw Features)
|
|
```
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## We've been experimenting with several error metrics on both small and large test sets
|
|
|
|
| | | | 100k Test Selectors | | 55M Test Selectors | |
|
|
|------------------------|------------------|----------|----------------------------------------|--------------------------|-----------------------------------|-----------------------------------|
|
|
| Training Data | Classifier | Features | False Alarm Rate at 50% Miss Rate | Mean Reciprocal Rank | Tasked Selectors in Top 500 | Tasked Selectors in Top 100 |
|
|
| None | Random | None | 50% | 1/23k (simulated) | 0.64 (active/Pak) | 0.13 (active/Pak) |
|
|
| Known Couriers | Centroid | All | 20% | 1/18k | | |
|
|
| | | Outgoing | 43% | 1/27k | | |
|
|
| | Random Forest | Outgoing | 0.18% | 1/9.9 | 5 | 1 |
|
|
| + Anchory Selectors | Random Forest | Outgoing | | | | |
|
|
|
|
Random Forest:
|
|
* 0.18% false alarm rate at 50% miss rate
|
|
* 7x improvement over random performance when evaluating its tasked precision at 100
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## To get more training data we scraped selectors from S2I11 Anchory reports containing keyword "courier”
|
|
|
|
Anchory Selectors
|
|
|
|
* Searched for reports containing "S2I11” AND "courier"
|
|
* Filtered out non-mobile numbers and kept selectors with "interesting” travel patterns seen in SmartTracker
|
|
|
|
```
|
|
Quetta
|
|
bad
|
|
Borth V tan
|
|
ahore
|
|
```
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Adding selectors from Anchory reports to the training data reduced the false alarm rates even further
|
|
|
|
Anchory Selectors
|
|
|
|
* Searched for reports containing "S2I11” AND "courier"
|
|
* Filtered out non-mobile numbers and kept selectors with "interesting” travel patterns seen in SmartTracker
|
|
|
|
```
|
|
miss probability (%)
|
|
99
|
|
95
|
|
80
|
|
60
|
|
40
|
|
20
|
|
5
|
|
1
|
|
0.1
|
|
0.01
|
|
1e-04
|
|
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
|
|
false alarm probability (%)
|
|
|
|
* Centroid(All Raw Features)
|
|
* Centroid(All Normalized Features)
|
|
* Centroid(Outgoing Raw Features)
|
|
* Centroid(Outgoing Normalized Features)
|
|
* Random Forest(All Raw Features)
|
|
* Random Forest (Outgoing Raw Features)
|
|
```
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## We've been experimenting with several error metrics on both small and large test sets
|
|
|
|
| | | | 100k Test Selectors | | 55M Test Selectors | |
|
|
|---------------------|---------------|----------|-----------------------------------|----------------------|---------------------------------|---------------------------------|
|
|
| Training Data | Classifier | Features | False Alarm Rate at 50% Miss Rate | Mean Reciprocal Rank | Tasked Selectors in Top 500 | Tasked Selectors in Top 100 |
|
|
| None | Random | None | 50% | 1/23k (simulated) | 0.64 (active/Pak) | 0.13 (active/Pak) |
|
|
| Known Couriers | Centroid | All | 20% | 1/18k | | |
|
|
| | | Outgoing | 43% | 1/27k | | |
|
|
| | Random Forest | Outgoing | 0.18% | 1/9.9 | 5 | 1 |
|
|
| + Anchory Selectors | Random Forest | Outgoing | 0.008% | 1/14 | 21 | 6 |
|
|
|
|
Random Forest trained on Known Couriers + Anchory Selectors:
|
|
* 0.008% false alarm rate at 50% miss rate
|
|
* 46x improvement over random performance when evaluating its tasked precision at 100
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Now, we'll investigate some findings after running these classifiers on +55M Pakistani selectors via MapReduce
|
|
|
|
* Behavioral Feature Extraction
|
|
* Cross Validation Experiment on AQSL Couriers
|
|
* **Preliminary SIGINT Findings**
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## The highest scoring selector that traveled to Peshawar and Lahore is PROB AHMED ZAIDAN
|
|
|
|
```
|
|
amabad
|
|
walpind
|
|
Khost
|
|
Miram Shah, North Wazirist
|
|
Wana, South Waziristan
|
|
Faisalaba
|
|
hore
|
|
|
|
Miles 100
|
|
Km 100 200 300
|
|
|
|
Path Legend
|
|
PROB AHMED MUWAFAK ZAIDAN
|
|
TIDE Person Number: [REDACTED]
|
|
* MEMBER OF AL-QA'IDA
|
|
* MEMBER OF MUSLIM BROTHERHOOD
|
|
* WORKS FOR AL JAZEERA
|
|
```
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## In the top 500 scoring selectors, 21 are tasked leading us to believe that we're on the right track
|
|
|
|
```
|
|
amabad
|
|
walpind
|
|
Shah North
|
|
Wana, South Waz n
|
|
hore
|
|
uetta
|
|
|
|
Miles 100
|
|
Km 100 200 300
|
|
|
|
Path Legend
|
|
[REDACTED]
|
|
[REDACTED]
|
|
[REDACTED]
|
|
[REDACTED]
|
|
[REDACTED]
|
|
[REDACTED]
|
|
```
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## We have also discovered many untasked selectors with interesting travel patterns
|
|
|
|
```
|
|
ost
|
|
iram Shah, North Waziristan
|
|
Wana, South Waziristan
|
|
ahore
|
|
Quetta
|
|
Karachi
|
|
|
|
Path Legend
|
|
[REDACTED]
|
|
[REDACTED]
|
|
[REDACTED]
|
|
[REDACTED]
|
|
[REDACTED]
|
|
[REDACTED]
|
|
```
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Preliminary results indicate that we're on the right track, but much remains to be done
|
|
|
|
Cross Validation Experiment:
|
|
|
|
* Random Forest classifier operating at 0.18% false alarm rate at 50% miss
|
|
* Enhancing training data with Anchory selectors reduced that to 0.008%
|
|
* Mean Reciprocal Rank is ~1/10
|
|
|
|
Preliminary SIGINT Findings:
|
|
|
|
* Behavioral features helped discover similar selectors with “courier-like" travel patterns
|
|
* High number of tasked selectors at the top is hopefully indicative of the detector performing well “in the wild”
|
|
|
|
```
|
|
miss probability (%)
|
|
99
|
|
95
|
|
80
|
|
60
|
|
40
|
|
20
|
|
5
|
|
1
|
|
0.1
|
|
0.01
|
|
1e-04
|
|
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
|
|
false alarm probability (%)
|
|
|
|
* Centroid(All Raw Features)
|
|
* Centroid(All Normalized Features)
|
|
* Centroid(Outgoing Raw Features)
|
|
* Centroid(Outgoing Normalized Features)
|
|
* Random Forest(All Raw Features)
|
|
* Random Forest (Outgoing Raw Features)
|
|
```
|
|
|
|
```
|
|
amabad
|
|
Wana, South Waz
|
|
quetta
|
|
Miles 100
|
|
Km 100 200 300
|
|
```
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY |