snowchive-meta/data/skynet-courier-detection-via-machine-learning/skynet-courier-detection-via-machine-learning.md
2026-11-19 04:39:11 +02:00

620 lines
No EOL
13 KiB
Markdown

TOP SECRET//COMINT//REL TO USA, FVEY
# SKYNET:
## Courier Detection via Machine Learning
* [REDACTED], R66F/JHU
* [REDACTED], R66F
* [REDACTED], R66F
* [REDACTED], T1211
* [REDACTED], T1211
* [REDACTED], S2I5
* [REDACTED], S2I5/TD
June 5, 2012
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Given a handful of courier selectors, can we find others that “behave similarly” by analyzing GSM metadata?
```
OPR249(I)
SAIDU SHARIF
NDB
JALALABAD
15600
NDB
ARDEZ
640-2°E
Khost
BANNU
Miram Shah, North Waziristan
11770
Wana, South Waziristan
DERA ISMAIL KHAN
13000
OPR217
OPR203
NDB
CHARSADDA
PESHAWAR
OPR205
TARBELA
DAM
HARIPUR
NHAS
OPR210
OPR202
KOHAT
MARI INDUS EAS
OPR217
NDB
DARWALI
OPR203
4992
OPR216
SARGODHA
OPR230
DERA ISMAIL KHAN
OPR218
JHANG SADR
OPR253
7980
OPR251
RANQUI
MULTAN
VEHARI
Abbottabad
Islamabad
Rawalpindi
OPR277
RAWALPINDI
TAXILA
CHAKIALA
NDB
BANIHA
OPR235
ABANDONED
RAHWALI
NDB
SHEIKRUF
Faisalabad
DME
LAHORE
NDB
FAISALABAD
OKARA
Lahore
Sahiwal
Miles 20 40 60 80
Km 100
CEASE
Paths Legend
It's worth noting that:
* we are looking for different people using phones in similar ways
* without using any call chaining techniques from known selectors
* by scanning through all selectors seen in Pakistan that have not left Af/Pak (~55M)
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## From GSM metadata, we can measure aspects of each selector's pattern-of-life, social network, and travel behavior
```
Path Legend
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Sunday
Monday
Tuesday
Wednesday
Thursday
Friday
Saturday
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## This presentation describes our search for AQSL couriers using behavioral profiling
* Behavioral Feature Extraction
* Cross Validation Experiment on AQSL Couriers
* Preliminary SIGINT Findings
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Counting unique UCELLIDs shows that couriers travel more often than typical Pakistani selectors
```
density
0.06
0.05
0.04
0.03
0.02
0.01
0.00
0 20 40 60 80 100 120
Number of Unique CellIDs
Group
AQSL Local Comms
AQSL Remote Comms
Seen in Pakistan
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## By examining multiple features at once, we can see some indicative behaviors of our courier selectors
```
Average Daily Outgoing Voice
10
8
6
4
2
0
Average Daily Incoming Voice
0 2 4 6 8 10
numCell
0
20
40
60
80
100
120
Group
Seen in Pakistan
AQSL Local Comms
AQSL Remote Comms
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Looking at a hierarchical clustering derived from all 80 features, the AQSL groups mostly stay together
```
Gen Pop : msi
Gen Pop : imsi
Gen Pop : imsi
AQSL Remote Comms : imsi
AQSL Remote Comms : imsi
AQSL Remote Comms : imsi
AQSL Remote Comms : imsi
Gen Pop: msi
AQSL Local Comms : imsi
AQSL Local Comms : imsi
Gen Pop : imsi
Gen Pop : msi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : msi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : msi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : msi
Gen Pop : msi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : imsi
4 days of collect AQSL Local Comms : imsi
Gen Pop : imsi
1.0
0.8
0.6
0.4
0.2
0.0
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Now, we'll describe a cross validation experiment on the AQSL selectors that we were provided
* Behavioral Feature Extraction
* **Cross Validation Experiment on AQSL Couriers**
* Preliminary SIGINT Findings
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Our initial detector uses the centroid of the AQSL couriers to “find other selectors like these”
AQSL Cross-Validation Experiment
* 7 MSISDN/IMSI pairs
* Hold each pair out and score them when training the centroid on the rest
* Assume that random draws of Pakistani selectors are nontargets
* How well do we do?
```
miss probability (%)
99
95
80
60
40
20
5
1
0.1
0.01
1e-04
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
false alarm probability (%)
* Centroid(All Raw Features)
* Centroid(All Normalized Features)
* Centroid(Outgoing Raw Features)
* Centroid(Outgoing Normalized Features)
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Our initial detector uses the centroid of the AQSL couriers to “find other selectors like these”
AQSL Cross-Validation Experiment
* Initial experiments showed EER in 10-20% range
* Here, performance is much worse against these nontargets:
* Seen in Pakistan
* Not seen outside of Af/Pak
* Not FVEY selectors
```
miss probability (%)
99
95
80
60
40
20
5
1
0.1
0.01
1e-04
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
false alarm probability (%)
* Centroid(All Raw Features)
* Centroid(All Normalized Features)
* Centroid(Outgoing Raw Features)
* Centroid(Outgoing Normalized Features)
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Statistical algorithms are able to find the couriers at very low false alarm rates, if we're allowed to miss half of them
Random Forest Classifier
* 7 MSISDN/IMSI pairs
* Hold each pair out and then try to find them after learning how to distinguish remaining couriers from other Pakistanis (using 100k random selectors here)
* Assume that random draws of Pakistani selectors are nontargets
* 0.18% False Alarm Rate at 50% Miss Rate
```
miss probability (%)
99
95
80
60
40
20
5
1
0.1
0.01
1e-04
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
false alarm probability (%)
better
* Centroid(All Raw Features)
* Centroid(All Normalized Features)
* Centroid(Outgoing Raw Features)
* Centroid(Outgoing Normalized Features)
* Random Forest(All Raw Features)
* Random Forest (Outgoing Raw Features)
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## We've been experimenting with several error metrics on both small and large test sets
| | | | 100k Test Selectors | | 55M Test Selectors | |
|------------------------|------------------|----------|----------------------------------------|--------------------------|-----------------------------------|-----------------------------------|
| Training Data | Classifier | Features | False Alarm Rate at 50% Miss Rate | Mean Reciprocal Rank | Tasked Selectors in Top 500 | Tasked Selectors in Top 100 |
| None | Random | None | 50% | 1/23k (simulated) | 0.64 (active/Pak) | 0.13 (active/Pak) |
| Known Couriers | Centroid | All | 20% | 1/18k | | |
| | | Outgoing | 43% | 1/27k | | |
| | Random Forest | Outgoing | 0.18% | 1/9.9 | 5 | 1 |
| + Anchory Selectors | Random Forest | Outgoing | | | | |
Random Forest:
* 0.18% false alarm rate at 50% miss rate
* 7x improvement over random performance when evaluating its tasked precision at 100
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## To get more training data we scraped selectors from S2I11 Anchory reports containing keyword "courier”
Anchory Selectors
* Searched for reports containing "S2I11” AND "courier"
* Filtered out non-mobile numbers and kept selectors with "interesting” travel patterns seen in SmartTracker
```
Quetta
bad
Borth V tan
ahore
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Adding selectors from Anchory reports to the training data reduced the false alarm rates even further
Anchory Selectors
* Searched for reports containing "S2I11” AND "courier"
* Filtered out non-mobile numbers and kept selectors with "interesting” travel patterns seen in SmartTracker
```
miss probability (%)
99
95
80
60
40
20
5
1
0.1
0.01
1e-04
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
false alarm probability (%)
* Centroid(All Raw Features)
* Centroid(All Normalized Features)
* Centroid(Outgoing Raw Features)
* Centroid(Outgoing Normalized Features)
* Random Forest(All Raw Features)
* Random Forest (Outgoing Raw Features)
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## We've been experimenting with several error metrics on both small and large test sets
| | | | 100k Test Selectors | | 55M Test Selectors | |
|---------------------|---------------|----------|-----------------------------------|----------------------|---------------------------------|---------------------------------|
| Training Data | Classifier | Features | False Alarm Rate at 50% Miss Rate | Mean Reciprocal Rank | Tasked Selectors in Top 500 | Tasked Selectors in Top 100 |
| None | Random | None | 50% | 1/23k (simulated) | 0.64 (active/Pak) | 0.13 (active/Pak) |
| Known Couriers | Centroid | All | 20% | 1/18k | | |
| | | Outgoing | 43% | 1/27k | | |
| | Random Forest | Outgoing | 0.18% | 1/9.9 | 5 | 1 |
| + Anchory Selectors | Random Forest | Outgoing | 0.008% | 1/14 | 21 | 6 |
Random Forest trained on Known Couriers + Anchory Selectors:
* 0.008% false alarm rate at 50% miss rate
* 46x improvement over random performance when evaluating its tasked precision at 100
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Now, we'll investigate some findings after running these classifiers on +55M Pakistani selectors via MapReduce
* Behavioral Feature Extraction
* Cross Validation Experiment on AQSL Couriers
* **Preliminary SIGINT Findings**
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## The highest scoring selector that traveled to Peshawar and Lahore is PROB AHMED ZAIDAN
```
amabad
walpind
Khost
Miram Shah, North Wazirist
Wana, South Waziristan
Faisalaba
hore
Miles 100
Km 100 200 300
Path Legend
PROB AHMED MUWAFAK ZAIDAN
TIDE Person Number: [REDACTED]
* MEMBER OF AL-QA'IDA
* MEMBER OF MUSLIM BROTHERHOOD
* WORKS FOR AL JAZEERA
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## In the top 500 scoring selectors, 21 are tasked leading us to believe that we're on the right track
```
amabad
walpind
Shah North
Wana, South Waz n
hore
uetta
Miles 100
Km 100 200 300
Path Legend
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## We have also discovered many untasked selectors with interesting travel patterns
```
ost
iram Shah, North Waziristan
Wana, South Waziristan
ahore
Quetta
Karachi
Path Legend
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Preliminary results indicate that we're on the right track, but much remains to be done
Cross Validation Experiment:
* Random Forest classifier operating at 0.18% false alarm rate at 50% miss
* Enhancing training data with Anchory selectors reduced that to 0.008%
* Mean Reciprocal Rank is ~1/10
Preliminary SIGINT Findings:
* Behavioral features helped discover similar selectors with “courier-like" travel patterns
* High number of tasked selectors at the top is hopefully indicative of the detector performing well “in the wild”
```
miss probability (%)
99
95
80
60
40
20
5
1
0.1
0.01
1e-04
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
false alarm probability (%)
* Centroid(All Raw Features)
* Centroid(All Normalized Features)
* Centroid(Outgoing Raw Features)
* Centroid(Outgoing Normalized Features)
* Random Forest(All Raw Features)
* Random Forest (Outgoing Raw Features)
```
```
amabad
Wana, South Waz
quetta
Miles 100
Km 100 200 300
```
TOP SECRET//COMINT//REL TO USA, FVEY