first commit
This commit is contained in:
commit
0d90363116
6510 changed files with 679146 additions and 0 deletions
|
|
@ -0,0 +1,70 @@
|
|||
{
|
||||
"title": "SKYNET: Courier Detection via Machine Learning",
|
||||
"type": "REPORTED",
|
||||
"description": "A June 2012 NSA research presentation detailing the machine learning methodology behind SKYNET's courier detection system, which applies a Random Forest classifier to 55 million Pakistani phone records to identify individuals exhibiting courier-like behavioral patterns — with one of the highest-scoring results being Ahmed Zaidan, Al Jazeera journalist flagged as a suspected al-Qaeda courier.",
|
||||
"long_description": "This top secret NSA research presentation, dated June 5, 2012 and produced collaboratively by analysts from S2I5, R66F, and T1211, describes the technical methodology underlying SKYNET's automated courier detection capability. The core research question is whether a machine learning classifier trained on a small set of known al-Qaeda Senior Leadership (AQSL) courier selectors can identify behaviorally similar individuals from a population of approximately 55 million Pakistani phone subscribers — without relying on call chaining from known targets. The system extracts up to 80 behavioral features from GSM metadata including unique cell tower visit counts, travel frequency, incoming-to-outgoing call ratios, SIM and handset swapping behavior, and social network characteristics, then uses these features to distinguish couriers from the general Pakistani population.\n\nThe presentation documents a progression of classifier performance improvements. An initial centroid-based detector achieved an equal error rate of 10-20% in a cross-validation experiment on seven known courier selector pairs. Switching to a Random Forest classifier reduced the false alarm rate to 0.18% at 50% miss rate — a sevenfold improvement over random performance. Adding training data scraped from Anchory intelligence reports containing the keyword \"courier\" further reduced the false alarm rate to 0.008% (at a 50% miss rate) and gave a 46-fold improvement over random in tasked precision, with 21 of the top 500 scoring selectors being already-tasked known targets. The system was then run via MapReduce against the full 55 million selector dataset, with results assessed both quantitatively and through manual SIGINT review.\n\nThe presentation's most significant and controversial disclosure is the identification of Ahmed Zaidan — described in the document as a member of al-Qaeda, a member of the Muslim Brotherhood, and a journalist working for Al Jazeera — as the highest-scoring selector among those who traveled between Peshawar and Lahore. Zaidan was at the time Al Jazeera's Islamabad bureau chief, and his travel patterns to conflict zones as a journalist were apparently sufficient for SKYNET's algorithm to flag him as a probable courier. The document does not indicate any corroborating evidence beyond behavioral metadata, raising significant questions about the reliability of algorithmic targeting against a population of 55 million people at a 0.008% false alarm rate — which would still produce thousands of false positives at population scale.",
|
||||
"tags": [
|
||||
"journalism",
|
||||
"Al Jazeera",
|
||||
"Al-Qaeda",
|
||||
"machine learning",
|
||||
"courier tracking",
|
||||
"behavioral profiling",
|
||||
"pattern-of-life",
|
||||
"GSM",
|
||||
"call detail records",
|
||||
"AQSL",
|
||||
"courier network",
|
||||
"journalist surveillance",
|
||||
"IMSI",
|
||||
"MSISDN",
|
||||
"cell towers",
|
||||
"travel analytics",
|
||||
"social network analysis",
|
||||
"Anchory",
|
||||
"targeting"
|
||||
],
|
||||
"document": {
|
||||
"creator": "NSA",
|
||||
"date_created": "2012-06-02",
|
||||
"classification": [
|
||||
"TOP SECRET",
|
||||
"COMINT"
|
||||
],
|
||||
"dissemination": [
|
||||
"REL TO USA, FVEY"
|
||||
],
|
||||
"banner": "TOP SECRET//COMINT//REL TO USA, FVEY",
|
||||
"genre": "slides",
|
||||
"programs": [
|
||||
"SMARTTRACKER",
|
||||
"MAPREDUCE",
|
||||
"SKYNET"
|
||||
],
|
||||
"countries": [
|
||||
"Pakistan",
|
||||
"Afghanistan"
|
||||
],
|
||||
"people": [
|
||||
"Zaidan, Ahmad Muaffaq"
|
||||
]
|
||||
},
|
||||
"publication": [
|
||||
{
|
||||
"publisher": "The Intercept",
|
||||
"date_published": "2015-05-08",
|
||||
"title": "U.S. Government Designated Prominent Al Jazeera Journalist as \"Member of Al Qaeda\"",
|
||||
"link": "https://theintercept.com/2015/05/08/u-s-government-designated-prominent-al-jazeera-journalist-al-qaeda-member-put-watch-list/",
|
||||
"reporters": [
|
||||
"Currier, Cora",
|
||||
"Greenwald, Glenn",
|
||||
"Fishman, Andrew"
|
||||
],
|
||||
"contributors": [],
|
||||
"doc_source": "https://theintercept.com/document/2015/05/08/skynet-courier/",
|
||||
"doc_url": "https://s3.documentcloud.org/documents/2074620/skynet-courier-detection-via-machine-learning.pdf"
|
||||
}
|
||||
],
|
||||
"pdf_sha256": "d64b2a68973e894fb1564d7ee8b691362a97f8812508341464d3d2ded20fa34c",
|
||||
"pages": 20
|
||||
}
|
||||
|
|
@ -0,0 +1,620 @@
|
|||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
# SKYNET:
|
||||
## Courier Detection via Machine Learning
|
||||
|
||||
* [REDACTED], R66F/JHU
|
||||
* [REDACTED], R66F
|
||||
* [REDACTED], R66F
|
||||
* [REDACTED], T1211
|
||||
* [REDACTED], T1211
|
||||
* [REDACTED], S2I5
|
||||
* [REDACTED], S2I5/TD
|
||||
|
||||
June 5, 2012
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Given a handful of courier selectors, can we find others that “behave similarly” by analyzing GSM metadata?
|
||||
|
||||
```
|
||||
OPR249(I)
|
||||
SAIDU SHARIF
|
||||
NDB
|
||||
JALALABAD
|
||||
15600
|
||||
NDB
|
||||
ARDEZ
|
||||
640-2°E
|
||||
Khost
|
||||
BANNU
|
||||
Miram Shah, North Waziristan
|
||||
11770
|
||||
Wana, South Waziristan
|
||||
DERA ISMAIL KHAN
|
||||
13000
|
||||
OPR217
|
||||
OPR203
|
||||
NDB
|
||||
CHARSADDA
|
||||
PESHAWAR
|
||||
OPR205
|
||||
TARBELA
|
||||
DAM
|
||||
HARIPUR
|
||||
NHAS
|
||||
OPR210
|
||||
OPR202
|
||||
KOHAT
|
||||
MARI INDUS EAS
|
||||
OPR217
|
||||
NDB
|
||||
DARWALI
|
||||
OPR203
|
||||
4992
|
||||
OPR216
|
||||
SARGODHA
|
||||
OPR230
|
||||
DERA ISMAIL KHAN
|
||||
OPR218
|
||||
JHANG SADR
|
||||
OPR253
|
||||
7980
|
||||
OPR251
|
||||
RANQUI
|
||||
MULTAN
|
||||
VEHARI
|
||||
Abbottabad
|
||||
Islamabad
|
||||
Rawalpindi
|
||||
OPR277
|
||||
RAWALPINDI
|
||||
TAXILA
|
||||
CHAKIALA
|
||||
NDB
|
||||
BANIHA
|
||||
OPR235
|
||||
ABANDONED
|
||||
RAHWALI
|
||||
NDB
|
||||
SHEIKRUF
|
||||
Faisalabad
|
||||
DME
|
||||
LAHORE
|
||||
NDB
|
||||
FAISALABAD
|
||||
OKARA
|
||||
Lahore
|
||||
Sahiwal
|
||||
Miles 20 40 60 80
|
||||
Km 100
|
||||
CEASE
|
||||
|
||||
Paths Legend
|
||||
|
||||
It's worth noting that:
|
||||
* we are looking for different people using phones in similar ways
|
||||
* without using any call chaining techniques from known selectors
|
||||
* by scanning through all selectors seen in Pakistan that have not left Af/Pak (~55M)
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## From GSM metadata, we can measure aspects of each selector's pattern-of-life, social network, and travel behavior
|
||||
|
||||
```
|
||||
Path Legend
|
||||
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
|
||||
Sunday
|
||||
Monday
|
||||
Tuesday
|
||||
Wednesday
|
||||
Thursday
|
||||
Friday
|
||||
Saturday
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## This presentation describes our search for AQSL couriers using behavioral profiling
|
||||
|
||||
* Behavioral Feature Extraction
|
||||
* Cross Validation Experiment on AQSL Couriers
|
||||
* Preliminary SIGINT Findings
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Counting unique UCELLIDs shows that couriers travel more often than typical Pakistani selectors
|
||||
|
||||
```
|
||||
density
|
||||
0.06
|
||||
0.05
|
||||
0.04
|
||||
0.03
|
||||
0.02
|
||||
0.01
|
||||
0.00
|
||||
|
||||
0 20 40 60 80 100 120
|
||||
Number of Unique CellIDs
|
||||
|
||||
Group
|
||||
AQSL Local Comms
|
||||
AQSL Remote Comms
|
||||
Seen in Pakistan
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## By examining multiple features at once, we can see some indicative behaviors of our courier selectors
|
||||
|
||||
```
|
||||
Average Daily Outgoing Voice
|
||||
10
|
||||
8
|
||||
6
|
||||
4
|
||||
2
|
||||
0
|
||||
|
||||
Average Daily Incoming Voice
|
||||
0 2 4 6 8 10
|
||||
|
||||
numCell
|
||||
0
|
||||
20
|
||||
40
|
||||
60
|
||||
80
|
||||
100
|
||||
120
|
||||
|
||||
Group
|
||||
Seen in Pakistan
|
||||
AQSL Local Comms
|
||||
AQSL Remote Comms
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Looking at a hierarchical clustering derived from all 80 features, the AQSL groups mostly stay together
|
||||
|
||||
```
|
||||
Gen Pop : msi
|
||||
Gen Pop : imsi
|
||||
Gen Pop : imsi
|
||||
AQSL Remote Comms : imsi
|
||||
AQSL Remote Comms : imsi
|
||||
AQSL Remote Comms : imsi
|
||||
AQSL Remote Comms : imsi
|
||||
Gen Pop: msi
|
||||
AQSL Local Comms : imsi
|
||||
AQSL Local Comms : imsi
|
||||
Gen Pop : imsi
|
||||
Gen Pop : msi
|
||||
Gen Pop : imsi
|
||||
Gen Pop : imsi
|
||||
Gen Pop : imsi
|
||||
Gen Pop : msi
|
||||
Gen Pop : imsi
|
||||
Gen Pop : imsi
|
||||
Gen Pop : imsi
|
||||
Gen Pop : msi
|
||||
Gen Pop : imsi
|
||||
Gen Pop : imsi
|
||||
Gen Pop : imsi
|
||||
Gen Pop : imsi
|
||||
Gen Pop : msi
|
||||
Gen Pop : msi
|
||||
Gen Pop : imsi
|
||||
Gen Pop : imsi
|
||||
Gen Pop : imsi
|
||||
Gen Pop : imsi
|
||||
4 days of collect AQSL Local Comms : imsi
|
||||
Gen Pop : imsi
|
||||
|
||||
1.0
|
||||
0.8
|
||||
0.6
|
||||
0.4
|
||||
0.2
|
||||
0.0
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Now, we'll describe a cross validation experiment on the AQSL selectors that we were provided
|
||||
|
||||
* Behavioral Feature Extraction
|
||||
* **Cross Validation Experiment on AQSL Couriers**
|
||||
* Preliminary SIGINT Findings
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Our initial detector uses the centroid of the AQSL couriers to “find other selectors like these”
|
||||
|
||||
AQSL Cross-Validation Experiment
|
||||
|
||||
* 7 MSISDN/IMSI pairs
|
||||
* Hold each pair out and score them when training the centroid on the rest
|
||||
* Assume that random draws of Pakistani selectors are nontargets
|
||||
* How well do we do?
|
||||
|
||||
```
|
||||
miss probability (%)
|
||||
99
|
||||
95
|
||||
80
|
||||
60
|
||||
40
|
||||
20
|
||||
5
|
||||
1
|
||||
0.1
|
||||
0.01
|
||||
1e-04
|
||||
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
|
||||
false alarm probability (%)
|
||||
|
||||
* Centroid(All Raw Features)
|
||||
* Centroid(All Normalized Features)
|
||||
* Centroid(Outgoing Raw Features)
|
||||
* Centroid(Outgoing Normalized Features)
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Our initial detector uses the centroid of the AQSL couriers to “find other selectors like these”
|
||||
|
||||
AQSL Cross-Validation Experiment
|
||||
|
||||
* Initial experiments showed EER in 10-20% range
|
||||
* Here, performance is much worse against these nontargets:
|
||||
* Seen in Pakistan
|
||||
* Not seen outside of Af/Pak
|
||||
* Not FVEY selectors
|
||||
|
||||
```
|
||||
miss probability (%)
|
||||
99
|
||||
95
|
||||
80
|
||||
60
|
||||
40
|
||||
20
|
||||
5
|
||||
1
|
||||
0.1
|
||||
0.01
|
||||
1e-04
|
||||
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
|
||||
false alarm probability (%)
|
||||
|
||||
* Centroid(All Raw Features)
|
||||
* Centroid(All Normalized Features)
|
||||
* Centroid(Outgoing Raw Features)
|
||||
* Centroid(Outgoing Normalized Features)
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Statistical algorithms are able to find the couriers at very low false alarm rates, if we're allowed to miss half of them
|
||||
|
||||
Random Forest Classifier
|
||||
|
||||
* 7 MSISDN/IMSI pairs
|
||||
* Hold each pair out and then try to find them after learning how to distinguish remaining couriers from other Pakistanis (using 100k random selectors here)
|
||||
* Assume that random draws of Pakistani selectors are nontargets
|
||||
* 0.18% False Alarm Rate at 50% Miss Rate
|
||||
|
||||
```
|
||||
miss probability (%)
|
||||
99
|
||||
95
|
||||
80
|
||||
60
|
||||
40
|
||||
20
|
||||
5
|
||||
1
|
||||
0.1
|
||||
0.01
|
||||
1e-04
|
||||
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
|
||||
false alarm probability (%)
|
||||
|
||||
better
|
||||
|
||||
* Centroid(All Raw Features)
|
||||
* Centroid(All Normalized Features)
|
||||
* Centroid(Outgoing Raw Features)
|
||||
* Centroid(Outgoing Normalized Features)
|
||||
* Random Forest(All Raw Features)
|
||||
* Random Forest (Outgoing Raw Features)
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## We've been experimenting with several error metrics on both small and large test sets
|
||||
|
||||
| | | | 100k Test Selectors | | 55M Test Selectors | |
|
||||
|------------------------|------------------|----------|----------------------------------------|--------------------------|-----------------------------------|-----------------------------------|
|
||||
| Training Data | Classifier | Features | False Alarm Rate at 50% Miss Rate | Mean Reciprocal Rank | Tasked Selectors in Top 500 | Tasked Selectors in Top 100 |
|
||||
| None | Random | None | 50% | 1/23k (simulated) | 0.64 (active/Pak) | 0.13 (active/Pak) |
|
||||
| Known Couriers | Centroid | All | 20% | 1/18k | | |
|
||||
| | | Outgoing | 43% | 1/27k | | |
|
||||
| | Random Forest | Outgoing | 0.18% | 1/9.9 | 5 | 1 |
|
||||
| + Anchory Selectors | Random Forest | Outgoing | | | | |
|
||||
|
||||
Random Forest:
|
||||
* 0.18% false alarm rate at 50% miss rate
|
||||
* 7x improvement over random performance when evaluating its tasked precision at 100
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## To get more training data we scraped selectors from S2I11 Anchory reports containing keyword "courier”
|
||||
|
||||
Anchory Selectors
|
||||
|
||||
* Searched for reports containing "S2I11” AND "courier"
|
||||
* Filtered out non-mobile numbers and kept selectors with "interesting” travel patterns seen in SmartTracker
|
||||
|
||||
```
|
||||
Quetta
|
||||
bad
|
||||
Borth V tan
|
||||
ahore
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Adding selectors from Anchory reports to the training data reduced the false alarm rates even further
|
||||
|
||||
Anchory Selectors
|
||||
|
||||
* Searched for reports containing "S2I11” AND "courier"
|
||||
* Filtered out non-mobile numbers and kept selectors with "interesting” travel patterns seen in SmartTracker
|
||||
|
||||
```
|
||||
miss probability (%)
|
||||
99
|
||||
95
|
||||
80
|
||||
60
|
||||
40
|
||||
20
|
||||
5
|
||||
1
|
||||
0.1
|
||||
0.01
|
||||
1e-04
|
||||
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
|
||||
false alarm probability (%)
|
||||
|
||||
* Centroid(All Raw Features)
|
||||
* Centroid(All Normalized Features)
|
||||
* Centroid(Outgoing Raw Features)
|
||||
* Centroid(Outgoing Normalized Features)
|
||||
* Random Forest(All Raw Features)
|
||||
* Random Forest (Outgoing Raw Features)
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## We've been experimenting with several error metrics on both small and large test sets
|
||||
|
||||
| | | | 100k Test Selectors | | 55M Test Selectors | |
|
||||
|---------------------|---------------|----------|-----------------------------------|----------------------|---------------------------------|---------------------------------|
|
||||
| Training Data | Classifier | Features | False Alarm Rate at 50% Miss Rate | Mean Reciprocal Rank | Tasked Selectors in Top 500 | Tasked Selectors in Top 100 |
|
||||
| None | Random | None | 50% | 1/23k (simulated) | 0.64 (active/Pak) | 0.13 (active/Pak) |
|
||||
| Known Couriers | Centroid | All | 20% | 1/18k | | |
|
||||
| | | Outgoing | 43% | 1/27k | | |
|
||||
| | Random Forest | Outgoing | 0.18% | 1/9.9 | 5 | 1 |
|
||||
| + Anchory Selectors | Random Forest | Outgoing | 0.008% | 1/14 | 21 | 6 |
|
||||
|
||||
Random Forest trained on Known Couriers + Anchory Selectors:
|
||||
* 0.008% false alarm rate at 50% miss rate
|
||||
* 46x improvement over random performance when evaluating its tasked precision at 100
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Now, we'll investigate some findings after running these classifiers on +55M Pakistani selectors via MapReduce
|
||||
|
||||
* Behavioral Feature Extraction
|
||||
* Cross Validation Experiment on AQSL Couriers
|
||||
* **Preliminary SIGINT Findings**
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## The highest scoring selector that traveled to Peshawar and Lahore is PROB AHMED ZAIDAN
|
||||
|
||||
```
|
||||
amabad
|
||||
walpind
|
||||
Khost
|
||||
Miram Shah, North Wazirist
|
||||
Wana, South Waziristan
|
||||
Faisalaba
|
||||
hore
|
||||
|
||||
Miles 100
|
||||
Km 100 200 300
|
||||
|
||||
Path Legend
|
||||
PROB AHMED MUWAFAK ZAIDAN
|
||||
TIDE Person Number: [REDACTED]
|
||||
* MEMBER OF AL-QA'IDA
|
||||
* MEMBER OF MUSLIM BROTHERHOOD
|
||||
* WORKS FOR AL JAZEERA
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## In the top 500 scoring selectors, 21 are tasked leading us to believe that we're on the right track
|
||||
|
||||
```
|
||||
amabad
|
||||
walpind
|
||||
Shah North
|
||||
Wana, South Waz n
|
||||
hore
|
||||
uetta
|
||||
|
||||
Miles 100
|
||||
Km 100 200 300
|
||||
|
||||
Path Legend
|
||||
[REDACTED]
|
||||
[REDACTED]
|
||||
[REDACTED]
|
||||
[REDACTED]
|
||||
[REDACTED]
|
||||
[REDACTED]
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## We have also discovered many untasked selectors with interesting travel patterns
|
||||
|
||||
```
|
||||
ost
|
||||
iram Shah, North Waziristan
|
||||
Wana, South Waziristan
|
||||
ahore
|
||||
Quetta
|
||||
Karachi
|
||||
|
||||
Path Legend
|
||||
[REDACTED]
|
||||
[REDACTED]
|
||||
[REDACTED]
|
||||
[REDACTED]
|
||||
[REDACTED]
|
||||
[REDACTED]
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Preliminary results indicate that we're on the right track, but much remains to be done
|
||||
|
||||
Cross Validation Experiment:
|
||||
|
||||
* Random Forest classifier operating at 0.18% false alarm rate at 50% miss
|
||||
* Enhancing training data with Anchory selectors reduced that to 0.008%
|
||||
* Mean Reciprocal Rank is ~1/10
|
||||
|
||||
Preliminary SIGINT Findings:
|
||||
|
||||
* Behavioral features helped discover similar selectors with “courier-like" travel patterns
|
||||
* High number of tasked selectors at the top is hopefully indicative of the detector performing well “in the wild”
|
||||
|
||||
```
|
||||
miss probability (%)
|
||||
99
|
||||
95
|
||||
80
|
||||
60
|
||||
40
|
||||
20
|
||||
5
|
||||
1
|
||||
0.1
|
||||
0.01
|
||||
1e-04
|
||||
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
|
||||
false alarm probability (%)
|
||||
|
||||
* Centroid(All Raw Features)
|
||||
* Centroid(All Normalized Features)
|
||||
* Centroid(Outgoing Raw Features)
|
||||
* Centroid(Outgoing Normalized Features)
|
||||
* Random Forest(All Raw Features)
|
||||
* Random Forest (Outgoing Raw Features)
|
||||
```
|
||||
|
||||
```
|
||||
amabad
|
||||
Wana, South Waz
|
||||
quetta
|
||||
Miles 100
|
||||
Km 100 200 300
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
Loading…
Add table
Add a link
Reference in a new issue