TOP SECRET//COMINT//REL TO USA, FVEY # SKYNET: ## Courier Detection via Machine Learning * [REDACTED], R66F/JHU * [REDACTED], R66F * [REDACTED], R66F * [REDACTED], T1211 * [REDACTED], T1211 * [REDACTED], S2I5 * [REDACTED], S2I5/TD June 5, 2012 TOP SECRET//COMINT//REL TO USA, FVEY --- TOP SECRET//COMINT//REL TO USA, FVEY ## Given a handful of courier selectors, can we find others that “behave similarly” by analyzing GSM metadata? ``` OPR249(I) SAIDU SHARIF NDB JALALABAD 15600 NDB ARDEZ 640-2°E Khost BANNU Miram Shah, North Waziristan 11770 Wana, South Waziristan DERA ISMAIL KHAN 13000 OPR217 OPR203 NDB CHARSADDA PESHAWAR OPR205 TARBELA DAM HARIPUR NHAS OPR210 OPR202 KOHAT MARI INDUS EAS OPR217 NDB DARWALI OPR203 4992 OPR216 SARGODHA OPR230 DERA ISMAIL KHAN OPR218 JHANG SADR OPR253 7980 OPR251 RANQUI MULTAN VEHARI Abbottabad Islamabad Rawalpindi OPR277 RAWALPINDI TAXILA CHAKIALA NDB BANIHA OPR235 ABANDONED RAHWALI NDB SHEIKRUF Faisalabad DME LAHORE NDB FAISALABAD OKARA Lahore Sahiwal Miles 20 40 60 80 Km 100 CEASE Paths Legend It's worth noting that: * we are looking for different people using phones in similar ways * without using any call chaining techniques from known selectors * by scanning through all selectors seen in Pakistan that have not left Af/Pak (~55M) ``` TOP SECRET//COMINT//REL TO USA, FVEY --- TOP SECRET//COMINT//REL TO USA, FVEY ## From GSM metadata, we can measure aspects of each selector's pattern-of-life, social network, and travel behavior ``` Path Legend 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Sunday Monday Tuesday Wednesday Thursday Friday Saturday ``` TOP SECRET//COMINT//REL TO USA, FVEY --- TOP SECRET//COMINT//REL TO USA, FVEY ## This presentation describes our search for AQSL couriers using behavioral profiling * Behavioral Feature Extraction * Cross Validation Experiment on AQSL Couriers * Preliminary SIGINT Findings TOP SECRET//COMINT//REL TO USA, FVEY --- TOP SECRET//COMINT//REL TO USA, FVEY ## Counting unique UCELLIDs shows that couriers travel more often than typical Pakistani selectors ``` density 0.06 0.05 0.04 0.03 0.02 0.01 0.00 0 20 40 60 80 100 120 Number of Unique CellIDs Group AQSL Local Comms AQSL Remote Comms Seen in Pakistan ``` TOP SECRET//COMINT//REL TO USA, FVEY --- TOP SECRET//COMINT//REL TO USA, FVEY ## By examining multiple features at once, we can see some indicative behaviors of our courier selectors ``` Average Daily Outgoing Voice 10 8 6 4 2 0 Average Daily Incoming Voice 0 2 4 6 8 10 numCell 0 20 40 60 80 100 120 Group Seen in Pakistan AQSL Local Comms AQSL Remote Comms ``` TOP SECRET//COMINT//REL TO USA, FVEY --- TOP SECRET//COMINT//REL TO USA, FVEY ## Looking at a hierarchical clustering derived from all 80 features, the AQSL groups mostly stay together ``` Gen Pop : msi Gen Pop : imsi Gen Pop : imsi AQSL Remote Comms : imsi AQSL Remote Comms : imsi AQSL Remote Comms : imsi AQSL Remote Comms : imsi Gen Pop: msi AQSL Local Comms : imsi AQSL Local Comms : imsi Gen Pop : imsi Gen Pop : msi Gen Pop : imsi Gen Pop : imsi Gen Pop : imsi Gen Pop : msi Gen Pop : imsi Gen Pop : imsi Gen Pop : imsi Gen Pop : msi Gen Pop : imsi Gen Pop : imsi Gen Pop : imsi Gen Pop : imsi Gen Pop : msi Gen Pop : msi Gen Pop : imsi Gen Pop : imsi Gen Pop : imsi Gen Pop : imsi 4 days of collect AQSL Local Comms : imsi Gen Pop : imsi 1.0 0.8 0.6 0.4 0.2 0.0 ``` TOP SECRET//COMINT//REL TO USA, FVEY --- TOP SECRET//COMINT//REL TO USA, FVEY ## Now, we'll describe a cross validation experiment on the AQSL selectors that we were provided * Behavioral Feature Extraction * **Cross Validation Experiment on AQSL Couriers** * Preliminary SIGINT Findings TOP SECRET//COMINT//REL TO USA, FVEY --- TOP SECRET//COMINT//REL TO USA, FVEY ## Our initial detector uses the centroid of the AQSL couriers to “find other selectors like these” AQSL Cross-Validation Experiment * 7 MSISDN/IMSI pairs * Hold each pair out and score them when training the centroid on the rest * Assume that random draws of Pakistani selectors are nontargets * How well do we do? ``` miss probability (%) 99 95 80 60 40 20 5 1 0.1 0.01 1e-04 1e-04 0.01 0.1 1 5 20 40 60 80 95 99 false alarm probability (%) * Centroid(All Raw Features) * Centroid(All Normalized Features) * Centroid(Outgoing Raw Features) * Centroid(Outgoing Normalized Features) ``` TOP SECRET//COMINT//REL TO USA, FVEY --- TOP SECRET//COMINT//REL TO USA, FVEY ## Our initial detector uses the centroid of the AQSL couriers to “find other selectors like these” AQSL Cross-Validation Experiment * Initial experiments showed EER in 10-20% range * Here, performance is much worse against these nontargets: * Seen in Pakistan * Not seen outside of Af/Pak * Not FVEY selectors ``` miss probability (%) 99 95 80 60 40 20 5 1 0.1 0.01 1e-04 1e-04 0.01 0.1 1 5 20 40 60 80 95 99 false alarm probability (%) * Centroid(All Raw Features) * Centroid(All Normalized Features) * Centroid(Outgoing Raw Features) * Centroid(Outgoing Normalized Features) ``` TOP SECRET//COMINT//REL TO USA, FVEY --- TOP SECRET//COMINT//REL TO USA, FVEY ## Statistical algorithms are able to find the couriers at very low false alarm rates, if we're allowed to miss half of them Random Forest Classifier * 7 MSISDN/IMSI pairs * Hold each pair out and then try to find them after learning how to distinguish remaining couriers from other Pakistanis (using 100k random selectors here) * Assume that random draws of Pakistani selectors are nontargets * 0.18% False Alarm Rate at 50% Miss Rate ``` miss probability (%) 99 95 80 60 40 20 5 1 0.1 0.01 1e-04 1e-04 0.01 0.1 1 5 20 40 60 80 95 99 false alarm probability (%) better * Centroid(All Raw Features) * Centroid(All Normalized Features) * Centroid(Outgoing Raw Features) * Centroid(Outgoing Normalized Features) * Random Forest(All Raw Features) * Random Forest (Outgoing Raw Features) ``` TOP SECRET//COMINT//REL TO USA, FVEY --- TOP SECRET//COMINT//REL TO USA, FVEY ## We've been experimenting with several error metrics on both small and large test sets | | | | 100k Test Selectors | | 55M Test Selectors | | |------------------------|------------------|----------|----------------------------------------|--------------------------|-----------------------------------|-----------------------------------| | Training Data | Classifier | Features | False Alarm Rate at 50% Miss Rate | Mean Reciprocal Rank | Tasked Selectors in Top 500 | Tasked Selectors in Top 100 | | None | Random | None | 50% | 1/23k (simulated) | 0.64 (active/Pak) | 0.13 (active/Pak) | | Known Couriers | Centroid | All | 20% | 1/18k | | | | | | Outgoing | 43% | 1/27k | | | | | Random Forest | Outgoing | 0.18% | 1/9.9 | 5 | 1 | | + Anchory Selectors | Random Forest | Outgoing | | | | | Random Forest: * 0.18% false alarm rate at 50% miss rate * 7x improvement over random performance when evaluating its tasked precision at 100 TOP SECRET//COMINT//REL TO USA, FVEY --- TOP SECRET//COMINT//REL TO USA, FVEY ## To get more training data we scraped selectors from S2I11 Anchory reports containing keyword "courier” Anchory Selectors * Searched for reports containing "S2I11” AND "courier" * Filtered out non-mobile numbers and kept selectors with "interesting” travel patterns seen in SmartTracker ``` Quetta bad Borth V tan ahore ``` TOP SECRET//COMINT//REL TO USA, FVEY --- TOP SECRET//COMINT//REL TO USA, FVEY ## Adding selectors from Anchory reports to the training data reduced the false alarm rates even further Anchory Selectors * Searched for reports containing "S2I11” AND "courier" * Filtered out non-mobile numbers and kept selectors with "interesting” travel patterns seen in SmartTracker ``` miss probability (%) 99 95 80 60 40 20 5 1 0.1 0.01 1e-04 1e-04 0.01 0.1 1 5 20 40 60 80 95 99 false alarm probability (%) * Centroid(All Raw Features) * Centroid(All Normalized Features) * Centroid(Outgoing Raw Features) * Centroid(Outgoing Normalized Features) * Random Forest(All Raw Features) * Random Forest (Outgoing Raw Features) ``` TOP SECRET//COMINT//REL TO USA, FVEY --- TOP SECRET//COMINT//REL TO USA, FVEY ## We've been experimenting with several error metrics on both small and large test sets | | | | 100k Test Selectors | | 55M Test Selectors | | |---------------------|---------------|----------|-----------------------------------|----------------------|---------------------------------|---------------------------------| | Training Data | Classifier | Features | False Alarm Rate at 50% Miss Rate | Mean Reciprocal Rank | Tasked Selectors in Top 500 | Tasked Selectors in Top 100 | | None | Random | None | 50% | 1/23k (simulated) | 0.64 (active/Pak) | 0.13 (active/Pak) | | Known Couriers | Centroid | All | 20% | 1/18k | | | | | | Outgoing | 43% | 1/27k | | | | | Random Forest | Outgoing | 0.18% | 1/9.9 | 5 | 1 | | + Anchory Selectors | Random Forest | Outgoing | 0.008% | 1/14 | 21 | 6 | Random Forest trained on Known Couriers + Anchory Selectors: * 0.008% false alarm rate at 50% miss rate * 46x improvement over random performance when evaluating its tasked precision at 100 TOP SECRET//COMINT//REL TO USA, FVEY --- TOP SECRET//COMINT//REL TO USA, FVEY ## Now, we'll investigate some findings after running these classifiers on +55M Pakistani selectors via MapReduce * Behavioral Feature Extraction * Cross Validation Experiment on AQSL Couriers * **Preliminary SIGINT Findings** TOP SECRET//COMINT//REL TO USA, FVEY --- TOP SECRET//COMINT//REL TO USA, FVEY ## The highest scoring selector that traveled to Peshawar and Lahore is PROB AHMED ZAIDAN ``` amabad walpind Khost Miram Shah, North Wazirist Wana, South Waziristan Faisalaba hore Miles 100 Km 100 200 300 Path Legend PROB AHMED MUWAFAK ZAIDAN TIDE Person Number: [REDACTED] * MEMBER OF AL-QA'IDA * MEMBER OF MUSLIM BROTHERHOOD * WORKS FOR AL JAZEERA ``` TOP SECRET//COMINT//REL TO USA, FVEY --- TOP SECRET//COMINT//REL TO USA, FVEY ## In the top 500 scoring selectors, 21 are tasked leading us to believe that we're on the right track ``` amabad walpind Shah North Wana, South Waz n hore uetta Miles 100 Km 100 200 300 Path Legend [REDACTED] [REDACTED] [REDACTED] [REDACTED] [REDACTED] [REDACTED] ``` TOP SECRET//COMINT//REL TO USA, FVEY --- TOP SECRET//COMINT//REL TO USA, FVEY ## We have also discovered many untasked selectors with interesting travel patterns ``` ost iram Shah, North Waziristan Wana, South Waziristan ahore Quetta Karachi Path Legend [REDACTED] [REDACTED] [REDACTED] [REDACTED] [REDACTED] [REDACTED] ``` TOP SECRET//COMINT//REL TO USA, FVEY --- TOP SECRET//COMINT//REL TO USA, FVEY ## Preliminary results indicate that we're on the right track, but much remains to be done Cross Validation Experiment: * Random Forest classifier operating at 0.18% false alarm rate at 50% miss * Enhancing training data with Anchory selectors reduced that to 0.008% * Mean Reciprocal Rank is ~1/10 Preliminary SIGINT Findings: * Behavioral features helped discover similar selectors with “courier-like" travel patterns * High number of tasked selectors at the top is hopefully indicative of the detector performing well “in the wild” ``` miss probability (%) 99 95 80 60 40 20 5 1 0.1 0.01 1e-04 1e-04 0.01 0.1 1 5 20 40 60 80 95 99 false alarm probability (%) * Centroid(All Raw Features) * Centroid(All Normalized Features) * Centroid(Outgoing Raw Features) * Centroid(Outgoing Normalized Features) * Random Forest(All Raw Features) * Random Forest (Outgoing Raw Features) ``` ``` amabad Wana, South Waz quetta Miles 100 Km 100 200 300 ``` TOP SECRET//COMINT//REL TO USA, FVEY