{ "title": "HIMR Data Mining Research Problem Book", "type": "REPORTED", "description": "This document is a research roadmap created by GCHQ for the Heilbronn Institute for Mathematical Research (HIMR). Its purpose is to guide long-term mathematical and data-mining research that could improve signals intelligence (SIGINT) capabilities. It outlines key research problems, datasets, and collaboration plans between intelligence agencies and academic researchers. The main goal is to develop new algorithms and mathematical tools to analyze extremely large communications datasets produced by modern surveillance systems.", "long_description": "The document is a research agenda produced for the Heilbronn Institute for Mathematical Research (HIMR) in collaboration with the Government Communications Headquarters (GCHQ) and other intelligence partners. It outlines a set of mathematical and data-mining problems intended to help intelligence analysts interpret the enormous volumes of communications data collected through signals intelligence systems. Rather than focusing on reading individual messages, the research emphasizes machine learning, graph theory, and large-scale statistical analysis to detect patterns, relationships, and anomalies within global communication networks.\n\nA central theme of the document is the shift in intelligence work from monitoring known suspects toward discovering previously unknown targets within massive datasets. Because modern interception systems capture vast streams of internet and telecommunications traffic, the challenge becomes identifying meaningful signals within this noise. The research therefore focuses on techniques such as semi-supervised learning, anomaly detection, and dynamic graph analysis that could automatically highlight unusual communication structures, clusters, or flows of information.\n\nOne of the more unsettling sections discusses the different legal and practical treatment of message content versus metadata. The document explains that “there are extremely stringent legal and policy constraints on what we can do with content,” while agencies are “much freer in how we can store and use metadata.” It also notes that the volume of raw message content is far larger than metadata, making it difficult to retain or process. As a result, metadata feeds are often collected without selection, meaning systems “pull everything we see” and store it for later analysis.\n\nOverall, the paper shows how intelligence agencies sought to develop automated tools capable of analyzing enormous metadata collections in real time. By modeling communications as evolving graphs and searching for statistical irregularities or cascading information flows, these systems aim to reveal hidden social networks, suspicious behavioral patterns, and emerging cyber threats. The research agenda therefore represents an attempt to transform large-scale communications interception into a form of computational pattern detection capable of identifying intelligence targets within global data streams.", "tags": [ "Tor", "botnets", "intelligence analysis", "surveillance infrastructure", "surveillance architecture", "communications interception", "network interception", "internet backbone", "cable tapping", "fiber optic cables", "telecommunications surveillance", "internet traffic interception", "packet capture", "network monitoring", "bearers", "network infrastructure", "data interception", "metadata collection", "metadata analysis", "communications metadata", "call detail records", "traffic analysis", "communications network", "social network analysis", "graph theory", "dynamic graphs", "streaming graphs", "streaming data", "real-time analysis", "big data", "large-scale data mining", "machine learning", "semi-supervised learning", "weakly labeled data", "active learning algorithms", "anomaly detection", "pattern detection", "behavioral analysis", "statistical modeling", "probabilistic models", "classification algorithms", "random forests", "clustering algorithms", "exploratory data analysis", "streaming exploratory analysis", "data reduction", "MVR", "packet filtering", "metadata processing", "intelligence infrastructure", "target discovery", "target identification", "communication cascade detection", "information cascades", "network analysis", "link analysis", "community detection", "centrality", "temporal network analysis", "evolving communication graphs", "command and control detection", "malware", "implants", "beaconing", "Computer Network Exploitation", "Computer Network Attack", "Computer Network Defense", "steganography", "spam", "protocol analysis", "website classification", "logo recognition", "applied mathematics research", "mathematical research agenda", "cryptography", "academia", "research", "data science", "data processing", "graph analysis" ], "document": { "creator": "GCHQ", "date_created": "2011-09-20", "classification": [ "TOP SECRET", "STRAP 1", "COMINT" ], "dissemination": [ "AUS,CAN,NZ,UK, US" ], "banner": "UK TOP SECRET STRAP1 COMINT AUS/CAN/NZ/UK/US EYES ONLY", "genre": "report", "programs": [ "CLOUD", "DISTILLERY", "DISCOVER", "PRESTON", "LECKWITH", "TERRAIN", "BROADOAK", "NGE", "FIVEALIVE", "CHARTBREAKER", "FIRSTCONTACT", "GEOFUSION", "RADONSHARPEN-B", "PRIMETIME", "HIDDENOTTER", "BAKERSDOZEN", "CLASP", "SALTYOTTER", "BIRCH", "CHARTBREAKER", "SOLIDINK", "FLUIDINK", "MAMBA", "GRINNINGROACH", "PIRATECAREBEAR", "AUTOASSOC", "CROUCHINGSQUIRREL", "HRMAP", "KACHINA", "SQUEAL", "QUESTA", "METEORSHOWER", "SUNSTORM", "GOLDMINE", "HAGERAWEL", "WOODY", "BUZZ", "SILVERLIBRARY", "SILVERLINING", "VALHALLA", "MOUNTMCKINLEY", "SEPANG", "BLACKHOLE", "SALAMANCA", "HAGELAWEL", "LUCKYSTRIKE", "SPIKYROCK", "MYOFIBRIL", "INJUNCTION", "PSYCHICSALMON", "RAGINGBULLFROG", "ROBOTICFISH", "TIMIDTOAD", "AURA", "CARBONCOPY", "CASK", "CHORDAL", "CRAN", "HRMAP", "INSTINCT", "MADFORGE", "SWAMP", "VOLSUNGA", "WHITERAVEN", "MVR", "PPF", "JTRIG", "MAPREDUCE", "MAILORDER" ], "countries": [ "Pakistan", "Australia", "Spain", "Barbados", "Suriname", "Russia", "China" ], "people": [ "Tukey, John" ] }, "publication": [ { "publisher": "Boing Boing", "date_published": "2016-02-02", "title": "Exclusive: Snowden intelligence docs reveal UK spooks' malware checklist", "link": "https://boingboing.net/2016/02/02/doxxing-sherlock-3.html", "reporters": [ "Doctorow, Cory" ], "contributors": [ "Moltke, Henrik" ] } ], "aka_id": [], "pdf_sha256": "247cda315f6ba5c1e0c2471d385fcef2850554f91c782401e6d3793807b96caf", "pages": 96 }