snowchive-meta/data/Xks-Helper-Notes-Redacted/metadata.json
2026-11-19 04:39:11 +02:00

85 lines
5.4 KiB
JSON
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

{
"title": "XKEYSCORE Helper Notes",
"type": "REPORTED",
"description": "This GCHQ document describes new features in the XKEYSCORE Palantir helper tool, including summary/histogram import capabilities that reduce large datasets by aggregating connection data rather than creating individual events for each entry, and data sourcing features that track which XKEYSCORE queries imported specific data into Palantir. The document provides step-by-step screenshots showing how analysts can histogram network traffic by source IP, destination IP, and ports, then import aggregated results showing metadata like total session sizes and connection counts.",
"long_description": "This technical guide documents enhancements to the integration between XKEYSCORE (NSA's massive signals intelligence database) and Palantir (the data analysis platform used by GCHQ and NSA). The primary innovation is the \"summary import\" feature, which mirrors XKEYSCORE's histogram functionality to dramatically reduce the volume of data imported into Palantir while preserving analytical value. Instead of creating separate events for each of 19 identical connection patterns (as in the example showing BitTorrent traffic), the new system creates a single aggregated \"Connection\" object with a \"Quantity\" property recording the count. The document walks through the workflow: analysts run queries in XKEYSCORE, select \"Summarise by...\" in the helper tool, choose fields to aggregate by (such as destination port), and import condensed results. The summarized connections preserve key metadata including the sum of all session sizes (showing total data transferred), the range of timestamps (first and last occurrence), and all application fingerprints triggered across the aggregated events. For example, if IP address X contacts Y multiple times triggering different detection signatures, the summary shows all fingerprints as \"applications\" on a single connection object.\n\nThe second major enhancement is comprehensive data sourcing, allowing analysts to trace any object in Palantir back to its originating XKEYSCORE query. The document demonstrates how imported data appears in Palantir's \"Data Sources\" application, organized hierarchically with folders for each query containing lists of IP addresses, connection events, and a document with query metadata (datetime, database coverage, hit count, status). Notably, the document includes an unredacted IP address—46.21.87.220—visible in one of the property panels, appearing as a \"To\" IP address in what the application fingerprints identify as BitTorrent peer-to-peer traffic. This data sourcing capability persists across multiple import sessions, with the system maintaining a growing list of XKEYSCORE queries that contributed data to an investigation. Analysts can view sourcing information either through the dedicated Data Sources application or directly from any imported object's properties panel, where a \"Sourcing...\" link and \"Data Sources\" section list contributing queries. The document emphasizes user convenience features like saving per-user preferences for histogram fields and auto-merge settings, eliminating repetitive configuration for common analysis workflows. These enhancements reflect the sophisticated integration between collection systems (XKEYSCORE) and analytical platforms (Palantir) that enable GCHQ and NSA analysts to efficiently process and investigate massive volumes of intercepted internet traffic.",
"tags": [
"Palantir",
"data fusion",
"data aggregation",
"data reduction",
"queries",
"connection events",
"IP addresses",
"network traffic",
"traffic analysis",
"BitTorrent",
"torrents",
"peer-to-peer",
"encoding",
"application fingerprints",
"quantity metadata",
"timestamp metadata",
"data collection",
"query tracking",
"target tracking",
"investigation",
"metadata storage",
"network infrastructure",
"workflow automation",
"implants",
"graph analysis",
"network analysis",
"internet traffic",
"intercepted communications",
"connection aggregation",
"connection mapping",
"event aggregation",
"collection systems",
"SIGINT collection",
"intelligence analysis",
"data processing",
"query integration",
"analytic tools",
"technical guide",
"screenshots",
"workflow",
"analyst workflow"
],
"document": {
"creator": "GCHQ",
"date_created": "",
"classification": [
"TOP SECRET",
"STRAP 1"
],
"dissemination": [],
"banner": "TOP SECRET STRAP 1",
"genre": "manual",
"programs": [
"XKEYSCORE"
],
"countries": [],
"people": []
},
"publication": [
{
"publisher": "The Intercept",
"date_published": "2017-02-22",
"title": "How Peter Thiel’s Palantir Helped the NSA Spy on the Whole World",
"link": "https://theintercept.com/2017/02/22/how-peter-thiels-palantir-helped-the-nsa-spy-on-the-whole-world/",
"reporters": [
"Biddle, Sam"
],
"contributors": [
"Thomason, John",
"Cooper, Talya"
],
"doc_source": "https://theintercept.com/document/2017/02/21/xkeyscore-helper-notes/",
"doc_url": "https://s3.documentcloud.org/documents/3469345/Xks-Helper-Notes-Redacted.pdf"
}
],
"aka_id": [],
"pdf_sha256": "7815098ddcccefb66fb802d8fd1cc01108ff158b89b2fc1eb11591ac14c8b5d6",
"pages": 6
}