first commit

This commit is contained in:
libroot 2026-11-19 04:39:11 +02:00
commit 0d90363116
Signed by: libroot
GPG key ID: 2AC68C029EBE9ACE
6510 changed files with 679146 additions and 0 deletions

View file

@ -0,0 +1,64 @@
{
"title": "XKEYSCORE Tech Strings in Documents (aka Tech Extractor) 2009",
"type": "REPORTED",
"description": "A December 2009 TOP SECRET XKeyscore training presentation on the Tech Extractor (Tech Strings in Documents) — an early context-aware scanning capability that tags keywords only when they appear inside email, chat and document bodies rather than anywhere in raw internet traffic, letting analysts find technically interesting documents by terms too generic for traditional content selection, with an example flagging a Pakistani document mentioning WiMAX and DVB satellite technology by scanning inside the file.",
"long_description": "This is a TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL XKeyscore training presentation dated December 2009 on the Tech Extractor, also called Tech Strings in Documents. It describes XKeyscore's first implementation of context-aware scanning — a refinement of content-based selection designed to reduce the volume of junk results that traditional \"soft selection\" produced.\n\nThe presentation frames the problem with existing selection engines: they were originally designed to handle TELEX traffic, a highly formatted, content-rich message type that bears little resemblance to raw DNI internet traffic containing HTML, web pages and base-64 encoded documents. When an analyst tasks a Boolean query like \"bomb\" AND \"chemical,\" they want communications mentioning those terms, not every web page, news story or blog post where the words happen to appear. The Tech Extractor addresses this by making selection context-aware — knowing where inside the raw DNI a term appears. The presentation uses a Crocodile Dundee reference (\"That's not a knife... THAT's a knife\") to contrast blunt soft selection with the \"surgical\" precision of contextual dictionaries.\n\nThe Tech Extractor focuses on three contexts: email bodies, chat bodies and document bodies (Microsoft Word, Excel, PowerPoint, Project and Visio; Adobe PDF and PostScript; and Rich Text Format). It works by scanning a list of keywords against those three contexts and tagging matching sessions with metadata — explicitly not filtering, selecting or forwarding data home, but tagging sessions much as AppIDs and fingerprints do. Analysts can then use these tags as part of USSID-18 compliant queries, though the presentation notes that Tech Extractor tags are not USSID-18 compliant by themselves and may need to be combined with a valid foreign IP, MAC address or country code. The term lists come from analysts, who supply the XKeyscore team with \"Tech Dictionaries\" containing multiple category names (\"Tech Names\").\n\nThe worked example demonstrates the capability's value. A Pakistani document (\"NIB Ranchor Line KHI.doc\") was tagged with three hits: the tech name \"wireless\" for the value WIMAX, \"satellite\" for DVB, and \"mac\" for a MAC address (00200E00953C) found inside the document. The presentation notes this document would have been nearly impossible to find with traditional methods — the terms \"wimax\" and \"dvb\" are too generic for CADENCE-style tasking, and the MAC address hit relied on an anchorless regular expression that corporate scanning engines could not perform. A second example shows a Nokia phone repair email (IMEI, model 6300) sent by a service centre coded GCC-KHI, and a third demonstrates full foreign-language support by tagging common Arabic expressions in emails from Pakistan's tribal regions.\n\nThe document is significant as the origin point for XKeyscore's context-aware scanning, which later evolved into the more sophisticated context-sensitive fingerprinting documented in the AppID presentations (html_title, http_host, email_body, doc_body and similar contexts). It demonstrates the analytical rationale behind context-aware selection — distinguishing communications content from incidental web content — and shows how the technique enabled discovery of technically valuable documents (network equipment specifications, WiMAX and satellite deployment details, MAC addresses) that traditional keyword selection would have buried in noise. The Pakistani tribal-region focus of the examples situates the capability within counterterrorism collection.",
"tags": [
"context-aware scanning",
"soft selection",
"content-based selection",
"email body",
"chat body",
"document body",
"Microsoft Office",
"TELEX",
"DNI",
"Nokia",
"IMEI",
"Karachi",
"tribal regions",
"metadata tagging",
"target discovery"
],
"document": {
"creator": "NSA",
"date_created": "2009-12",
"classification": [
"TOP SECRET",
"COMINT"
],
"dissemination": [
"REL TO USA, AUS, CAN, GBR, NZL"
],
"banner": "TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL",
"genre": "slides",
"programs": [
"XKEYSCORE",
"CADENCE",
"UIS"
],
"countries": [
"Pakistan"
],
"people": []
},
"publication": [
{
"publisher": "The Intercept",
"date_published": "2015-07-01",
"title": "XKEYSCORE: NSA’s Google for the World’s Private Communications",
"link": "https://theintercept.com/2015/07/01/nsas-google-worlds-private-communications/",
"reporters": [
"Marquis-Boire, Morgan",
"Greenwald, Glenn",
"Lee, Micah"
],
"contributors": [],
"doc_source": "https://theintercept.com/document/2015/07/01/xks-tech-extractor-2009/",
"doc_url": "https://s3.documentcloud.org/documents/2116496/xks-tech-extractor-2009.pdf"
}
],
"aka_id": [],
"pdf_sha256": "5479741c928afc690ede67e52ab520a76899bc53c08a95e9fd6c66ea38c752f8",
"pages": 14
}

View file

@ -0,0 +1,187 @@
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL//20291123
# KEYSCORE Tech Strings in Documents
## (aka Tech Extractor)
December 2009
Derived From: NSA/CSSM 1-52
Dated: 20070108
Declassify On: 20291123
DERIVED FROM: NSA/CSSM 1-52
08
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL//20291123
---
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## What is the Tech. Extractor?
* The "Tech Extractor" is a way of finding valuable intelligence based on keywords in the content of DNI sessions but it is a departure from traditional "soft selection" which tends to bring back a lot of junk.
---
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## What is soft selection?
* Soft selection, aka content based selection, is an approach at targeting traffic by looking for keywords or phrases rather than specific E-mail accounts
* Content based selection has suffered because of the poor design of content based selection engines
---
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Soft Selection vs Surgical Selection
* Existing selection techniques are blunt instruments
* XKEYSCORE contextual dictionaries provide an extremely sharp knife to make accurate selection decisions
"That's not a knife.....THAT's a knife!"
---
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Communication vs DNI Content
* Selection engines in use today were based on designs built to handle TELEX traffic
* TELEX is a highly formatted content rich type of traffic that does not resemble raw DNI seen with Internet traffic
* Raw Internet traffic contains HTML, web-pages, raw base-64 encoded documents etc.
* When analysts think of DNI "content" they are more referring to "communication content" then raw DNI content.
---
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Communication vs DNI Content
* If an analyst tasks a Boolean equation "bomb” and "chemical” they likely want to see all communication that mentions 'bomb' and 'chemical' and not all web pages, news stories, blog posts etc. where those two words appear
* What we need is a context-aware scanning engine that knows where it is inside of the raw DNI in order to properly apply analyst tasking
---
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## What is the Tech Extractor
* The Tech Extractor was X-KEYSCORE's first stab at context-aware scanning and it only focuses on three contexts:
* E-mail Bodies
* Chat Bodies
* Document Bodies:
* Microsoft Word, Excel, PowerPoint, Project, Visio
* Adobe PDF, Postscript
* Rich Text Format (RTF)
---
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## How does the Tech Extractor work?
* The Tech Extractor works by scanning a list of keywords against those three contexts and then tagging the results.
* It's important to note that this is not "filtering and selection" and we're not forwarding any data home
* XKS is simply tagging sessions with meta-data, much like we do with appids+fingerprints
---
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## How does the Tech Extractor work?
* After the meta-data tag is applied, analysts can then use that meta-data tag as part of a **USSID-18 compliant query** for traffic
* It's important to note, just like AppIDs+Fingerprints, Tech Extractor tags aren't necessarily USSID-18 compliant by themselves. You may need to add a valid foreign IP address, MAC address or country code before you query!
---
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Where does XKS get its list of terms?
* Analysts provide the XKS team with lists of terms, called "Tech Dictionaries" which can contain multiple category names (aka "Tech Names"
* Only after the XKS team is supplied with those terms can the system begin scanning and tagging.
---
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
| ID | DATETIME | DATETIME END | TECH NAME | TECH VALUE | TECH FILENAME |
| ---|---|---|---|---|---|
| 1 | 2008-01-01 04:55:00 | 2008-01-01 04:55:01 | wireless | WIMAX | NIB Ranchor Line KHI.doc |
| 2 | 2008-01-01 04:55:00 | 2008-01-01 04:55:01 | satellite | DVB | NIB Ranchor Line KHI.doc |
| 3 | 2008-01-01 04:55:00 | 2008-01-01 04:55:01 | mac | 00200E00953C | NIB Ranchor Line KHI.doc |
This document would have been nearly impossible to find without the context aware tasking. The terms 'wimax' and 'dvb' are too generic for CADENCE style tasking and the MAC address hit on an anchorless regular expression, impossible with current corporate scanning engines
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Context-Aware Tagging
```
Subject: NFF-66024-GCC-KHI
From: [REDACTED]
To: [REDACTED]
Cc: [REDACTED]
Date: Tue Dec 30 10:57:48 GMT 2008
HTML Plain Text Attachment
IMEI: [REDACTED]
Model: 6300
WON: 66024
ASC:GCC-KHI
Symptom: 4100
Comments: no fault found phone is working properly kindly confirm the fault in detail when and in which condition it
creates problem related to mention symptom
[REDACTED]
GSM Repair Engineer
Tel: [REDACTED]
Mob: [REDACTED]
Fax: [REDACTED]
```
```
Event T
email_b
Fm City
KLOST
```
---
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Full Foreign Language Support
Supports full foreign language tagging and querying
Ex look for common Arabic expressions in E-mails coming from the Pakistan tribal regions:
```
UIS Webmail Display Windows Live™ Mail Beta
Active user: Unknown
From: [REDACTED] ([REDACTED]@gmail.com)
Medium risk You may not know this sender Mark as safe|Mark as unsafe
Sent: Thu 1/01/09 12:07 PM
To: [REDACTED]
السلام عليكم ورحمة الله وبركات
```
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Live Demo