first commit
This commit is contained in:
commit
0d90363116
6510 changed files with 679146 additions and 0 deletions
64
data/xks-tech-extractor-2010/metadata.json
Normal file
64
data/xks-tech-extractor-2010/metadata.json
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
{
|
||||
"title": "XKEYSCORE Tech Strings in Documents (aka Tech Extractor) 2010",
|
||||
"type": "REPORTED",
|
||||
"description": "An April 2010 TOP SECRET XKeyscore training presentation on the Tech Extractor — an expanded version of the December 2009 briefing — teaching analysts to find intelligence by tagging keywords inside email, chat and document bodies, with new detail on tasking rules (including that terms hit as substrings so \"ricin\" matches \"pricing\"), its non-retrospective operation, its limitations versus the 65+ context fingerprint language, and confirmation that the capability was operated jointly with GCHQ, whose Cheltenham analysts submitted tasking via an Excel template and wiki.",
|
||||
"long_description": "This is a TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL XKeyscore training presentation dated April 2010 on the Tech Extractor (Tech Strings in Documents). It is an expanded revision of the December 2009 Tech Extractor briefing, retaining the core explanation while adding tasking rules, operational limitations, a comparison with the fingerprint language, and significant GCHQ-specific content.\n\nThe core material is unchanged from the earlier version: the Tech Extractor is XKeyscore's first context-aware scanning capability, addressing the weakness of traditional soft selection (content-based keyword selection) which returned excessive junk because legacy engines designed for formatted TELEX traffic could not distinguish communication content from incidental web content. It tags keywords appearing in three contexts — email bodies, chat bodies, and document bodies (Microsoft Office, PDF, PostScript, RTF) — with metadata, which analysts then use in USSID-18 compliant queries (with the caveat that tags alone are not compliant and may need a foreign IP, MAC address or country code).\n\nThis version adds several operational details. Tasking rules are now specified: all terms must be classified REL FVEY, terms are case-insensitive by default (but can be forced case-sensitive), and terms hit as substrings by default — the example warning that \"ricin\" will hit inside \"pricing\" unless forced to match as a whole word via surrounding spaces or a regular expression. The capability is explicitly non-retrospective: after a list is tasked, XKeyscore scans only data collected from that point forward, not previously stored data. The term lists (Tech Dictionaries with category Tech Names) are still supplied by analysts to the XKeyscore team, but a GUI for analyst self-entry is noted as almost complete.\n\nThe limitations section is new and candid: even restricted to document, email and chat bodies, that is still a large volume of traffic; generic or short terms still produce many false hits; and the term scans all data even when results are later narrowed by country or IP. The comparison with fingerprints is the key analytical addition — the Tech Extractor treats email, chat and document bodies as a single combined context, whereas the XKeyscore fingerprint language offers over 65 contexts that can be combined into powerful, specific signatures. Analysts are advised to switch to the full fingerprint language when generic terms return too many poor results. The Tech Extractor's distinct advantage is preserved, however: it shows exactly which term hit in the metadata results, whereas a fingerprint only indicates that its full (potentially complex) equation was satisfied without revealing which specific terms matched.\n\nThe worked example results table shows GSM-related tech dictionary hits — HLR, ICCID, IMEI, IMSI, MSISDN — extracted from document and email bodies including files with revealing paths (\"25 APR 10 Daily Break Down.xls,\" \"Evo_Complaints_sheet_25-Apr-2010.xls,\" a file referencing \"SE EWSD\" and \"New Franchisees Status\"), suggesting collection against telecommunications operator business documents.\n\nThe most significant new content is the confirmation of GCHQ's operational role. The presentation directs analysts to a GCHQ wiki page (wiki.gchq/index.php/Tech_dictionaries) for more information, instructs them to submit tasking using an Excel spreadsheet template \"developed by GCHQ CP\" (Cheltenham/a GCHQ unit), and to email the term lists to a gchq.gov.uk address. This confirms that the Tech Extractor tasking process was administered by GCHQ, not NSA — demonstrating that this XKeyscore capability was operated as a genuinely joint NSA-GCHQ function with GCHQ handling the dictionary submission workflow. The document is significant both as documentation of the evolution of context-aware scanning toward the mature fingerprint language and as concrete evidence of the depth of NSA-GCHQ integration in XKeyscore's day-to-day operation.",
|
||||
"tags": [
|
||||
"context-aware scanning",
|
||||
"soft selection",
|
||||
"content-based selection",
|
||||
"fingerprints",
|
||||
"email body",
|
||||
"chat body",
|
||||
"document body",
|
||||
"Microsoft Office",
|
||||
"TELEX",
|
||||
"GSM",
|
||||
"MSISDN",
|
||||
"MSISDN",
|
||||
"EWSD",
|
||||
"telecommunications",
|
||||
"Nokia",
|
||||
"metadata tagging"
|
||||
],
|
||||
"document": {
|
||||
"creator": "NSA",
|
||||
"date_created": "2010-04",
|
||||
"classification": [
|
||||
"TOP SECRET",
|
||||
"COMINT"
|
||||
],
|
||||
"dissemination": [
|
||||
"REL TO USA, AUS, CAN, GBR, NZL"
|
||||
],
|
||||
"banner": "TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL",
|
||||
"genre": "slides",
|
||||
"programs": [
|
||||
"XKEYSCORE",
|
||||
"UIS"
|
||||
],
|
||||
"countries": [
|
||||
"Pakistan"
|
||||
],
|
||||
"people": []
|
||||
},
|
||||
"publication": [
|
||||
{
|
||||
"publisher": "The Intercept",
|
||||
"date_published": "2015-07-01",
|
||||
"title": "XKEYSCORE: NSA’s Google for the World’s Private Communications",
|
||||
"link": "https://theintercept.com/2015/07/01/nsas-google-worlds-private-communications/",
|
||||
"reporters": [
|
||||
"Marquis-Boire, Morgan",
|
||||
"Greenwald, Glenn",
|
||||
"Lee, Micah"
|
||||
],
|
||||
"contributors": [],
|
||||
"doc_source": "https://theintercept.com/document/2015/07/01/xks-tech-extractor-2010/",
|
||||
"doc_url": "https://s3.documentcloud.org/documents/2116499/xks-tech-extractor-2010.pdf"
|
||||
}
|
||||
],
|
||||
"aka_id": [],
|
||||
"pdf_sha256": "3b1a6fd56a0b7e796f4aa6df7695b2ef1a63e7d2bf7755727c54b3f635b8b691",
|
||||
"pages": 23
|
||||
}
|
||||
322
data/xks-tech-extractor-2010/xks-tech-extractor-2010.md
Normal file
322
data/xks-tech-extractor-2010/xks-tech-extractor-2010.md
Normal file
|
|
@ -0,0 +1,322 @@
|
|||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
# Tech Strings in Documents
|
||||
## (aka Tech Extractor)
|
||||
|
||||
April 2010
|
||||
|
||||
Derived From: NSA/CSSM 1-52
|
||||
Dated: 20070108
|
||||
Declassify On: 20340701
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Agenda
|
||||
|
||||
* Overview and History of Tech Strings in Documents
|
||||
* Why is it important?
|
||||
* Limitations of capability - advance to fingerprints
|
||||
* Examples and live demo
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Content-based Selection
|
||||
|
||||
* How do you find DNI data if you don't have a strong selector like IP or E-mail address?
|
||||
* What if you only know keywords, part names, phrases etc. expected to be used by your target?
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## What is the Tech. Extractor?
|
||||
|
||||
* The "Tech Extractor" is a way of finding valuable intelligence based on keywords in the content of DNI sessions but it is a departure from traditional "soft selection" which tends to bring back a lot of junk.
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## What is soft selection?
|
||||
|
||||
* Soft selection, aka content based selection, is an approach at targeting traffic by looking for keywords or phrases rather than specific E-mail accounts
|
||||
* Content based selection has suffered because of the poor design of content based selection engines
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Communication vs. DNI Content
|
||||
|
||||
* Selection engines in use today were based on designs built to handle TELEX traffic
|
||||
* TELEX is a highly formatted content rich type of traffic that does not resemble raw DNI seen with Internet traffic
|
||||
* Raw Internet traffic contains HTML, web-pages, raw base-64 encoded documents etc.
|
||||
* When analysts think of DNI "content" they are more referring to "communication content" then raw DNI content.
|
||||
* Current DNI selection does not allow you to restrict hits to the "type" of traffic you want eg. Emails (including Webmail) or Documents
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Communication vs. DNI Content
|
||||
|
||||
* If an analyst tasks a Boolean equation "bomb" and "chemical" they likely want to see all communication that mentions 'bomb' and 'chemical' and not all web pages, news stories, blog posts etc. where those two words appear
|
||||
* What we need is a context-aware scanning engine that knows where it is inside of the raw DNI in order to properly apply analyst tasking
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Soft Selection vs. Surgical Selection
|
||||
|
||||
* Existing selection techniques are blunt instruments
|
||||
* XKEYSCORE contextual dictionaries provide an extremely sharp knife to make accurate selection decisions
|
||||
|
||||
"That's not a knife.....THAT's a knife!"
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## What is the Tech Extractor
|
||||
|
||||
* The Tech Extractor was X-KEYSCORE's first stab at context-aware scanning and it only focuses on three contexts:
|
||||
* E-mail Bodies
|
||||
* Chat Bodies
|
||||
* Document Bodies:
|
||||
* Microsoft Word, Excel, PowerPoint, Project, Visio
|
||||
* Adobe PDF, Postscript
|
||||
* Rich Text Format (RTF)
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## How does the Tech Extractor work?
|
||||
|
||||
* The Tech Extractor works by scanning a list of keywords (or regular expressions) against those three contexts and then tags the results.
|
||||
* This is not "filtering and selection" and we're not forwarding any data home
|
||||
* XKS is simply tagging sessions with meta-data, much like we do with appids+fingerprints
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## How does the Tech Extractor work?
|
||||
|
||||
* After the meta-data tag is applied, analysts can then use that meta-data tag as part of a _USSID-18_ compliant query for traffic
|
||||
* It's important to note, just like AppIDs+Fingerprints, Tech Extractor tags aren't necessarily USSID-18 compliant by themselves.
|
||||
* You may need to add a valid foreign IP address, MAC address or country code before you query!
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Context-Aware Tagging
|
||||
|
||||
```
|
||||
Subject NFF-66024-GCC-KHI
|
||||
From: [REDACTED]
|
||||
To: [REDACTED]
|
||||
CC: [REDACTED]
|
||||
Date: Tue Dec 30 10:57:48 GMT 2008
|
||||
|
||||
HTML Plain Text Attachment
|
||||
|
||||
IMEI: [REDACTED]
|
||||
Model: 6300
|
||||
WON: 66024
|
||||
ASC: GCC-KHI
|
||||
|
||||
Symptom: 4100
|
||||
|
||||
Comments: no fault found phone is working properly kindly confirm the fault in detail when and in which condition it
|
||||
creates problem related to mention symptom
|
||||
|
||||
[REDACTED]
|
||||
GSM Repair Engineer
|
||||
[REDACTED]
|
||||
Tel: [REDACTED]
|
||||
Mob: [REDACTED]
|
||||
Fax: [REDACTED]
|
||||
```
|
||||
|
||||
```
|
||||
Event T
|
||||
email_b
|
||||
Fm City
|
||||
KLOST
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## How does the Tech Extractor work?
|
||||
|
||||
* Also this is not retrospective.
|
||||
* After a list is tasked, XKS will scan data collected from that point on looking hits.
|
||||
* Any data previously collected and stored by XKS will not be scanned.
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Where does XKS get its list of terms?
|
||||
|
||||
* Analysts provide the XKS team with lists of terms, called "Tech Dictionaries" which can contain multiple category names (aka "Tech Names")
|
||||
* Only after the XKS team is supplied with those terms can the system begin scanning and tagging.
|
||||
* GUI to allow analysts' entry of tech terms almost complete
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Tech Extractor Tasking Rules
|
||||
|
||||
* Currently, all terms need to be classified REL FVEY
|
||||
* Terms are case insensitive by default, but can be forced to be case sensitive
|
||||
* Terms can hit as a substrings by default
|
||||
ex: 'ricin' will hit in 'pricing'
|
||||
* However, terms can be forced to hit as a unique word (either by tasking them with a space at the beginning and end or by using a regular expression)
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Full Foreign Language Support
|
||||
|
||||
* Supports full foreign language tagging and querying
|
||||
* Ex look for common Arabic expressions in E-mails coming from the Pakistan tr
|
||||
|
||||
```
|
||||
UIS Webmail Display Windows Live Mail Beta
|
||||
Active user: Unknown
|
||||
|
||||
From: [REDACTED] ([REDACTED]@gmail.com)
|
||||
Medium risk You may not know this sender Mark as safe | Mark as unsafe
|
||||
Sent: Thu 1/01/09 12:07 PM
|
||||
To: [REDACTED]
|
||||
|
||||
السلام عليكم ورحمة الله وبركات
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Tech Extractor Limitations
|
||||
|
||||
* While terms tasked for the Tech Extractor are applied only to Document, E-mail and Chat bodies, that is still a lot of traffic!
|
||||
* If the term is too generic (or short) you're still likely to run into a lot of false hits.
|
||||
* Also, while you can limit your results by adding more search criteria (country code, IP address etc), the term will be scanning all data looking for hits
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Tech Extractor vs. Fingerprints
|
||||
|
||||
* Tech Extractor treats E-mail, Chat and Document bodies as a single "context"
|
||||
* The XKS Fingerprint language gives you over 65+ contexts that can be used together to form powerful and specific signatures
|
||||
* When terms are generic and are returning too many poor results through Tech Extractor, then it's time to make the switch to the full fingerprint language
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Why use the Tech Extractor at all?
|
||||
|
||||
* One of the most powerful feature of the Tech Extractor is that it shows you exactly which term hit in the meta-data results:
|
||||
|
||||
| Event Type | Tech Dictionary | Tech Name | Tech Value ^ | Tech Filename |
|
||||
| :------------- | :-------------- | :-------- | :----------- | :----------------------------------------------------------- |
|
||||
| `document_body` | classic | gsm | HLR | `C:\Documents and Settings\SE EWSD\Desktop\25 APR 10 Daily Break Down.x` |
|
||||
| `document body` | classic | gsm | ICCID | `C:\Documents and Settings\Administrator\Desktop\New Franchisees Status fo` |
|
||||
| `email_body` | classic | gsm | IMEI | |
|
||||
| `document_body` | classic | gsm | IMSI | `Evo_Complaints_sheet_25-Apr-2010.xls` |
|
||||
| `email_body` | classic | gsm | MSISDN | |
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Why not Fingerprints?
|
||||
|
||||
* With fingerprints, you only see that the full equation (which can be very complex) was satisfied and you won't see which specific terms from the equation hit.
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Live Demo
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## More information:
|
||||
|
||||
* On GCHQ wiki:
|
||||
|
||||
http://wiki.gchq/index.php/Tech_dictionaries
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## To submit tasking
|
||||
|
||||
* Please use the Excel Spreadsheet template developed by GCHQ CP
|
||||
Microsoft Excel Worksheet icon
|
||||
* And then E-mail [REDACTED]@gchq.gov.uk with the list
|
||||
* In the near future analysts will be able to enter the terms themselves through a web-based GUI
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
Loading…
Add table
Add a link
Reference in a new issue