first commit
This commit is contained in:
commit
0d90363116
6510 changed files with 679146 additions and 0 deletions
112
data/xks-search-forms/metadata.json
Normal file
112
data/xks-search-forms/metadata.json
Normal file
|
|
@ -0,0 +1,112 @@
|
|||
{
|
||||
"title": "XKEYSCORE Search Forms",
|
||||
"type": "REPORTED",
|
||||
"description": "A secret XKeyscore reference guide from March 2009 cataloguing all of the system's search forms — one entry per metadata plug-in — revealing the full breadth of what XKeyscore extracts and indexes from intercepted internet traffic, from emails, logins, passwords, phone numbers and documents to Cisco router passwords, TOR routers, VoIP calls, WLAN data, Blackberry PINs, satellite terminals, passport images, internet café addresses, and exfiltrated data from TAO hacking operations.",
|
||||
"long_description": "This is a SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL XKeyscore reference guide dated March 2009. It systematically documents every search form available in XKeyscore, with each corresponding to a metadata plug-in that extracts and indexes a particular type of data from intercepted sessions.\n\nThe guide opens with query syntax — wildcards, Boolean operators, comparison operators, regular expressions, and Google-like full-text search with required/excluded terms and exact phrases. It then documents each search form. The core logging plug-ins include Full Log DNI (one record per session with SIGAD, casenotation, session UUID, MAC/IP addresses, ports and full application ID), Email Addresses (username, domain, subject), Logins & Passwords (anything resembling credentials, with examples showing captured HTML form fields and FTP logins), and Phone Numbers (parsed from signature blocks and address books).\n\nThe dictionary scanning plug-ins include Alert (log of sessions tipped to TRAFFICTHIEF, including the DECODEORDAIN permutation that triggered the tip and a copy of the XML sent to TRAFFICTHIEF) and Category DNI (category hits from CADENCE and other dictionaries). User Activity indexes webmail, chat and webcam activity with strong selectors — the worked example shows how a Yahoo user's session exposes machine-specific cookies (yahooBcookie, yahooGSS), proxy information and X-Forwarded-For IPs, and how these cookies can be correlated back to the username.\n\nThe document processing plug-ins include Extracted Files, Document Tagging (also called Tech Strings — labelling document and email bodies with second-level dictionary hits categorised as government, monetary, proliferation, satellite, wireless, etc.), Document Metadata (author, organisation, language, unique ID, timestamps, and hashes of the entire document and any embedded images), and PDF Metadata.\n\nThe protocol processing plug-ins are extensive and reveal the breadth of XKeyscore's collection: Blackberry (source/destination PIN and BES, payload type), Cellular DNI (IMSI, TMSI, IMEI, cell ID, latitude/longitude, spotbeam), Cisco Passwords (logged and decoded using the known weak obfuscation key), HTTP Activity (host, URL, parsed search terms from Google/Yahoo, user-agent, cookies, server type, geolocation from weather reports), IKE (Internet Key Exchange VPN parameters), IRC Café Geolocation (extracting café street addresses from IRC QUIT messages), Passport detection (OCR of passport images extracting name, passport number, issuing state, DOB — under development), Radius Logs (dial-up authentication), RBGAN and Cellular satellite terminal collection, RTP (VoIP audio/video decodable to playable media), SIP (VoIP call setup), SSL (certificate and encryption parameters), TOR Log (identified TOR routers), Web File Transfer (uploads/downloads from RapidShare, DepositFiles etc., under development by GCHQ/MHS), Web Proxy (X-Forwarded-For and leaked IPs from STUN and Google Earth), Wireshark (BGP, OSPF, VoIP, SMB, SNMP, GTP tunnelling), and WLAN (SSID, BSSID, MAC addresses).\n\nThe miscellaneous plug-ins include Call Logs (DNR telephony metadata from JUGGERNAUT, CERF, FASCIA and DURT), Network Logs (from MOONSHINE wireless collection), and two CNE-specific forms populated from TAO hacking operations — CNE (project name, collection technique, filename from TAO exfiltration) and Registry (Windows registry data including hive, key, subkey and value from compromised machines), both on the limited-deployment xks-cne cluster. The guide closes with Simple Search and Multi Search convenience forms that federate a single selector across multiple plug-in databases.\n\nThe document is significant as the definitive catalogue of XKeyscore's extraction capabilities, demonstrating that the system indexed not merely emails and web traffic but an extraordinary range of data types — router passwords, VPN handshake parameters, SSL certificates, TOR nodes, satellite terminal locations, Blackberry PINs, WLAN identifiers, passport images, internet café street addresses, telephony metadata from named DNR databases, and the exfiltrated contents of TAO-compromised machines. The inclusion of Cisco password decoding, IKE/SSL cryptographic parameter capture and CNE/Registry data shows XKeyscore serving simultaneously as a passive collection index, a cryptanalytic support tool and a repository for offensive hacking output. Several plug-ins credited to GCHQ/MHS and populated from MOONSHINE confirm the system's Five Eyes-wide development and deployment.",
|
||||
"tags": [
|
||||
"metadata",
|
||||
"email addresses",
|
||||
"phone numbers",
|
||||
"user activity",
|
||||
"extracted files",
|
||||
"embedded images",
|
||||
"BlackBerry",
|
||||
"Cellular DNI",
|
||||
"IMSI",
|
||||
"IMEI",
|
||||
"Cisco",
|
||||
"HTTP Activity",
|
||||
"IKE",
|
||||
"IRC Cafe Geolocation",
|
||||
"passports",
|
||||
"Tor",
|
||||
"Web File Transfer",
|
||||
"RapidShare",
|
||||
"Google Earth",
|
||||
"WiFi",
|
||||
"call logs",
|
||||
"Computer Network Exploitation",
|
||||
"Tailored Access Operations",
|
||||
"VoIP",
|
||||
"geolocation",
|
||||
"MHS"
|
||||
],
|
||||
"document": {
|
||||
"creator": "NSA",
|
||||
"date_created": "2009-03",
|
||||
"classification": [
|
||||
"SECRET",
|
||||
"COMINT"
|
||||
],
|
||||
"dissemination": [
|
||||
"REL TO USA, AUS, CAN, GBR"
|
||||
],
|
||||
"banner": "SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL",
|
||||
"genre": "slides",
|
||||
"programs": [
|
||||
"XKEYSCORE",
|
||||
"TRAFFICTHIEF",
|
||||
"DECODEORDAIN",
|
||||
"CADENCE",
|
||||
"JUGGERNAUT",
|
||||
"FASCIA",
|
||||
"CERF",
|
||||
"DURT",
|
||||
"MOONSHINE",
|
||||
"SPOTBEAM",
|
||||
"RADIUS"
|
||||
],
|
||||
"countries": [],
|
||||
"people": []
|
||||
},
|
||||
"publication": [
|
||||
{
|
||||
"publisher": "The Intercept",
|
||||
"date_published": "2015-07-01",
|
||||
"title": "XKEYSCORE: NSA’s Google for the World’s Private Communications",
|
||||
"link": "https://theintercept.com/2015/07/01/nsas-google-worlds-private-communications/",
|
||||
"reporters": [
|
||||
"Marquis-Boire, Morgan",
|
||||
"Greenwald, Glenn",
|
||||
"Lee, Micah"
|
||||
],
|
||||
"contributors": [],
|
||||
"doc_source": "https://theintercept.com/document/2015/07/01/xks-search-forms/",
|
||||
"doc_url": "https://s3.documentcloud.org/documents/2116401/xks-search-forms.pdf"
|
||||
},
|
||||
{
|
||||
"publisher": "SVT Nyheter",
|
||||
"date_published": "2013-12-11",
|
||||
"title": "FRA has access to controversial surveillance system",
|
||||
"link": "https://www.svt.se/nyheter/granskning/ug/fra-has-access-to-controversial-surveillance-system",
|
||||
"reporters": [
|
||||
"Bergman, Sven",
|
||||
"Dyfvermark, Joachim",
|
||||
"Gallagher, Ryan",
|
||||
"Greenwald, Glenn",
|
||||
"Laurin, Fredrik",
|
||||
"Struwe, Filip"
|
||||
],
|
||||
"contributors": [],
|
||||
"doc_source": "http://www.svt.se/ug/read-the-snowden-documents-from-the-nsa",
|
||||
"doc_url": "https://www.documentcloud.org/documents/894406-nsa-slides-xkeyscore/",
|
||||
"version": "nsa-slides-xkeyscore",
|
||||
"note": "SVT Nyheter published a single slide from this deck — \"TOR Log\", page 37 of 49 — on 11 December 2013, a year and a half before The Intercept released the deck itself on 1 July 2015. The slide appeared inside a three-slide PDF accompanying SVT's reporting on NSA's cooperation with Sweden's FRA; the other two slides in that PDF are from X-KEYSCORE as a SIGDEV tool (xks-as-a-sigdev-tool)."
|
||||
}
|
||||
],
|
||||
"aka_id": [
|
||||
{
|
||||
"id": "nsa-slides-xkeyscore"
|
||||
},
|
||||
{
|
||||
"id": "EFF_-_20131211-svt-nsa_slides_xkeyscore"
|
||||
}
|
||||
],
|
||||
"pdf_sha256": "1ced284b2f47df5f962a2a8888b154aa5ca4ad09cdbf614ddf699006ad2b4a60",
|
||||
"pages": 49,
|
||||
"versions": {
|
||||
"nsa-slides-xkeyscore": {
|
||||
"pdf_sha256": "05397b912612a684f937469e66c5a8290b58aa3569b8a2adea2e3a74f36236ff"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## TOR Log
|
||||
|
||||
* Logs any identified TOR routers used for anonymizing Internet traffic
|
||||
* Searchable fields
|
||||
* TOR from server
|
||||
* TOR to server
|
||||
* Router nickname
|
||||
|
||||
SECRET//COMINT// REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//ORCON,REL TO USA, AUS, CAN, GBR and NZL//20291123
|
||||
|
||||
## "Slowing down the Internet"
|
||||
|
||||
* XKS goal is to store the full-take content for 3-5 days, effectively "slowing down the Internet" so that analysts can go back and recover sessions that otherwise would have been dropped by the front end
|
||||
* Meta-data is saved off longer, with the goal of 30 days retention
|
||||
* A lot of analysis can be done through meta-data only (MARINA is meta-data only)
|
||||
|
||||
TOP SECRET//COMINT//ORCON,REL TO USA, AUS, CAN, GBR and NZL//20291123
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//ORCON,REL TO UAS, AUS, CAN, GBR AND NZL//20291123
|
||||
|
||||
## What makes XKS so good at SIGDEV?
|
||||
|
||||
* XKS gives analysts unique access to terabytes of content and meta-data
|
||||
* Typically sites select and forward to PINWALE less than 5% of the DNI they're processing
|
||||
* The rest of that data used to be dropped but is now being retained temporarily and made available to analysts through X-KEYSCORE
|
||||
* As an example, at one our sites XKS sees more data per day than all of PINWALE
|
||||
|
||||
TOP SECRET//COMINT//ORCON,REL TO USA, AUS, CAN, GBR and NZL//20291123
|
||||
760
data/xks-search-forms/xks-search-forms.md
Normal file
760
data/xks-search-forms/xks-search-forms.md
Normal file
|
|
@ -0,0 +1,760 @@
|
|||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
# XKEYSCORE Search Forms
|
||||
|
||||
March 2009
|
||||
|
||||
DERIVED FROM: NSA/CSSM 1-52
|
||||
DATED: 20070108
|
||||
DECLASSIFY ON: 20320108
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Standard search fields
|
||||
|
||||
### Wildcards
|
||||
|
||||
* `*` - multiple characters anywhere in word
|
||||
* `_` - single character anywhere in word
|
||||
* Some fields are auto-wildcarded - the field name will have a `*` before and/or after it
|
||||
|
||||
### Operators
|
||||
|
||||
* Boolean AND, OR - do not use in the same field
|
||||
* `!` - NOT (e.g. `!joe`)
|
||||
* `<`, `>` - comparison (e.g. `>00080`)
|
||||
* `regex:` - regular expression (e.g. `regex:[0-9]*`)
|
||||
* Enter `!` to require a field to be non-empty
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Special (full-text) search fields
|
||||
|
||||
* Google-like syntax - just list your terms and the query will return sessions that match any of them
|
||||
* Wildcards only allowed at the end of a word
|
||||
* Search terms must be at least 4 characters
|
||||
* Use `+` or `-` to require that a word must or must not be present
|
||||
* Use `""` to find an exact phrase
|
||||
* Use `()` for grouping
|
||||
* You can still use “classic” syntax - we convert it for you
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Special (full-text) search fields
|
||||
|
||||
### Examples:
|
||||
|
||||
| Search terms | Returned results... |
|
||||
| :----------------- | :--------------------------------------------------------- |
|
||||
| apple banana | contain 'apple' or 'banana' or both |
|
||||
| +apple +juice | contain both 'apple' and 'juice' |
|
||||
| +apple -macintosh | contain 'apple' but not 'macintosh' |
|
||||
| +apple +(turnover strudel) | contain 'apple' AND either 'turnover' or 'strudel' |
|
||||
| apple* | contain words like 'apple' or 'apples' or 'applesauce' or 'applet' |
|
||||
| "apple juice" | contain the exact phrase 'apple juice' |
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## This plug-in has no data!
|
||||
|
||||
* Under development
|
||||
* Menu items and search forms may show up before a plug-in goes “live” in the field
|
||||
* Limited deployment
|
||||
* Some sites run different sets of plug-ins
|
||||
* Populated by front end
|
||||
* Some plug-ins simply database metadata provided by the system that feeds XKS, and not all sites are set up the same way
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Simple Logging
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Full Log DNI
|
||||
|
||||
* One record for every session processed
|
||||
* Collection fields
|
||||
* SIGAD
|
||||
* Casenotation
|
||||
* Session ID (UUID)
|
||||
* Protocol fields
|
||||
* MAC addresses
|
||||
* IP addresses
|
||||
* Port numbers
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Full Log DNI
|
||||
|
||||
* Application ID fields
|
||||
* Application Name – full application ID
|
||||
* Application Type – top level of application ID
|
||||
* Application Info – extra info
|
||||
* Appid+fingerprints – full application ID plus any matching fingerprints
|
||||
* Example:
|
||||
* Application name: mail/webmail/yahoo
|
||||
* Application type: mail
|
||||
* Application info: viewFolder_webmail
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Full Log DNI
|
||||
|
||||
* Fields populated by other plug-ins
|
||||
* Username (from User Activity)
|
||||
* Category hits (from Category DNI)
|
||||
* Client IP/X-Forwarded-For (from Web Proxy)
|
||||
* Most Full Log search fields are available on every other search form
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Email Addresses
|
||||
|
||||
* Anything that looks like an email address
|
||||
* Searchable fields
|
||||
* Email username – the part before the @ only!
|
||||
* Domain – the part after the @
|
||||
* Subject – email subject, if present
|
||||
* Example:
|
||||
|
||||
```
|
||||
Sender: user1@yahoo.com
|
||||
MIME-Version: 1.0
|
||||
Subject: check this out
|
||||
Date: Tue, 02 Jan 2007 13:27:31 -0000
|
||||
Message-ID: <AAAAAAAAAAAAAAAAA123456789@yahoo.com>
|
||||
From: "User One" <user1@yahoo.com>
|
||||
To: “User Two" <user2@hotmail.com>
|
||||
```
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Logins & Passwords
|
||||
|
||||
* Anything that looks like a login or password
|
||||
* Searchable fields
|
||||
* Username
|
||||
* Password
|
||||
* Examples:
|
||||
|
||||
```
|
||||
<input name="username" value="badguy">
|
||||
<input name="password" value="asdf123">
|
||||
|
||||
USER badguy
|
||||
PASS asdf123
|
||||
```
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Phone Numbers in DNI
|
||||
|
||||
* Anything that looks like a phone number
|
||||
* Searchable fields
|
||||
* Phone number
|
||||
* Number type (fax, telephone, mobile, etc.)
|
||||
* Example:
|
||||
|
||||
```
|
||||
--
|
||||
John Smith
|
||||
Executive Assistant
|
||||
Phone: 555-1234
|
||||
Fax: 555-2345
|
||||
```
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Dictionary Scanning
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Alert
|
||||
|
||||
* Log of sessions tipped to TRAFFICTHIEF
|
||||
* Searchable fields
|
||||
* Target (strong selector)
|
||||
* Weight (confirmed/unconfirmed)
|
||||
* Other fields
|
||||
* Permutation that triggered the tip (DECODEORDAIN)
|
||||
* Copy of XML document sent to TRAFFICTHIEF
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Category – DNI
|
||||
|
||||
* Category hits from CADENCE and other dictionaries
|
||||
* Searchable fields
|
||||
* Dictionary
|
||||
* Category
|
||||
* Keywords
|
||||
* Target (TRAFFICTHIEF)
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## User Activity
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## User Activity
|
||||
|
||||
* Metadata from applications with a strong selector – webmail, chat, webcam
|
||||
* Searchable fields
|
||||
* Active username ("search value")
|
||||
* Activity – what the active user was doing
|
||||
* Attribute type – type of metadata
|
||||
* Attribute value – metadata value
|
||||
* Source – which plug-in provided the data
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## User Activity
|
||||
|
||||
### Example:
|
||||
|
||||
| Username | Activity | Source | Attribute type | Attribute value |
|
||||
| :--------------- | :----------------- | :------ | :------------- | :-------------- |
|
||||
| badguy@yahoo | viewFolder_webmail | appproc | app_provider | Yahoo |
|
||||
| badguy@yahoo | viewFolder_webmail | appproc | app_type | webmail |
|
||||
| badguy@yahoo | viewFolder_webmail | appproc | direction | client |
|
||||
| badguy@yahoo | viewFolder_webmail | appproc | previous_user | user@yahoo |
|
||||
| badguy@yahoo | viewFolder_webmail | appproc | user_realm | yahoo |
|
||||
| badguy@yahoo | viewFolder_webmail | appproc | via | squid/2.5 |
|
||||
| badguy@yahoo | viewFolder_webmail | appproc | x-forwarded_ip | 10.0.123.45 |
|
||||
| badguy@yahoo | viewFolder_webmail | appproc | yahooBcookie | zxcv1234 |
|
||||
| badguy@yahoo | viewFolder_webmail | appproc | yahooGSS | asdf1234asdf |
|
||||
| asdf1234asdf | viewFolder_webmail | appproc | user_realm | yahooGSS |
|
||||
| asdf1234asdf | viewFolder_webmail | appproc | yahoo | badguy@yahoo |
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Document Processing
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Extracted Files
|
||||
|
||||
* Log of files transmitted as email attachments, web uploads, etc.
|
||||
* Searchable fields
|
||||
* Filename
|
||||
* File extension
|
||||
* File type/MIME type
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Document Tagging
|
||||
|
||||
* Document bodies and email bodies are labeled with hits from a custom second-level dictionary
|
||||
* Idea: "embassy" by itself is not so interesting, but inside a Word document, maybe it is
|
||||
* Searchable fields
|
||||
* Filename
|
||||
* Tech name (tag/category) - government, monetary, proliferation, satellite, wireless, etc.
|
||||
* Tech value - word or phrase that hit
|
||||
* Note: also called Tech Strings search
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Document Metadata
|
||||
|
||||
* Metadata from Office docs, videos, etc.
|
||||
* Searchable fields
|
||||
* Filename and extension, document type
|
||||
* Author, organization
|
||||
* Language
|
||||
* Unique ID
|
||||
* Creation/modification timestamps
|
||||
* Hash of the entire document and any embedded images
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## PDF Metadata
|
||||
|
||||
* Metadata from PDF documents
|
||||
* Searchable fields
|
||||
* Unique ID
|
||||
* Filename
|
||||
* Title
|
||||
* Author, creator, producer
|
||||
* Version
|
||||
* Language
|
||||
* Also available in Document Metadata search
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Protocol Processing
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Blackberry
|
||||
|
||||
* Id numbers and payload info from Blackberry devices
|
||||
* Searchable fields
|
||||
* Source and destination PIN and BES
|
||||
* Direction
|
||||
* Payload type and encoding
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Cellular DNI
|
||||
|
||||
* Metadata from DNI over cellular modems
|
||||
* Searchable fields
|
||||
* IMSI, TMSI, IMEI, MCC, RAC, TLLI, etc.
|
||||
* Cell ID, Tunnel ID, Access point
|
||||
* Latitude, longitude
|
||||
* Spotbeam, direction
|
||||
* Limited deplyment - populated in SOTF by front end
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Cisco Passwords
|
||||
|
||||
* Logs Cisco router passwords
|
||||
* Searchable fields
|
||||
* Password
|
||||
* Decoded password (simple obfuscation with known key)
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## HTTP Activity
|
||||
|
||||
* Metadata from HTTP (WWW) traffic
|
||||
* Searchable fields
|
||||
* Host, URL file path, URL query string
|
||||
* Search terms - parsed from URLs for common search providers (Google, Yahoo)
|
||||
* Language, character encoding
|
||||
* Referrer
|
||||
* User-Agent
|
||||
* Cookies
|
||||
* Server type (Apache, etc.)
|
||||
* Via - proxy info
|
||||
* Geolocation info - e.g. city names from weather reports
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## IKE
|
||||
|
||||
* Metadata from IKE (Internet Key Exchange) sessions
|
||||
* Searchable fields
|
||||
* Version
|
||||
* Vendor ID
|
||||
* Encryption parameters - key length, field size, group curve, etc.
|
||||
* Cookies
|
||||
* Nonce
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## IRC Café Geolocation
|
||||
|
||||
* QUIT messages from IRC - Internet cafés often configure their IRC clients to advertise the café’s street address
|
||||
* Searchable fields:
|
||||
* Username
|
||||
* Nick name
|
||||
* Café address (the QUIT message)
|
||||
* Café IP
|
||||
* Example:
|
||||
|
||||
```
|
||||
:nickname!~username@10.0.27.134 QUIT :Quit: MainStreet
|
||||
Internet Cafe, 350 Main Street, P4 , 512M, Webcam, MP3,
|
||||
128Kbps
|
||||
```
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Passport detection
|
||||
|
||||
* Detect images of passports (code from R6)
|
||||
* OCR machine-readable information
|
||||
* Searchable fields
|
||||
* Original filename
|
||||
* Passport detection score
|
||||
* Info from machine readable area - name, passport number, issuing state, DOB, expiration, etc.
|
||||
* Under development
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Radius Logs
|
||||
|
||||
* Metadata from RADIUS sessions for dial-up authentication and IP assignment
|
||||
* Searchable fields
|
||||
* Username
|
||||
* Phone number
|
||||
* IP address
|
||||
* Account information
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## RBGAN
|
||||
|
||||
* Metadata from RBGAN satellite internet terminal collection
|
||||
* Searchable fields
|
||||
* Username
|
||||
* IMEI
|
||||
* Latitude and longitude
|
||||
* Spotbeam and direction
|
||||
* Limited deployment - populated by front end
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## RTP
|
||||
|
||||
* Metadata from RTP audio and video sessions
|
||||
* Searchable fields
|
||||
* Payload type
|
||||
* SSRC
|
||||
* Number of bytes and packets
|
||||
* Timestamps and sequence numbers
|
||||
* The RTP formatter in the session viewer can decode certain payload types into playable audio or video
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## SIP
|
||||
|
||||
* Metadata from SIP (Session Initiation Protocol) used for VoIP setup, etc.
|
||||
* Stored as multiple type-value pairs per session
|
||||
* Searchable fields
|
||||
* Message type
|
||||
* Attribute type (call-id, content-type, from, to, user-agent, via, etc.)
|
||||
* Attribute value
|
||||
* Subsession ID
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## SSL
|
||||
|
||||
* Metadata from SSL sessions
|
||||
* Searchable fields
|
||||
* Version
|
||||
* Encryption parameters - key length, modulus, exponent, etc.
|
||||
* Signature info
|
||||
* Certificate info
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## TOR Log
|
||||
|
||||
* Logs any identified TOR routers used for anonymizing Internet traffic
|
||||
* Searchable fields
|
||||
* TOR from server
|
||||
* TOR to server
|
||||
* Router nickname
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Web File Transfer
|
||||
|
||||
* Log of uploads and downloads from public file-sharing sites (rapidshare, depositfiles, etc.)
|
||||
* Searchable fields
|
||||
* Filename
|
||||
* File size
|
||||
* Number of downloads
|
||||
* Uploader
|
||||
* Username and password
|
||||
* Under development (GCHQ/MHS)
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Web Proxy
|
||||
|
||||
* Log of X-Forwarded-For IP addresses and other leaked public/private IP
|
||||
* Currently contains XFF plus leaked info from STUN and Google Earth
|
||||
* Searchable fields
|
||||
* Internal from IP
|
||||
* Internal to IP
|
||||
* External from IP
|
||||
* External to IP
|
||||
* Source - plug-in that provided the info
|
||||
* Network path - chain of XFF addresses
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Wireshark
|
||||
|
||||
* Metadata from various protocols processed by the wireshark library
|
||||
* Protocols
|
||||
* Routing - BGP, OSPF
|
||||
* VoIP - H225, Skinny, Clarent, Megaco, SCTP
|
||||
* Net management - SMB, SNMP
|
||||
* Tunneling - GTP
|
||||
* Searchable fields
|
||||
* Protocol
|
||||
* Field name
|
||||
* Field value
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## WLAN
|
||||
|
||||
* Metadata from WLAN collection
|
||||
* Searchable fields
|
||||
* Channel
|
||||
* SSID
|
||||
* BSSID
|
||||
* MAC addresses
|
||||
* Username
|
||||
* Private IP
|
||||
* Limited deplyment - populated in SOTF by front end
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Miscellaneous
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Call Logs
|
||||
|
||||
* DNR metadata from JUGGERNAUT, CERF, FASCIA, DURT, etc.
|
||||
* Searchable fields
|
||||
* Phone numbers
|
||||
* Signaling type
|
||||
* OPC, DPC, CIC, IMSI
|
||||
* Limited deployment
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Network Logs
|
||||
|
||||
* Network metadata from MOONSHINE logs
|
||||
* Searchable fields
|
||||
* Net type
|
||||
* ESSID, BSSID
|
||||
* Channel
|
||||
* Carrier
|
||||
* Latitude and longitude
|
||||
* Limited deployment
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## CNE
|
||||
|
||||
* CNE data from TAO
|
||||
* Searchable fields
|
||||
* Project name
|
||||
* Collection technique
|
||||
* Filename and extension
|
||||
* Limited deployment (xks-cne)
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Registry
|
||||
|
||||
* Windows registry data from TAO (CNE)
|
||||
* Searchable fields
|
||||
* Collection technique
|
||||
* Hive
|
||||
* Key, subkey, value
|
||||
* Limited deployment (xks-cne)
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Custom Search Forms
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Simple Search
|
||||
|
||||
* Simple way to search for usernames, IP addresses, and machine ID cookies
|
||||
* Just enter your search term and select what type of thing it is, and the form sends it to User Activity or HTTP Activity as appropriate
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
---
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
|
||||
## Multi Search
|
||||
|
||||
* Problem: XKS may have info about "badguy@yahoo" in Email Addresses, User Activity, Logins & Passwords, etc.
|
||||
* Solution: submit multiple searches from a single form
|
||||
* Enter the username and select which databases to search, and the form translates that into the proper queries
|
||||
* Similar MultiSearches for IP addresses and MAC addresses
|
||||
* Optional: merge results into one table
|
||||
|
||||
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
|
||||
Loading…
Add table
Add a link
Reference in a new issue