first commit

This commit is contained in:
libroot 2026-11-19 04:39:11 +02:00
commit 0d90363116
Signed by: libroot
GPG key ID: 2AC68C029EBE9ACE
6510 changed files with 679146 additions and 0 deletions

View file

@ -0,0 +1,760 @@
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
# XKEYSCORE Search Forms
March 2009
DERIVED FROM: NSA/CSSM 1-52
DATED: 20070108
DECLASSIFY ON: 20320108
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Standard search fields
### Wildcards
* `*` - multiple characters anywhere in word
* `_` - single character anywhere in word
* Some fields are auto-wildcarded - the field name will have a `*` before and/or after it
### Operators
* Boolean AND, OR - do not use in the same field
* `!` - NOT (e.g. `!joe`)
* `<`, `>` - comparison (e.g. `>00080`)
* `regex:` - regular expression (e.g. `regex:[0-9]*`)
* Enter `!` to require a field to be non-empty
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Special (full-text) search fields
* Google-like syntax - just list your terms and the query will return sessions that match any of them
* Wildcards only allowed at the end of a word
* Search terms must be at least 4 characters
* Use `+` or `-` to require that a word must or must not be present
* Use `""` to find an exact phrase
* Use `()` for grouping
* You can still use “classic” syntax - we convert it for you
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Special (full-text) search fields
### Examples:
| Search terms | Returned results... |
| :----------------- | :--------------------------------------------------------- |
| apple banana | contain 'apple' or 'banana' or both |
| +apple +juice | contain both 'apple' and 'juice' |
| +apple -macintosh | contain 'apple' but not 'macintosh' |
| +apple +(turnover strudel) | contain 'apple' AND either 'turnover' or 'strudel' |
| apple* | contain words like 'apple' or 'apples' or 'applesauce' or 'applet' |
| "apple juice" | contain the exact phrase 'apple juice' |
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## This plug-in has no data!
* Under development
* Menu items and search forms may show up before a plug-in goes “live” in the field
* Limited deployment
* Some sites run different sets of plug-ins
* Populated by front end
* Some plug-ins simply database metadata provided by the system that feeds XKS, and not all sites are set up the same way
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Simple Logging
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Full Log DNI
* One record for every session processed
* Collection fields
* SIGAD
* Casenotation
* Session ID (UUID)
* Protocol fields
* MAC addresses
* IP addresses
* Port numbers
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Full Log DNI
* Application ID fields
* Application Name – full application ID
* Application Type – top level of application ID
* Application Info – extra info
* Appid+fingerprints – full application ID plus any matching fingerprints
* Example:
* Application name: mail/webmail/yahoo
* Application type: mail
* Application info: viewFolder_webmail
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Full Log DNI
* Fields populated by other plug-ins
* Username (from User Activity)
* Category hits (from Category DNI)
* Client IP/X-Forwarded-For (from Web Proxy)
* Most Full Log search fields are available on every other search form
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Email Addresses
* Anything that looks like an email address
* Searchable fields
* Email username – the part before the @ only!
* Domain – the part after the @
* Subject – email subject, if present
* Example:
```
Sender: user1@yahoo.com
MIME-Version: 1.0
Subject: check this out
Date: Tue, 02 Jan 2007 13:27:31 -0000
Message-ID: <AAAAAAAAAAAAAAAAA123456789@yahoo.com>
From: "User One" <user1@yahoo.com>
To: “User Two" <user2@hotmail.com>
```
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Logins & Passwords
* Anything that looks like a login or password
* Searchable fields
* Username
* Password
* Examples:
```
<input name="username" value="badguy">
<input name="password" value="asdf123">
USER badguy
PASS asdf123
```
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Phone Numbers in DNI
* Anything that looks like a phone number
* Searchable fields
* Phone number
* Number type (fax, telephone, mobile, etc.)
* Example:
```
--
John Smith
Executive Assistant
Phone: 555-1234
Fax: 555-2345
```
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Dictionary Scanning
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Alert
* Log of sessions tipped to TRAFFICTHIEF
* Searchable fields
* Target (strong selector)
* Weight (confirmed/unconfirmed)
* Other fields
* Permutation that triggered the tip (DECODEORDAIN)
* Copy of XML document sent to TRAFFICTHIEF
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Category – DNI
* Category hits from CADENCE and other dictionaries
* Searchable fields
* Dictionary
* Category
* Keywords
* Target (TRAFFICTHIEF)
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## User Activity
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## User Activity
* Metadata from applications with a strong selector – webmail, chat, webcam
* Searchable fields
* Active username ("search value")
* Activity – what the active user was doing
* Attribute type – type of metadata
* Attribute value – metadata value
* Source – which plug-in provided the data
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## User Activity
### Example:
| Username | Activity | Source | Attribute type | Attribute value |
| :--------------- | :----------------- | :------ | :------------- | :-------------- |
| badguy@yahoo | viewFolder_webmail | appproc | app_provider | Yahoo |
| badguy@yahoo | viewFolder_webmail | appproc | app_type | webmail |
| badguy@yahoo | viewFolder_webmail | appproc | direction | client |
| badguy@yahoo | viewFolder_webmail | appproc | previous_user | user@yahoo |
| badguy@yahoo | viewFolder_webmail | appproc | user_realm | yahoo |
| badguy@yahoo | viewFolder_webmail | appproc | via | squid/2.5 |
| badguy@yahoo | viewFolder_webmail | appproc | x-forwarded_ip | 10.0.123.45 |
| badguy@yahoo | viewFolder_webmail | appproc | yahooBcookie | zxcv1234 |
| badguy@yahoo | viewFolder_webmail | appproc | yahooGSS | asdf1234asdf |
| asdf1234asdf | viewFolder_webmail | appproc | user_realm | yahooGSS |
| asdf1234asdf | viewFolder_webmail | appproc | yahoo | badguy@yahoo |
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Document Processing
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Extracted Files
* Log of files transmitted as email attachments, web uploads, etc.
* Searchable fields
* Filename
* File extension
* File type/MIME type
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Document Tagging
* Document bodies and email bodies are labeled with hits from a custom second-level dictionary
* Idea: "embassy" by itself is not so interesting, but inside a Word document, maybe it is
* Searchable fields
* Filename
* Tech name (tag/category) - government, monetary, proliferation, satellite, wireless, etc.
* Tech value - word or phrase that hit
* Note: also called Tech Strings search
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Document Metadata
* Metadata from Office docs, videos, etc.
* Searchable fields
* Filename and extension, document type
* Author, organization
* Language
* Unique ID
* Creation/modification timestamps
* Hash of the entire document and any embedded images
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## PDF Metadata
* Metadata from PDF documents
* Searchable fields
* Unique ID
* Filename
* Title
* Author, creator, producer
* Version
* Language
* Also available in Document Metadata search
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Protocol Processing
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Blackberry
* Id numbers and payload info from Blackberry devices
* Searchable fields
* Source and destination PIN and BES
* Direction
* Payload type and encoding
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Cellular DNI
* Metadata from DNI over cellular modems
* Searchable fields
* IMSI, TMSI, IMEI, MCC, RAC, TLLI, etc.
* Cell ID, Tunnel ID, Access point
* Latitude, longitude
* Spotbeam, direction
* Limited deplyment - populated in SOTF by front end
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Cisco Passwords
* Logs Cisco router passwords
* Searchable fields
* Password
* Decoded password (simple obfuscation with known key)
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## HTTP Activity
* Metadata from HTTP (WWW) traffic
* Searchable fields
* Host, URL file path, URL query string
* Search terms - parsed from URLs for common search providers (Google, Yahoo)
* Language, character encoding
* Referrer
* User-Agent
* Cookies
* Server type (Apache, etc.)
* Via - proxy info
* Geolocation info - e.g. city names from weather reports
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## IKE
* Metadata from IKE (Internet Key Exchange) sessions
* Searchable fields
* Version
* Vendor ID
* Encryption parameters - key length, field size, group curve, etc.
* Cookies
* Nonce
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## IRC Café Geolocation
* QUIT messages from IRC - Internet cafés often configure their IRC clients to advertise the café’s street address
* Searchable fields:
* Username
* Nick name
* Café address (the QUIT message)
* Café IP
* Example:
```
:nickname!~username@10.0.27.134 QUIT :Quit: MainStreet
Internet Cafe, 350 Main Street, P4 , 512M, Webcam, MP3,
128Kbps
```
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Passport detection
* Detect images of passports (code from R6)
* OCR machine-readable information
* Searchable fields
* Original filename
* Passport detection score
* Info from machine readable area - name, passport number, issuing state, DOB, expiration, etc.
* Under development
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Radius Logs
* Metadata from RADIUS sessions for dial-up authentication and IP assignment
* Searchable fields
* Username
* Phone number
* IP address
* Account information
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## RBGAN
* Metadata from RBGAN satellite internet terminal collection
* Searchable fields
* Username
* IMEI
* Latitude and longitude
* Spotbeam and direction
* Limited deployment - populated by front end
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## RTP
* Metadata from RTP audio and video sessions
* Searchable fields
* Payload type
* SSRC
* Number of bytes and packets
* Timestamps and sequence numbers
* The RTP formatter in the session viewer can decode certain payload types into playable audio or video
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## SIP
* Metadata from SIP (Session Initiation Protocol) used for VoIP setup, etc.
* Stored as multiple type-value pairs per session
* Searchable fields
* Message type
* Attribute type (call-id, content-type, from, to, user-agent, via, etc.)
* Attribute value
* Subsession ID
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## SSL
* Metadata from SSL sessions
* Searchable fields
* Version
* Encryption parameters - key length, modulus, exponent, etc.
* Signature info
* Certificate info
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## TOR Log
* Logs any identified TOR routers used for anonymizing Internet traffic
* Searchable fields
* TOR from server
* TOR to server
* Router nickname
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Web File Transfer
* Log of uploads and downloads from public file-sharing sites (rapidshare, depositfiles, etc.)
* Searchable fields
* Filename
* File size
* Number of downloads
* Uploader
* Username and password
* Under development (GCHQ/MHS)
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Web Proxy
* Log of X-Forwarded-For IP addresses and other leaked public/private IP
* Currently contains XFF plus leaked info from STUN and Google Earth
* Searchable fields
* Internal from IP
* Internal to IP
* External from IP
* External to IP
* Source - plug-in that provided the info
* Network path - chain of XFF addresses
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Wireshark
* Metadata from various protocols processed by the wireshark library
* Protocols
* Routing - BGP, OSPF
* VoIP - H225, Skinny, Clarent, Megaco, SCTP
* Net management - SMB, SNMP
* Tunneling - GTP
* Searchable fields
* Protocol
* Field name
* Field value
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## WLAN
* Metadata from WLAN collection
* Searchable fields
* Channel
* SSID
* BSSID
* MAC addresses
* Username
* Private IP
* Limited deplyment - populated in SOTF by front end
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Miscellaneous
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Call Logs
* DNR metadata from JUGGERNAUT, CERF, FASCIA, DURT, etc.
* Searchable fields
* Phone numbers
* Signaling type
* OPC, DPC, CIC, IMSI
* Limited deployment
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Network Logs
* Network metadata from MOONSHINE logs
* Searchable fields
* Net type
* ESSID, BSSID
* Channel
* Carrier
* Latitude and longitude
* Limited deployment
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## CNE
* CNE data from TAO
* Searchable fields
* Project name
* Collection technique
* Filename and extension
* Limited deployment (xks-cne)
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Registry
* Windows registry data from TAO (CNE)
* Searchable fields
* Collection technique
* Hive
* Key, subkey, value
* Limited deployment (xks-cne)
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Custom Search Forms
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Simple Search
* Simple way to search for usernames, IP addresses, and machine ID cookies
* Just enter your search term and select what type of thing it is, and the form sends it to User Activity or HTTP Activity as appropriate
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Multi Search
* Problem: XKS may have info about "badguy@yahoo" in Email Addresses, User Activity, Logins & Passwords, etc.
* Solution: submit multiple searches from a single form
* Enter the username and select which databases to search, and the form translates that into the proper queries
* Similar MultiSearches for IP addresses and MAC addresses
* Optional: merge results into one table
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL