snowchive-meta/data/xks-search-forms/xks-search-forms.md
2026-11-19 04:39:11 +02:00

760 lines
No EOL
17 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
# XKEYSCORE Search Forms
March 2009
DERIVED FROM: NSA/CSSM 1-52
DATED: 20070108
DECLASSIFY ON: 20320108
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Standard search fields
### Wildcards
* `*` - multiple characters anywhere in word
* `_` - single character anywhere in word
* Some fields are auto-wildcarded - the field name will have a `*` before and/or after it
### Operators
* Boolean AND, OR - do not use in the same field
* `!` - NOT (e.g. `!joe`)
* `<`, `>` - comparison (e.g. `>00080`)
* `regex:` - regular expression (e.g. `regex:[0-9]*`)
* Enter `!` to require a field to be non-empty
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Special (full-text) search fields
* Google-like syntax - just list your terms and the query will return sessions that match any of them
* Wildcards only allowed at the end of a word
* Search terms must be at least 4 characters
* Use `+` or `-` to require that a word must or must not be present
* Use `""` to find an exact phrase
* Use `()` for grouping
* You can still use “classic” syntax - we convert it for you
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Special (full-text) search fields
### Examples:
| Search terms | Returned results... |
| :----------------- | :--------------------------------------------------------- |
| apple banana | contain 'apple' or 'banana' or both |
| +apple +juice | contain both 'apple' and 'juice' |
| +apple -macintosh | contain 'apple' but not 'macintosh' |
| +apple +(turnover strudel) | contain 'apple' AND either 'turnover' or 'strudel' |
| apple* | contain words like 'apple' or 'apples' or 'applesauce' or 'applet' |
| "apple juice" | contain the exact phrase 'apple juice' |
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## This plug-in has no data!
* Under development
* Menu items and search forms may show up before a plug-in goes “live” in the field
* Limited deployment
* Some sites run different sets of plug-ins
* Populated by front end
* Some plug-ins simply database metadata provided by the system that feeds XKS, and not all sites are set up the same way
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Simple Logging
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Full Log DNI
* One record for every session processed
* Collection fields
* SIGAD
* Casenotation
* Session ID (UUID)
* Protocol fields
* MAC addresses
* IP addresses
* Port numbers
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Full Log DNI
* Application ID fields
* Application Name – full application ID
* Application Type – top level of application ID
* Application Info – extra info
* Appid+fingerprints – full application ID plus any matching fingerprints
* Example:
* Application name: mail/webmail/yahoo
* Application type: mail
* Application info: viewFolder_webmail
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Full Log DNI
* Fields populated by other plug-ins
* Username (from User Activity)
* Category hits (from Category DNI)
* Client IP/X-Forwarded-For (from Web Proxy)
* Most Full Log search fields are available on every other search form
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Email Addresses
* Anything that looks like an email address
* Searchable fields
* Email username – the part before the @ only!
* Domain – the part after the @
* Subject – email subject, if present
* Example:
```
Sender: user1@yahoo.com
MIME-Version: 1.0
Subject: check this out
Date: Tue, 02 Jan 2007 13:27:31 -0000
Message-ID: <AAAAAAAAAAAAAAAAA123456789@yahoo.com>
From: "User One" <user1@yahoo.com>
To: “User Two" <user2@hotmail.com>
```
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Logins & Passwords
* Anything that looks like a login or password
* Searchable fields
* Username
* Password
* Examples:
```
<input name="username" value="badguy">
<input name="password" value="asdf123">
USER badguy
PASS asdf123
```
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Phone Numbers in DNI
* Anything that looks like a phone number
* Searchable fields
* Phone number
* Number type (fax, telephone, mobile, etc.)
* Example:
```
--
John Smith
Executive Assistant
Phone: 555-1234
Fax: 555-2345
```
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Dictionary Scanning
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Alert
* Log of sessions tipped to TRAFFICTHIEF
* Searchable fields
* Target (strong selector)
* Weight (confirmed/unconfirmed)
* Other fields
* Permutation that triggered the tip (DECODEORDAIN)
* Copy of XML document sent to TRAFFICTHIEF
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Category – DNI
* Category hits from CADENCE and other dictionaries
* Searchable fields
* Dictionary
* Category
* Keywords
* Target (TRAFFICTHIEF)
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## User Activity
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## User Activity
* Metadata from applications with a strong selector – webmail, chat, webcam
* Searchable fields
* Active username ("search value")
* Activity – what the active user was doing
* Attribute type – type of metadata
* Attribute value – metadata value
* Source – which plug-in provided the data
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## User Activity
### Example:
| Username | Activity | Source | Attribute type | Attribute value |
| :--------------- | :----------------- | :------ | :------------- | :-------------- |
| badguy@yahoo | viewFolder_webmail | appproc | app_provider | Yahoo |
| badguy@yahoo | viewFolder_webmail | appproc | app_type | webmail |
| badguy@yahoo | viewFolder_webmail | appproc | direction | client |
| badguy@yahoo | viewFolder_webmail | appproc | previous_user | user@yahoo |
| badguy@yahoo | viewFolder_webmail | appproc | user_realm | yahoo |
| badguy@yahoo | viewFolder_webmail | appproc | via | squid/2.5 |
| badguy@yahoo | viewFolder_webmail | appproc | x-forwarded_ip | 10.0.123.45 |
| badguy@yahoo | viewFolder_webmail | appproc | yahooBcookie | zxcv1234 |
| badguy@yahoo | viewFolder_webmail | appproc | yahooGSS | asdf1234asdf |
| asdf1234asdf | viewFolder_webmail | appproc | user_realm | yahooGSS |
| asdf1234asdf | viewFolder_webmail | appproc | yahoo | badguy@yahoo |
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Document Processing
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Extracted Files
* Log of files transmitted as email attachments, web uploads, etc.
* Searchable fields
* Filename
* File extension
* File type/MIME type
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Document Tagging
* Document bodies and email bodies are labeled with hits from a custom second-level dictionary
* Idea: "embassy" by itself is not so interesting, but inside a Word document, maybe it is
* Searchable fields
* Filename
* Tech name (tag/category) - government, monetary, proliferation, satellite, wireless, etc.
* Tech value - word or phrase that hit
* Note: also called Tech Strings search
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Document Metadata
* Metadata from Office docs, videos, etc.
* Searchable fields
* Filename and extension, document type
* Author, organization
* Language
* Unique ID
* Creation/modification timestamps
* Hash of the entire document and any embedded images
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## PDF Metadata
* Metadata from PDF documents
* Searchable fields
* Unique ID
* Filename
* Title
* Author, creator, producer
* Version
* Language
* Also available in Document Metadata search
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Protocol Processing
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Blackberry
* Id numbers and payload info from Blackberry devices
* Searchable fields
* Source and destination PIN and BES
* Direction
* Payload type and encoding
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Cellular DNI
* Metadata from DNI over cellular modems
* Searchable fields
* IMSI, TMSI, IMEI, MCC, RAC, TLLI, etc.
* Cell ID, Tunnel ID, Access point
* Latitude, longitude
* Spotbeam, direction
* Limited deplyment - populated in SOTF by front end
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Cisco Passwords
* Logs Cisco router passwords
* Searchable fields
* Password
* Decoded password (simple obfuscation with known key)
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## HTTP Activity
* Metadata from HTTP (WWW) traffic
* Searchable fields
* Host, URL file path, URL query string
* Search terms - parsed from URLs for common search providers (Google, Yahoo)
* Language, character encoding
* Referrer
* User-Agent
* Cookies
* Server type (Apache, etc.)
* Via - proxy info
* Geolocation info - e.g. city names from weather reports
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## IKE
* Metadata from IKE (Internet Key Exchange) sessions
* Searchable fields
* Version
* Vendor ID
* Encryption parameters - key length, field size, group curve, etc.
* Cookies
* Nonce
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## IRC Café Geolocation
* QUIT messages from IRC - Internet cafés often configure their IRC clients to advertise the café’s street address
* Searchable fields:
* Username
* Nick name
* Café address (the QUIT message)
* Café IP
* Example:
```
:nickname!~username@10.0.27.134 QUIT :Quit: MainStreet
Internet Cafe, 350 Main Street, P4 , 512M, Webcam, MP3,
128Kbps
```
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Passport detection
* Detect images of passports (code from R6)
* OCR machine-readable information
* Searchable fields
* Original filename
* Passport detection score
* Info from machine readable area - name, passport number, issuing state, DOB, expiration, etc.
* Under development
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Radius Logs
* Metadata from RADIUS sessions for dial-up authentication and IP assignment
* Searchable fields
* Username
* Phone number
* IP address
* Account information
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## RBGAN
* Metadata from RBGAN satellite internet terminal collection
* Searchable fields
* Username
* IMEI
* Latitude and longitude
* Spotbeam and direction
* Limited deployment - populated by front end
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## RTP
* Metadata from RTP audio and video sessions
* Searchable fields
* Payload type
* SSRC
* Number of bytes and packets
* Timestamps and sequence numbers
* The RTP formatter in the session viewer can decode certain payload types into playable audio or video
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## SIP
* Metadata from SIP (Session Initiation Protocol) used for VoIP setup, etc.
* Stored as multiple type-value pairs per session
* Searchable fields
* Message type
* Attribute type (call-id, content-type, from, to, user-agent, via, etc.)
* Attribute value
* Subsession ID
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## SSL
* Metadata from SSL sessions
* Searchable fields
* Version
* Encryption parameters - key length, modulus, exponent, etc.
* Signature info
* Certificate info
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## TOR Log
* Logs any identified TOR routers used for anonymizing Internet traffic
* Searchable fields
* TOR from server
* TOR to server
* Router nickname
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Web File Transfer
* Log of uploads and downloads from public file-sharing sites (rapidshare, depositfiles, etc.)
* Searchable fields
* Filename
* File size
* Number of downloads
* Uploader
* Username and password
* Under development (GCHQ/MHS)
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Web Proxy
* Log of X-Forwarded-For IP addresses and other leaked public/private IP
* Currently contains XFF plus leaked info from STUN and Google Earth
* Searchable fields
* Internal from IP
* Internal to IP
* External from IP
* External to IP
* Source - plug-in that provided the info
* Network path - chain of XFF addresses
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Wireshark
* Metadata from various protocols processed by the wireshark library
* Protocols
* Routing - BGP, OSPF
* VoIP - H225, Skinny, Clarent, Megaco, SCTP
* Net management - SMB, SNMP
* Tunneling - GTP
* Searchable fields
* Protocol
* Field name
* Field value
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## WLAN
* Metadata from WLAN collection
* Searchable fields
* Channel
* SSID
* BSSID
* MAC addresses
* Username
* Private IP
* Limited deplyment - populated in SOTF by front end
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Miscellaneous
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Call Logs
* DNR metadata from JUGGERNAUT, CERF, FASCIA, DURT, etc.
* Searchable fields
* Phone numbers
* Signaling type
* OPC, DPC, CIC, IMSI
* Limited deployment
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Network Logs
* Network metadata from MOONSHINE logs
* Searchable fields
* Net type
* ESSID, BSSID
* Channel
* Carrier
* Latitude and longitude
* Limited deployment
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## CNE
* CNE data from TAO
* Searchable fields
* Project name
* Collection technique
* Filename and extension
* Limited deployment (xks-cne)
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Registry
* Windows registry data from TAO (CNE)
* Searchable fields
* Collection technique
* Hive
* Key, subkey, value
* Limited deployment (xks-cne)
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Custom Search Forms
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Simple Search
* Simple way to search for usernames, IP addresses, and machine ID cookies
* Just enter your search term and select what type of thing it is, and the form sends it to User Activity or HTTP Activity as appropriate
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
---
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
## Multi Search
* Problem: XKS may have info about "badguy@yahoo" in Email Addresses, User Activity, Logins & Passwords, etc.
* Solution: submit multiple searches from a single form
* Enter the username and select which databases to search, and the form translates that into the proper queries
* Similar MultiSearches for IP addresses and MAC addresses
* Optional: merge results into one table
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL