17 KiB
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
XKEYSCORE Search Forms
March 2009
DERIVED FROM: NSA/CSSM 1-52 DATED: 20070108 DECLASSIFY ON: 20320108
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Standard search fields
Wildcards
*- multiple characters anywhere in word_- single character anywhere in word- Some fields are auto-wildcarded - the field name will have a
*before and/or after it
Operators
- Boolean AND, OR - do not use in the same field
!- NOT (e.g.!joe)<,>- comparison (e.g.>00080)regex:- regular expression (e.g.regex:[0-9]*)- Enter
!to require a field to be non-empty
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Special (full-text) search fields
- Google-like syntax - just list your terms and the query will return sessions that match any of them
- Wildcards only allowed at the end of a word
- Search terms must be at least 4 characters
- Use
+or-to require that a word must or must not be present - Use
""to find an exact phrase - Use
()for grouping - You can still use “classic” syntax - we convert it for you
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Special (full-text) search fields
Examples:
| Search terms | Returned results... |
|---|---|
| apple banana | contain 'apple' or 'banana' or both |
| +apple +juice | contain both 'apple' and 'juice' |
| +apple -macintosh | contain 'apple' but not 'macintosh' |
| +apple +(turnover strudel) | contain 'apple' AND either 'turnover' or 'strudel' |
| apple* | contain words like 'apple' or 'apples' or 'applesauce' or 'applet' |
| "apple juice" | contain the exact phrase 'apple juice' |
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
This plug-in has no data!
- Under development
- Menu items and search forms may show up before a plug-in goes “live” in the field
- Limited deployment
- Some sites run different sets of plug-ins
- Populated by front end
- Some plug-ins simply database metadata provided by the system that feeds XKS, and not all sites are set up the same way
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Simple Logging
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Full Log DNI
- One record for every session processed
- Collection fields
- SIGAD
- Casenotation
- Session ID (UUID)
- Protocol fields
- MAC addresses
- IP addresses
- Port numbers
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Full Log DNI
- Application ID fields
- Application Name – full application ID
- Application Type – top level of application ID
- Application Info – extra info
- Appid+fingerprints – full application ID plus any matching fingerprints
- Example:
- Application name: mail/webmail/yahoo
- Application type: mail
- Application info: viewFolder_webmail
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Full Log DNI
- Fields populated by other plug-ins
- Username (from User Activity)
- Category hits (from Category DNI)
- Client IP/X-Forwarded-For (from Web Proxy)
- Most Full Log search fields are available on every other search form
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Email Addresses
- Anything that looks like an email address
- Searchable fields
- Email username – the part before the @ only!
- Domain – the part after the @
- Subject – email subject, if present
- Example:
Sender: user1@yahoo.com
MIME-Version: 1.0
Subject: check this out
Date: Tue, 02 Jan 2007 13:27:31 -0000
Message-ID: <AAAAAAAAAAAAAAAAA123456789@yahoo.com>
From: "User One" <user1@yahoo.com>
To: “User Two" <user2@hotmail.com>
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Logins & Passwords
- Anything that looks like a login or password
- Searchable fields
- Username
- Password
- Examples:
<input name="username" value="badguy">
<input name="password" value="asdf123">
USER badguy
PASS asdf123
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Phone Numbers in DNI
- Anything that looks like a phone number
- Searchable fields
- Phone number
- Number type (fax, telephone, mobile, etc.)
- Example:
--
John Smith
Executive Assistant
Phone: 555-1234
Fax: 555-2345
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Dictionary Scanning
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Alert
- Log of sessions tipped to TRAFFICTHIEF
- Searchable fields
- Target (strong selector)
- Weight (confirmed/unconfirmed)
- Other fields
- Permutation that triggered the tip (DECODEORDAIN)
- Copy of XML document sent to TRAFFICTHIEF
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Category – DNI
- Category hits from CADENCE and other dictionaries
- Searchable fields
- Dictionary
- Category
- Keywords
- Target (TRAFFICTHIEF)
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
User Activity
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
User Activity
- Metadata from applications with a strong selector – webmail, chat, webcam
- Searchable fields
- Active username ("search value")
- Activity – what the active user was doing
- Attribute type – type of metadata
- Attribute value – metadata value
- Source – which plug-in provided the data
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
User Activity
Example:
| Username | Activity | Source | Attribute type | Attribute value |
|---|---|---|---|---|
| badguy@yahoo | viewFolder_webmail | appproc | app_provider | Yahoo |
| badguy@yahoo | viewFolder_webmail | appproc | app_type | webmail |
| badguy@yahoo | viewFolder_webmail | appproc | direction | client |
| badguy@yahoo | viewFolder_webmail | appproc | previous_user | user@yahoo |
| badguy@yahoo | viewFolder_webmail | appproc | user_realm | yahoo |
| badguy@yahoo | viewFolder_webmail | appproc | via | squid/2.5 |
| badguy@yahoo | viewFolder_webmail | appproc | x-forwarded_ip | 10.0.123.45 |
| badguy@yahoo | viewFolder_webmail | appproc | yahooBcookie | zxcv1234 |
| badguy@yahoo | viewFolder_webmail | appproc | yahooGSS | asdf1234asdf |
| asdf1234asdf | viewFolder_webmail | appproc | user_realm | yahooGSS |
| asdf1234asdf | viewFolder_webmail | appproc | yahoo | badguy@yahoo |
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Document Processing
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Extracted Files
- Log of files transmitted as email attachments, web uploads, etc.
- Searchable fields
- Filename
- File extension
- File type/MIME type
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Document Tagging
- Document bodies and email bodies are labeled with hits from a custom second-level dictionary
- Idea: "embassy" by itself is not so interesting, but inside a Word document, maybe it is
- Searchable fields
- Filename
- Tech name (tag/category) - government, monetary, proliferation, satellite, wireless, etc.
- Tech value - word or phrase that hit
- Note: also called Tech Strings search
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Document Metadata
- Metadata from Office docs, videos, etc.
- Searchable fields
- Filename and extension, document type
- Author, organization
- Language
- Unique ID
- Creation/modification timestamps
- Hash of the entire document and any embedded images
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
PDF Metadata
- Metadata from PDF documents
- Searchable fields
- Unique ID
- Filename
- Title
- Author, creator, producer
- Version
- Language
- Also available in Document Metadata search
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Protocol Processing
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Blackberry
- Id numbers and payload info from Blackberry devices
- Searchable fields
- Source and destination PIN and BES
- Direction
- Payload type and encoding
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Cellular DNI
- Metadata from DNI over cellular modems
- Searchable fields
- IMSI, TMSI, IMEI, MCC, RAC, TLLI, etc.
- Cell ID, Tunnel ID, Access point
- Latitude, longitude
- Spotbeam, direction
- Limited deplyment - populated in SOTF by front end
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Cisco Passwords
- Logs Cisco router passwords
- Searchable fields
- Password
- Decoded password (simple obfuscation with known key)
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
HTTP Activity
- Metadata from HTTP (WWW) traffic
- Searchable fields
- Host, URL file path, URL query string
- Search terms - parsed from URLs for common search providers (Google, Yahoo)
- Language, character encoding
- Referrer
- User-Agent
- Cookies
- Server type (Apache, etc.)
- Via - proxy info
- Geolocation info - e.g. city names from weather reports
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
IKE
- Metadata from IKE (Internet Key Exchange) sessions
- Searchable fields
- Version
- Vendor ID
- Encryption parameters - key length, field size, group curve, etc.
- Cookies
- Nonce
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
IRC Café Geolocation
- QUIT messages from IRC - Internet cafés often configure their IRC clients to advertise the café’s street address
- Searchable fields:
- Username
- Nick name
- Café address (the QUIT message)
- Café IP
- Example:
:nickname!~username@10.0.27.134 QUIT :Quit: MainStreet
Internet Cafe, 350 Main Street, P4 , 512M, Webcam, MP3,
128Kbps
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Passport detection
- Detect images of passports (code from R6)
- OCR machine-readable information
- Searchable fields
- Original filename
- Passport detection score
- Info from machine readable area - name, passport number, issuing state, DOB, expiration, etc.
- Under development
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Radius Logs
- Metadata from RADIUS sessions for dial-up authentication and IP assignment
- Searchable fields
- Username
- Phone number
- IP address
- Account information
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
RBGAN
- Metadata from RBGAN satellite internet terminal collection
- Searchable fields
- Username
- IMEI
- Latitude and longitude
- Spotbeam and direction
- Limited deployment - populated by front end
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
RTP
- Metadata from RTP audio and video sessions
- Searchable fields
- Payload type
- SSRC
- Number of bytes and packets
- Timestamps and sequence numbers
- The RTP formatter in the session viewer can decode certain payload types into playable audio or video
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SIP
- Metadata from SIP (Session Initiation Protocol) used for VoIP setup, etc.
- Stored as multiple type-value pairs per session
- Searchable fields
- Message type
- Attribute type (call-id, content-type, from, to, user-agent, via, etc.)
- Attribute value
- Subsession ID
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SSL
- Metadata from SSL sessions
- Searchable fields
- Version
- Encryption parameters - key length, modulus, exponent, etc.
- Signature info
- Certificate info
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOR Log
- Logs any identified TOR routers used for anonymizing Internet traffic
- Searchable fields
- TOR from server
- TOR to server
- Router nickname
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Web File Transfer
- Log of uploads and downloads from public file-sharing sites (rapidshare, depositfiles, etc.)
- Searchable fields
- Filename
- File size
- Number of downloads
- Uploader
- Username and password
- Under development (GCHQ/MHS)
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Web Proxy
- Log of X-Forwarded-For IP addresses and other leaked public/private IP
- Currently contains XFF plus leaked info from STUN and Google Earth
- Searchable fields
- Internal from IP
- Internal to IP
- External from IP
- External to IP
- Source - plug-in that provided the info
- Network path - chain of XFF addresses
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Wireshark
- Metadata from various protocols processed by the wireshark library
- Protocols
- Routing - BGP, OSPF
- VoIP - H225, Skinny, Clarent, Megaco, SCTP
- Net management - SMB, SNMP
- Tunneling - GTP
- Searchable fields
- Protocol
- Field name
- Field value
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
WLAN
- Metadata from WLAN collection
- Searchable fields
- Channel
- SSID
- BSSID
- MAC addresses
- Username
- Private IP
- Limited deplyment - populated in SOTF by front end
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Miscellaneous
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Call Logs
- DNR metadata from JUGGERNAUT, CERF, FASCIA, DURT, etc.
- Searchable fields
- Phone numbers
- Signaling type
- OPC, DPC, CIC, IMSI
- Limited deployment
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Network Logs
- Network metadata from MOONSHINE logs
- Searchable fields
- Net type
- ESSID, BSSID
- Channel
- Carrier
- Latitude and longitude
- Limited deployment
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
CNE
- CNE data from TAO
- Searchable fields
- Project name
- Collection technique
- Filename and extension
- Limited deployment (xks-cne)
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Registry
- Windows registry data from TAO (CNE)
- Searchable fields
- Collection technique
- Hive
- Key, subkey, value
- Limited deployment (xks-cne)
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Custom Search Forms
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Simple Search
- Simple way to search for usernames, IP addresses, and machine ID cookies
- Just enter your search term and select what type of thing it is, and the form sends it to User Activity or HTTP Activity as appropriate
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Multi Search
- Problem: XKS may have info about "badguy@yahoo" in Email Addresses, User Activity, Logins & Passwords, etc.
- Solution: submit multiple searches from a single form
- Enter the username and select which databases to search, and the form translates that into the proper queries
- Similar MultiSearches for IP addresses and MAC addresses
- Optional: merge results into one table
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL