snowchive-meta/data/xks-search-forms/xks-search-forms.md
2026-11-19 04:39:11 +02:00

17 KiB
Raw Permalink Blame History

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

XKEYSCORE Search Forms

March 2009

DERIVED FROM: NSA/CSSM 1-52 DATED: 20070108 DECLASSIFY ON: 20320108

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Standard search fields

Wildcards

  • * - multiple characters anywhere in word
  • _ - single character anywhere in word
  • Some fields are auto-wildcarded - the field name will have a * before and/or after it

Operators

  • Boolean AND, OR - do not use in the same field
  • ! - NOT (e.g. !joe)
  • <, > - comparison (e.g. >00080)
  • regex: - regular expression (e.g. regex:[0-9]*)
  • Enter ! to require a field to be non-empty

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Special (full-text) search fields

  • Google-like syntax - just list your terms and the query will return sessions that match any of them
  • Wildcards only allowed at the end of a word
  • Search terms must be at least 4 characters
  • Use + or - to require that a word must or must not be present
  • Use "" to find an exact phrase
  • Use () for grouping
  • You can still use “classic” syntax - we convert it for you

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Special (full-text) search fields

Examples:

Search terms Returned results...
apple banana contain 'apple' or 'banana' or both
+apple +juice contain both 'apple' and 'juice'
+apple -macintosh contain 'apple' but not 'macintosh'
+apple +(turnover strudel) contain 'apple' AND either 'turnover' or 'strudel'
apple* contain words like 'apple' or 'apples' or 'applesauce' or 'applet'
"apple juice" contain the exact phrase 'apple juice'

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

This plug-in has no data!

  • Under development
    • Menu items and search forms may show up before a plug-in goes “live” in the field
  • Limited deployment
    • Some sites run different sets of plug-ins
  • Populated by front end
    • Some plug-ins simply database metadata provided by the system that feeds XKS, and not all sites are set up the same way

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Simple Logging


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Full Log DNI

  • One record for every session processed
  • Collection fields
    • SIGAD
    • Casenotation
    • Session ID (UUID)
  • Protocol fields
    • MAC addresses
    • IP addresses
    • Port numbers

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Full Log DNI

  • Application ID fields
    • Application Name – full application ID
    • Application Type – top level of application ID
    • Application Info – extra info
    • Appid+fingerprints – full application ID plus any matching fingerprints
  • Example:
    • Application name: mail/webmail/yahoo
    • Application type: mail
    • Application info: viewFolder_webmail

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Full Log DNI

  • Fields populated by other plug-ins
    • Username (from User Activity)
    • Category hits (from Category DNI)
    • Client IP/X-Forwarded-For (from Web Proxy)
  • Most Full Log search fields are available on every other search form

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Email Addresses

  • Anything that looks like an email address
  • Searchable fields
    • Email username – the part before the @ only!
    • Domain – the part after the @
    • Subject – email subject, if present
  • Example:
Sender: user1@yahoo.com
MIME-Version: 1.0
Subject: check this out
Date: Tue, 02 Jan 2007 13:27:31 -0000
Message-ID: <AAAAAAAAAAAAAAAAA123456789@yahoo.com>
From: "User One" <user1@yahoo.com>
To: “User Two" <user2@hotmail.com>

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Logins & Passwords

  • Anything that looks like a login or password
  • Searchable fields
    • Username
    • Password
  • Examples:
<input name="username" value="badguy">
<input name="password" value="asdf123">

USER badguy
PASS asdf123

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Phone Numbers in DNI

  • Anything that looks like a phone number
  • Searchable fields
    • Phone number
    • Number type (fax, telephone, mobile, etc.)
  • Example:
--
John Smith
Executive Assistant
Phone: 555-1234
Fax: 555-2345

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Dictionary Scanning

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Alert

  • Log of sessions tipped to TRAFFICTHIEF
  • Searchable fields
    • Target (strong selector)
    • Weight (confirmed/unconfirmed)
  • Other fields
    • Permutation that triggered the tip (DECODEORDAIN)
    • Copy of XML document sent to TRAFFICTHIEF

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Category – DNI

  • Category hits from CADENCE and other dictionaries
  • Searchable fields
    • Dictionary
    • Category
    • Keywords
    • Target (TRAFFICTHIEF)

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

User Activity

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

User Activity

  • Metadata from applications with a strong selector – webmail, chat, webcam
  • Searchable fields
    • Active username ("search value")
    • Activity – what the active user was doing
    • Attribute type – type of metadata
    • Attribute value – metadata value
    • Source – which plug-in provided the data

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

User Activity

Example:

Username Activity Source Attribute type Attribute value
badguy@yahoo viewFolder_webmail appproc app_provider Yahoo
badguy@yahoo viewFolder_webmail appproc app_type webmail
badguy@yahoo viewFolder_webmail appproc direction client
badguy@yahoo viewFolder_webmail appproc previous_user user@yahoo
badguy@yahoo viewFolder_webmail appproc user_realm yahoo
badguy@yahoo viewFolder_webmail appproc via squid/2.5
badguy@yahoo viewFolder_webmail appproc x-forwarded_ip 10.0.123.45
badguy@yahoo viewFolder_webmail appproc yahooBcookie zxcv1234
badguy@yahoo viewFolder_webmail appproc yahooGSS asdf1234asdf
asdf1234asdf viewFolder_webmail appproc user_realm yahooGSS
asdf1234asdf viewFolder_webmail appproc yahoo badguy@yahoo

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Document Processing

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Extracted Files

  • Log of files transmitted as email attachments, web uploads, etc.
  • Searchable fields
    • Filename
    • File extension
    • File type/MIME type

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Document Tagging

  • Document bodies and email bodies are labeled with hits from a custom second-level dictionary
  • Idea: "embassy" by itself is not so interesting, but inside a Word document, maybe it is
  • Searchable fields
    • Filename
    • Tech name (tag/category) - government, monetary, proliferation, satellite, wireless, etc.
    • Tech value - word or phrase that hit
  • Note: also called Tech Strings search

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Document Metadata

  • Metadata from Office docs, videos, etc.
  • Searchable fields
    • Filename and extension, document type
    • Author, organization
    • Language
    • Unique ID
    • Creation/modification timestamps
    • Hash of the entire document and any embedded images

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

PDF Metadata

  • Metadata from PDF documents
  • Searchable fields
    • Unique ID
    • Filename
    • Title
    • Author, creator, producer
    • Version
    • Language
  • Also available in Document Metadata search

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Protocol Processing

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Blackberry

  • Id numbers and payload info from Blackberry devices
  • Searchable fields
    • Source and destination PIN and BES
    • Direction
    • Payload type and encoding

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Cellular DNI

  • Metadata from DNI over cellular modems
  • Searchable fields
    • IMSI, TMSI, IMEI, MCC, RAC, TLLI, etc.
    • Cell ID, Tunnel ID, Access point
    • Latitude, longitude
    • Spotbeam, direction
  • Limited deplyment - populated in SOTF by front end

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Cisco Passwords

  • Logs Cisco router passwords
  • Searchable fields
    • Password
    • Decoded password (simple obfuscation with known key)

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Activity

  • Metadata from HTTP (WWW) traffic
  • Searchable fields
    • Host, URL file path, URL query string
    • Search terms - parsed from URLs for common search providers (Google, Yahoo)
    • Language, character encoding
    • Referrer
    • User-Agent
    • Cookies
    • Server type (Apache, etc.)
    • Via - proxy info
    • Geolocation info - e.g. city names from weather reports

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

IKE

  • Metadata from IKE (Internet Key Exchange) sessions
  • Searchable fields
    • Version
    • Vendor ID
    • Encryption parameters - key length, field size, group curve, etc.
    • Cookies
    • Nonce

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

IRC Café Geolocation

  • QUIT messages from IRC - Internet cafés often configure their IRC clients to advertise the café’s street address
  • Searchable fields:
    • Username
    • Nick name
    • Café address (the QUIT message)
    • Café IP
  • Example:
:nickname!~username@10.0.27.134 QUIT :Quit: MainStreet
Internet Cafe, 350 Main Street, P4 , 512M, Webcam, MP3,
128Kbps

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Passport detection

  • Detect images of passports (code from R6)
  • OCR machine-readable information
  • Searchable fields
    • Original filename
    • Passport detection score
    • Info from machine readable area - name, passport number, issuing state, DOB, expiration, etc.
  • Under development

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Radius Logs

  • Metadata from RADIUS sessions for dial-up authentication and IP assignment
  • Searchable fields
    • Username
    • Phone number
    • IP address
    • Account information

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

RBGAN

  • Metadata from RBGAN satellite internet terminal collection
  • Searchable fields
    • Username
    • IMEI
    • Latitude and longitude
    • Spotbeam and direction
  • Limited deployment - populated by front end

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

RTP

  • Metadata from RTP audio and video sessions
  • Searchable fields
    • Payload type
    • SSRC
    • Number of bytes and packets
    • Timestamps and sequence numbers
  • The RTP formatter in the session viewer can decode certain payload types into playable audio or video

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

SIP

  • Metadata from SIP (Session Initiation Protocol) used for VoIP setup, etc.
  • Stored as multiple type-value pairs per session
  • Searchable fields
    • Message type
    • Attribute type (call-id, content-type, from, to, user-agent, via, etc.)
    • Attribute value
    • Subsession ID

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

SSL

  • Metadata from SSL sessions
  • Searchable fields
    • Version
    • Encryption parameters - key length, modulus, exponent, etc.
    • Signature info
    • Certificate info

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

TOR Log

  • Logs any identified TOR routers used for anonymizing Internet traffic
  • Searchable fields
    • TOR from server
    • TOR to server
    • Router nickname

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Web File Transfer

  • Log of uploads and downloads from public file-sharing sites (rapidshare, depositfiles, etc.)
  • Searchable fields
    • Filename
    • File size
    • Number of downloads
    • Uploader
    • Username and password
  • Under development (GCHQ/MHS)

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Web Proxy

  • Log of X-Forwarded-For IP addresses and other leaked public/private IP
  • Currently contains XFF plus leaked info from STUN and Google Earth
  • Searchable fields
    • Internal from IP
    • Internal to IP
    • External from IP
    • External to IP
    • Source - plug-in that provided the info
    • Network path - chain of XFF addresses

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Wireshark

  • Metadata from various protocols processed by the wireshark library
  • Protocols
    • Routing - BGP, OSPF
    • VoIP - H225, Skinny, Clarent, Megaco, SCTP
    • Net management - SMB, SNMP
    • Tunneling - GTP
  • Searchable fields
    • Protocol
    • Field name
    • Field value

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

WLAN

  • Metadata from WLAN collection
  • Searchable fields
    • Channel
    • SSID
    • BSSID
    • MAC addresses
    • Username
    • Private IP
  • Limited deplyment - populated in SOTF by front end

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Miscellaneous

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Call Logs

  • DNR metadata from JUGGERNAUT, CERF, FASCIA, DURT, etc.
  • Searchable fields
    • Phone numbers
    • Signaling type
    • OPC, DPC, CIC, IMSI
  • Limited deployment

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Network Logs

  • Network metadata from MOONSHINE logs
  • Searchable fields
    • Net type
    • ESSID, BSSID
    • Channel
    • Carrier
    • Latitude and longitude
  • Limited deployment

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

CNE

  • CNE data from TAO
  • Searchable fields
    • Project name
    • Collection technique
    • Filename and extension
  • Limited deployment (xks-cne)

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Registry

  • Windows registry data from TAO (CNE)
  • Searchable fields
    • Collection technique
    • Hive
    • Key, subkey, value
  • Limited deployment (xks-cne)

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Custom Search Forms

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

  • Simple way to search for usernames, IP addresses, and machine ID cookies
  • Just enter your search term and select what type of thing it is, and the form sends it to User Activity or HTTP Activity as appropriate

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

  • Problem: XKS may have info about "badguy@yahoo" in Email Addresses, User Activity, Logins & Passwords, etc.
  • Solution: submit multiple searches from a single form
  • Enter the username and select which databases to search, and the form translates that into the proper queries
  • Similar MultiSearches for IP addresses and MAC addresses
  • Optional: merge results into one table

SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL