327 lines
No EOL
6.1 KiB
Markdown
327 lines
No EOL
6.1 KiB
Markdown
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
# XKEYSCORE, Cipher Detection, and You!
|
|
|
|
[REDACTED]
|
|
|
|
Mathematics Research Group
|
|
21 August 2008
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
UNCLASSIFIED
|
|
|
|
## The Protocol Stack
|
|
|
|
* Application Layer (HTTP, FTP, etc.)
|
|
* Transport Layer (TCP, UDP)
|
|
* Network Layer (IPv4, IPv6)
|
|
* Data Link Layer (PPP)
|
|
* Physical Layer (Copper, Fiber)
|
|
|
|
UNCLASSIFIED
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Data Sources
|
|
|
|
* FORNSAT (downlink)
|
|
* Overhead (uplink)
|
|
* Special Source
|
|
* Tailored Access
|
|
* F6
|
|
* FISA (limited)
|
|
* 3rd party
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Front-end Processing
|
|
|
|
```
|
|
WEALTHYCLUSTER2/TURMOIL
|
|
Demodulate Demultiplex Packetize Sessionize
|
|
Decrypt?
|
|
```
|
|
|
|
SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## What does XEYSCORE do?
|
|
|
|
* Selection of tasked CADENCE/UTT terms.
|
|
* Send hits to PINWALE/PRESSUREWAVE.
|
|
* Tipping to TRAFFICTHIEF.
|
|
* Fingerprinting.
|
|
* SIGINT development using two rolling buffers:
|
|
* Metadata
|
|
* Content (data)
|
|
|
|
SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Retrospective Searching
|
|
|
|
XKEYSCORE
|
|
|
|
* All data are stored, not just hits.
|
|
* Queries are distributed to entire network of sites.
|
|
|
|
Metadata Buffer
|
|
* ~30 days
|
|
* Searchable
|
|
* MySQL database
|
|
|
|
Content Buffer
|
|
* ~7 days
|
|
* Retrievable
|
|
* Archived on disk
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Fingerprinting
|
|
|
|
* Pattern matching against the data.
|
|
* Session is marked, but not sent to PINWALE.
|
|
* Fingerprint stored as metadata.
|
|
* Have to search for it.
|
|
* Rich set of patterns
|
|
* Strings have a minimum of three* anchors (fixed bytes).
|
|
* [Exception: Two bytes at the beginning of a session]
|
|
* Regular expressions allowed (require nonoptional string of three\* bytes within regex)
|
|
* Context-dependent terms.
|
|
* \*XKS reserves the right to increase this to four.
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Examples
|
|
|
|
```fingerprint('encryption/helixstronghold',
|
|
7.0) = 'helix stronghold encrypted file';
|
|
```
|
|
|
|
```fingerprint('encryption/wharfrat', 3.0) =
|
|
'\xd6\x56\x34\xb7\x80\x05\xfe\x8b'c and
|
|
'\xaf\x52\x72\x60\xdd\xfe\x72\xc2'c and
|
|
(port(443) or port(80));
|
|
```
|
|
|
|
```
|
|
fingerprint('encryption/the_algorithm',
|
|
3.0)=
|
|
/-XYZ-.{0,30}mp[eg]/;
|
|
```
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Syntax Features
|
|
|
|
* Case Sensitivity
|
|
|
|
```
|
|
fingerprint('certificate/digital_id') =
|
|
'-BEGIN CERTIFICATE-'c;
|
|
```
|
|
|
|
* Full Boolean logic
|
|
* Grouping with parentheses
|
|
* Operators: `and`, `or`, `not`
|
|
* Variables
|
|
|
|
```
|
|
$udp = protocol('udp');
|
|
fingerprint('vpn/openvpn/x509/wera') =
|
|
$udp and 'openvpn_wera'c;
|
|
```
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Available Functions
|
|
|
|
* port
|
|
* first
|
|
* hex
|
|
|
|
```
|
|
fingerprint('encryption/kryptel') =
|
|
hex('E8E2454300040004635C4EE9A2F9D111A
|
|
489E498F70C0B43404F4BFA50F2D111A4898E6
|
|
30458E285');
|
|
```
|
|
|
|
* pos
|
|
|
|
```
|
|
fingerprint('encryption/cipherpad') =
|
|
pos('CPAD1'c) < 4000;
|
|
```
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
* Distance (similar to pos, but for distance between tokens)
|
|
* Lpos
|
|
|
|
```
|
|
Spop_basic = lpos('+OK 'c) or '\nQUIT';
|
|
```
|
|
|
|
* First
|
|
|
|
```
|
|
appid('mail/smtp/...) = first('ehlo') and ... ;
|
|
```
|
|
|
|
* Last (similar to first)
|
|
* Follows (one token after another)
|
|
* Between (one token between two others)
|
|
* Order
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Other Features
|
|
|
|
* Fingerprint definitions updated hourly throughout the entire enterprise.
|
|
* Workflows
|
|
* Submit through user interface.
|
|
* Standing queries that run like cron jobs.
|
|
* Limited follow-on processing.
|
|
* User interface for fingerprint submission (coming soon).
|
|
* Currently done by XKS personnel.
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Plug-ins
|
|
|
|
* Full power of C++ for when pattern matching does not suffice.
|
|
* Usually limited to certain file types
|
|
* Huge JPEG volume from web surfing
|
|
* Current steg/encryption plugins that fingerprint sessions:
|
|
* PHOSPHORESSENCE library of steg detectors
|
|
* SHELLLOCK steg detection
|
|
* SEDENA indigenous encryption software
|
|
* Drawback: Must wait for site upgrade to deploy.
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Trade-off
|
|
|
|
* Fingerprints easily deployed, but limited to pattern matching.
|
|
* Plug-ins slow to deploy, but allow for complex testing.
|
|
* New compromise:
|
|
* Snippets of C++ code in fingerprint
|
|
* Deployed hourly like fingerprint with most of the flexibility of a full plug-in.
|
|
* Very complicated tests probably still need to be plug-ins.
|
|
* Currently stood up at only a few sites.
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Example
|
|
|
|
```
|
|
fingerprint('encryption/archive/rar') =
|
|
'\x52\x61\x72\x21\x1a\x07\x00'c
|
|
: c++ {{
|
|
const uint8_t *ptr =
|
|
find_first("\x52\x61\x72\x21\x1a\x07\x00");
|
|
if (ptr == NULL)
|
|
return false;
|
|
if (end()-ptr < 64)
|
|
return false;
|
|
if ((ptr[23] &0x04) != 0x04)
|
|
return false;
|
|
if ((ptr[10] &0x80) != 0x80)
|
|
return false;
|
|
return true;
|
|
}};
|
|
```
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Advanced Feature
|
|
|
|
* Follow-on check with anchorless regexes:
|
|
|
|
```
|
|
%dhcp_check = regex {{
|
|
^[\x01\x02][\x01- ]\x06.*c\x82sc
|
|
}};
|
|
```
|
|
|
|
```
|
|
appid('netmanagement/dhcp/client_to_server',
|
|
3.0) =
|
|
from_port(68) and to_port(67)
|
|
: %dhcp_check;
|
|
```
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
---
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY
|
|
|
|
## Releasability Issues
|
|
|
|
* Nearly all XKS personnel have PICARESQUE!
|
|
* Those that don't have PRIVAC.
|
|
* XKS distribution comes in two flavors
|
|
* 1st & 2nd party
|
|
* 3rd party
|
|
* No NOFORN capabilities permitted.
|
|
* Special dispensation from [REDACTED] for some capabilities to SMOKYSINK.
|
|
* Can keep PICARESQUE code running on R1's rednet if absolutely necessary.
|
|
|
|
TOP SECRET//COMINT//REL TO USA, FVEY |