846 lines
21 KiB
Markdown
846 lines
21 KiB
Markdown
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
# XKEYSCORE Appids & Fingerprints
|
||
|
||
xkeyscore@nsa
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Basic Syntax
|
||
|
||
* Syntax is similar to C:
|
||
|
||
`function('name', level, <optional info> ) = 'search terms and patterns';`
|
||
|
||
* Two main functions -- appid and fingerprint:
|
||
|
||
`appid('chat/icq', 8.5, wireshark='icq', chatproc='ICQ') = /[^o]icq/c and $icq;`
|
||
|
||
`fingerprint('fingerprint/phone/nokia/generic') = 'user-agent: nokia' or 'profile: http://nds.nokia.com/uaprof/n';`
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Naming Conventions
|
||
|
||
Appids are named using a pseudo directory
|
||
convention:
|
||
|
||
`/application_type/sub_type/name`
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Levels
|
||
|
||
Levels are 1.0 – 9.9 with lower numbers meaning higher priority.
|
||
|
||
This allows multiple signatures to match a piece of traffic, but only the most specific appid will be applied. For example:
|
||
|
||
`appid('chat', 9.9) = ...`
|
||
`appid('chat/yahoo', 9.8) = ...`
|
||
`appid('chat/yahoo/incoming', 9.7 = ...`
|
||
|
||
If a session matches all three signatures, the appid will be 'chat/yahoo/incoming' since that has the best priority.
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Application Type
|
||
|
||
Third parameter is the _application type_; if missing, we use the appid name up to the first slash as the type
|
||
|
||
`appid('http/response', 9.2, 'web') = ...`
|
||
|
||
`appid('chat/yahoo/incoming', 9.1) = ...`
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
--
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Basic Search Patterns
|
||
|
||
XKEYSCORE supports Boolean operations and regular expressions
|
||
|
||
Raw text must be encapsulated between single quotes
|
||
|
||
* 'search term'
|
||
|
||
Terms can be combined with Boolean logic
|
||
|
||
* 'search term' and 'another term' and not 'defeat term'
|
||
* 'search term' or 'another term'
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Example
|
||
|
||
```
|
||
appid('voip/sip/IMS', 6.0, wireshark='sip') =
|
||
('via: sip' or 'v: sip') and 'cseq:' and (
|
||
'p-access-network-info:' or
|
||
'p-called-party-id:' or
|
||
'p-charging-vector:' or
|
||
'p-charging-vector-addresses:' or
|
||
'p-media-authorization:' or
|
||
'security-verify:' or
|
||
'proxy-authorization:' and 'scscf' or
|
||
'path:' and 'pcscf' or
|
||
'path:' and 'scscf'
|
||
);
|
||
```
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Binary and Regex Patterns
|
||
|
||
Binary patterns can be represented by putting a \x in front of each value:
|
||
|
||
`'\xff\xff\x00\x02'`
|
||
|
||
Or use the hex function:
|
||
|
||
`hex('ff:ff:00:02')`
|
||
|
||
Use slashes to enclose regular expressions:
|
||
|
||
`/[^a-zA-Z0-9]BTE/`
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Case sensitivity
|
||
|
||
Keywords and regular expressions are NOT case sensitive by default.
|
||
|
||
Append a 'c' to request case-sensitive evaluation:
|
||
|
||
`'keyword'c`
|
||
`/regex/c`
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Keyword Length
|
||
|
||
Keywords must be at least 3 characters or they will never hit. This minimum is increased to 4 at some sites for performance reasons.
|
||
|
||
Regular expressions must include a fixed "anchor" meeting the minimum keyword length.
|
||
|
||
Bad: `/[A-Z]{3}-[0-9]{3,5}/`
|
||
OK: `/ABC-[0-9]{3,5}/`
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Appid vs Fingerprint
|
||
|
||
Each session gets one appid -- lowest level wins. It gets databased in the 'application' field.
|
||
|
||
All matching fingerprints are stored in the 'fingerprint' field. Level is ignored and can be omitted from fingerprint definitions.
|
||
|
||
```
|
||
Application Type*:
|
||
Application Info*:
|
||
Application:
|
||
AppID (+Fingerprints)* [fulltext]: [Field Builder]
|
||
|
||
Winning appid
|
||
Winning appid + all fingerprints
|
||
```
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Example
|
||
|
||
```
|
||
appid('mail/yahoo', 9.0) = 'Host: mail.yahoo';
|
||
appid('mail/yahoo/login, 8.0) = 'Host: mail.yahoo' and '/login';
|
||
|
||
fingerprint('mail/arabic') = 'mail' and /language[:=] ?ar/;
|
||
fingerprint('mail/yahoo/ymbm') = 'Host: mail.yahoo' and 'YMBM='c;
|
||
|
||
GET /login.html HTTP/1.1
|
||
Referer: http://us.f359.mail.yahoo.com/ym/ShowLetter
|
||
Accept-Language: ar
|
||
Accept-Encoding: gzip, deflate
|
||
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
|
||
Host: mail.yahoo.com
|
||
Connection: Keep-Alive
|
||
Cookie: B=fn50ehd2612o2&b=3&s=rp; YMBM=d=&v=1;
|
||
```
|
||
|
||
Application: `mail/yahoo/login`
|
||
Fingerprint: `mail/yahoo/login mail/arabic mail/yahoo/ymbm`
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Real world example
|
||
|
||
```
|
||
لوحة التحكم
|
||
PM 08:04, 2008-18-02
|
||
Mar 2007
|
||
1,199
|
||
tawab
|
||
الإعدادات والخيارات
|
||
تعديل التوقيع
|
||
البريد الالكتروني / كلمة المرور
|
||
تعديل الملف الشخصي
|
||
تعديل الخيارات
|
||
الرسائل الخاصة
|
||
قائمة الرسائل
|
||
ارسال رسالة جديدة
|
||
رسالة خاصة: تفضل
|
||
غير متواجد حالياً
|
||
نائب المشرف
|
||
افتراضي تفضل
|
||
تتبع الرسائل
|
||
تعديل المجلدات
|
||
مواضيع مشترك بها
|
||
قائمة الاشتراكات
|
||
تعديل المجلدات
|
||
الرقابة
|
||
المواضيع المحذوفة
|
||
المشاركات المحذوفة
|
||
المواضيع المراقبة
|
||
المشاركات المراقبة
|
||
المتنوعات
|
||
البذكر بالحدث
|
||
قوائم الأصدقاء / التجاهل
|
||
الملفات المرفقة
|
||
تاريخ التسجيل : 2007 Mar
|
||
المشاركات : 1,199
|
||
|
||
### Begin ASRAR EI Mojahedeen v2.0 Encrypted Message ###
|
||
r/RgTzT/ATRhN2E1Zjg1OWQyNWRjMmE2ZTdINzZmZDhI0DUxZWZhMDQ1MjYwMjVIZGUO
|
||
ZGYwMjdkMmJmNTA4ZDY2Yjk0MGU2NGNYjg6MzNjZTc6MThjY2 Y1ZmY6MTgzZDIkYjhjMTE
|
||
xOGYzYjc1ZDdiMDAxNTQzZmVINDVIY2YyMGJjYjU20DkyYjdmYjFjYjAzMWM6ZDQ20WFIMzg
|
||
4NThhM211Mjc5ODkzZGNh0GRmNWJmNjVIZjQ0MjMxNDM4MDIyOTg1MmRjMGJINGNKYTN
|
||
KYTQ4MzMxZjRIN2FiNjI3MjE1NGI3MTA3ZDQ4NWRmYzMyOTUzZjZIMjg3NjQ10GQ4MTA3N
|
||
TU2N2ZkN2ZjYzUzYzYyMjFIODAwN2VkM2U5MTZINDY2MmM2ZTV IYjQ2YzI0OGQ20DUxNW
|
||
VkMjI2MWVINDAyOGI0MThkMTdhNTY1YzIxMDgyOGZIM2IwZWZj MDgwM2U4MzNINDg1OD
|
||
UxZTc4ODc1MTY2M210NjU5ZjBhZjVhNjk00TlhNGExOThmYWVI NmFIZjlyNmMwZDA3MDMO
|
||
NjJkZDhhMml4ZmRhYjc3NmZINDFk0DkyYjBhYjY3MDQ10GVIMj dhYmUwZTlyNGIxYmQyZDIz
|
||
ZjliM2E5ZGQ5NmNhZDQx0TM4NT|0Mjc3MzBIOWEwZWE1Njk3Yj gxY2VINTQ10WULnoi/VD
|
||
ULIJTEuDJqne0GMRHesi/BPTnZj02yqbmKbFkIPjwMhe7FUhFAOw74S+i+PokOREo5XhdP+ y9
|
||
Gul3juYTvrlE0xGx20sSfNS5kfRXXH1DaTnb7Oyufe9r6mMIQ6
|
||
1
|
||
e6E0SRUIdU6YVupz0hhgd4Dof
|
||
SBbFR30vgOS+pUxDYgmE0r/RA+fYi47tuHQMh+dynZqQspNdmRUmkjEpFqF03sPHS/10injqo
|
||
e1GsfB+xn52XE2q/WdnU+4XjWnl/isVNAjv2nsL+s2TG1lHbgocmpQoxy0B0SXPcRv/+2JekV37
|
||
XyONZk9YH+DV3aWYPXt+ym+wG0XNTqPHIU1JWAZqI2NK/cSXt9DMtCtcb8czRj6G9/XvJ9
|
||
Eny7t06xPd9BGio9M+3QuUkZHLEmJiAvgvB6R/X/3whBqk6zMHQLfo+VJcX9umW5mRtgCjzS
|
||
PW6lzzFCGt/B4SK4PxT52ZC0B2kWD8VMyNffrlsTG4XUesgx47Nd5xML8w5pj/fZwKNK+EfKIP
|
||
==Z1 ow29A9N3uLIXBX62LhOyj/1iqfJ2FNR7AIONSEjwKoggVmkxDiuGaQi+TurpxBgat1g
|
||
### End ASRAR EI Mojahedeen v2.0 Encrypted Message ###
|
||
|
||
Displaying 1 items Hidden fields
|
||
```
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Distribution
|
||
|
||
* Appids and fingerprints are distributed across the XKEYSCORE network every hour
|
||
* Changes will take effect within 2 hours of check-in
|
||
* Current definitions are available on the website:
|
||
|
||
http://xkeyscore.r1.r.nsa/documents/appid.html
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Intermediate Syntax
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Append Option
|
||
|
||
You can append derived metadata fields onto the end of an appid:
|
||
|
||
```
|
||
appid('p2p/kazaa', 7.7, append='mime_type') =
|
||
Ipos('x-kazaa') and not $http;
|
||
```
|
||
|
||
This will result in an appid like 'p2p/kazaa/image/jpeg'.
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Built in functions
|
||
|
||
| Function | Description |
|
||
| :--------------------- | :----------- |
|
||
| `ip(expr)` | Matches against an IP Address looks in to address and from address in the session headere<br>`• ip('10.10.10.1');` |
|
||
| `toport(expr)` | Matches against the Destination/To port. Note this must be a numeric representation of a port.<br> `• toport(1920);` |
|
||
| `fromport( expr)` | Matches against the Source/From port. Note this must be a numeric representation of a port.<br> `• fromport(80);` |
|
||
| `port(expr)` | Matches against the either port. Note this must be a numeric representation of a port.<br> `• port(6667);` |
|
||
| `next_protocol(expr)` | Matches against the integer version of the next protocol.<br> `• next_protocol(250);` |
|
||
| `protocol('text')` | Will only work for IP next protocol names as defined in the IANA next protocol numbers document<br> `• protocol('tcp');` |
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Built in functions
|
||
|
||
| Function | Description |
|
||
| :------------------ | :------------------------------------------------- |
|
||
| `email_address(sel)` | permutes just like strong_selector (just like DECODEORDAIN |
|
||
| `mac_address(addr)` | Tasks a mac address |
|
||
| `smac(addr)` | |
|
||
| `dmac(addr)` | |
|
||
| `ip(addr)` | tasks this IP address (either to or from) |
|
||
| `from_ip(addr)` | tasks this IP address only when it is the originator |
|
||
| `to_ip(addr)` | tasks this IP address only when it is the destination |
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## More built in functions
|
||
|
||
| Function | Description |
|
||
| :------------------ | :----------- |
|
||
| `first(expr)` | Matches against a pattern at the beginning of the session |
|
||
| `Ipos(expr)` | Matches against a pattern at the beginning of each line (\n) |
|
||
| `pos( expr)` | expression occurs at offset X in the session<br>`• pos('Hello') == 5,`<br>`• pos(/Good.*Grief/) <= 10` |
|
||
| `between(expr)` | `• between('Hello', 'World', 10, 100)`<br>Separation between 'Hello' and 'World' is greater than or equal to 10 bytes and less than or equal to 100 bytes<br>This is the same as using the following regular expression:<br>`• /Hello.{10,100}World/` |
|
||
| `'term'c` | Does a case sensitive match of the term |
|
||
| `'term'u` | Treats the term as UTF-16 |
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Example
|
||
|
||
`appid('voip/skinny(port2000)', 9.9, wireshark='skinny') = port(2000);`
|
||
|
||
`appid('voip/skinny/keep-alive', 3.0, wireshark='skinny') = toport(2000) and first('\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00');`
|
||
|
||
`appid('voip/skinny/keep-alive-ack', 3.0, wireshark='skinny') = fromport(2000) and first('\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00');`
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Example
|
||
|
||
```
|
||
appid('mail/smtp/to_server', 8.5, direction=$from_server,
|
||
wireshark='smtp') =
|
||
toport(25) and
|
||
( first('helo') or
|
||
first('ehlo') or
|
||
first('data') or
|
||
(Ipos('To: 'c) and Ipos('From:'c)) or
|
||
Ipos('QUIT'c) or
|
||
Ipos('mail from:') or
|
||
Ipos('rcpt to:') );
|
||
```
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## CHAINWORDs
|
||
|
||
You can assign a pattern to a variable (CHAINWORD) and reuse the variable in many patterns.
|
||
|
||
`$sip = 'via: sip' and 'cseq:' and 'SIP/2'c;`
|
||
|
||
Now we can use this variable in future definitions:
|
||
|
||
`appid('voip/sip', 7.2) = $sip;`
|
||
|
||
`appid('voip/sip/invite', 6.9) = $sip and 'INVITE';`
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Predefined Chainwords
|
||
|
||
There are a number of chainwords predefined for convenience:
|
||
|
||
* `$tcp`
|
||
* `$udp`
|
||
* `$icmp`
|
||
* `$sctp`
|
||
* `$rpc`
|
||
* `$arp`
|
||
* `$ssl`
|
||
* `$http_cmd`
|
||
* `$http`
|
||
* `$http_get`
|
||
* `$http_put`
|
||
* `$http_post`
|
||
* `$http_delete`
|
||
* `$http_trace`
|
||
* `$http_head`
|
||
* `$http_options`
|
||
* `$http_partial`
|
||
* `$vbulletin`
|
||
* `$mime_type`
|
||
* `$user_agent`
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Example
|
||
|
||
`$icq = 'ICQ'c and $http and not (port(80) or $html_body or $http_cmd);`
|
||
|
||
`appid('chat/icq', 8.5, wireshark='icq', chatproc='ICQ') = /[^o]icq/c and $icq;`
|
||
|
||
`appid('chat/icq', 9.0, wireshark='icq', chatproc='ICQ') = first('icq') and not port(25);`
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Context sensitivity
|
||
|
||
Expressions are evaluated only with a certain context instead of across the session as a whole.
|
||
|
||
`html_title('Yahoo! Mail' or 'Yahoo! Address Book')`
|
||
`... only hits if those keywords are seen within the title of a web page`
|
||
|
||
`http_host('maps.google.com')`
|
||
`... only hits within the "Host:" HTTP header`
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Context sensitivity
|
||
|
||
Why use context-sensitive scanning?
|
||
|
||
* More intuitive - you can say what you mean
|
||
* More accurate - if 'maps.google.com' is mentioned in a blog post, you don't want to try processing it as a Google Maps session
|
||
* Better performance for XKEYSCORE
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Context sensitivity
|
||
|
||
Sample contexts:
|
||
|
||
`html_title`
|
||
`url`
|
||
`http_host`
|
||
`http_referer`
|
||
`http_cookie`
|
||
`http_server`
|
||
`user_agent`
|
||
`web_search`
|
||
`to_cc`
|
||
`from_cc`
|
||
`filename`
|
||
`file_ext`
|
||
`doc_title`
|
||
`doc_subject`
|
||
`doc_author`
|
||
`doc_org`
|
||
`doc_hash`
|
||
`doc_body`
|
||
`email_body`
|
||
`chat_body`
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Example
|
||
|
||
```
|
||
appid('finance/currency_conversion/generic', 8.0) =
|
||
html_title('currency' and ('exchange' or 'conver')) or
|
||
http_server('currency' or 'x-rates.com');`
|
||
|
||
appid('finance/currency_conversion/xe', 8.0) =
|
||
http_host('xe.com') or
|
||
html_title(/^XE -/c or 'XE.com'c);
|
||
```
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Appid utility
|
||
|
||
appid options:
|
||
`--help` this help message
|
||
`--list-all` list all the application/fingerprint names and levels
|
||
`--list-appids` list all the application names (no fingerprints)
|
||
`--list-fingerprints` list all the application names (no appids)
|
||
`--list-types` list all the application types
|
||
`--list-levels` list all the application levels
|
||
`--unit-test` perform unit tests with data in the heirachy 'datadir', with files matching 'filespec'
|
||
`--quiet` don't print any load messages
|
||
`--appid_fname arg` location of appid.cfg
|
||
`--input-file arg` input file to test
|
||
`--datadir arg` The test data directory. Defaults to $(XSCORE_TEST_DATA_DIR)/appids
|
||
`--filespec arg (=.*\.u124)` A regular expression to match against files to check
|
||
`--noexit arg (=0)` do not stop on the first error
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Appid Validation
|
||
|
||
```
|
||
appid sample.u124
|
||
Loading appids
|
||
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/appid_definitions.cfg
|
||
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/anonymizer.appid
|
||
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/bulletin_board.appid
|
||
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/tao_vpn.appid
|
||
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/tdmoip.appid
|
||
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/terminal.appid
|
||
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/voip.appid
|
||
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/appid_definitions.cfg
|
||
Finished loading appids
|
||
Filename: sample.u124
|
||
|
||
Appid: encryption/https
|
||
|
||
Total Size: 19.36Kbits
|
||
Total Time: 0.01secs
|
||
Rate: 1.936Mbits/s
|
||
Overall performance:
|
||
Total Time: 0.01secs
|
||
Total Bits: 0.01936Mbits
|
||
Overall Rate: 1.936Mbits/s
|
||
```
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Advanced Syntax
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Code-based appids
|
||
|
||
Keywords and regular expressions don't work for everything
|
||
|
||
* Looking down columns in packet data
|
||
* Checksums
|
||
* Decoding (urlencoding, base64, gzip, etc.)
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Code-based appids
|
||
|
||
Basic idea:
|
||
|
||
1. Preliminary "trigger" using standard keywords and regular expressions
|
||
|
||
2. Secondary test using a snippet of C++ code
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Code-based appids
|
||
|
||
Example -- verifying a length field:
|
||
|
||
```
|
||
appid('netmanagement/ospf', 2, wireshark='ospf') =
|
||
protocol('ospf')
|
||
: c++ {{
|
||
|
||
if (size() < 4)
|
||
return false;
|
||
|
||
const uint8_t *data = begin();
|
||
return (data[3]==size());
|
||
}};
|
||
```
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Code-based appids
|
||
|
||
Example -- packet data:
|
||
|
||
```
|
||
fingerprint('vpn/xxp_example') =
|
||
'Next Protocol 250'
|
||
: c++ {{
|
||
packet_t pkt;
|
||
int count = 0;
|
||
while ((pkt = get_packet()) && count < 20) {
|
||
++count;
|
||
if (pkt.size < 16)
|
||
return false;
|
||
|
||
if (pkt.data[4] != 0xCC ||
|
||
pkt.data[5] != 0x45 ||
|
||
pkt.data[15] != 0x72)
|
||
return false;
|
||
}
|
||
return (count > 0);
|
||
}};
|
||
```
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Context-sensitive code
|
||
|
||
Example -- code-based check on certain extracted files:
|
||
|
||
```
|
||
fingerprint('crazy/office') =
|
||
extracted_file('doc' or 'xls' or 'ppt' : c++ {{
|
||
return xks::filename().find("crazy") != std::string::npos;
|
||
}});
|
||
```
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Extractors
|
||
|
||
Simplified regex-based metadata extraction
|
||
|
||
```
|
||
fingerprint('maps/google/example') =
|
||
http_host('.google.') and url('/maps?')
|
||
: c++
|
||
extractors : {{
|
||
ka = /Keep-Alive: (\d+)/;
|
||
accept[] = /Accept-([^:]+): ([\w-]+)/;
|
||
}}
|
||
main : {{
|
||
if(ka && ka[0] == "300") {
|
||
for(size_t i = 0; i < accept.size(); ++i)
|
||
if(accept[i][0] == "Encoding" && accept[i][1] == "gzip")
|
||
return true;
|
||
}
|
||
return false;
|
||
}};
|
||
```
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Flex
|
||
|
||
Support for flex-based pattern matching
|
||
|
||
```
|
||
fingerprint('maps/google/firefox') =
|
||
http_host('.google.') and url('/maps?')
|
||
: c++
|
||
flex : {{
|
||
USER_AGENT_CHAR [^\r\n]
|
||
%%
|
||
"User-Agent: "{USER_AGENT_CHAR}+ {
|
||
std::string agent(yytext);
|
||
if(agent.find("Firefox") != std::string::npos)
|
||
return true;
|
||
}
|
||
}};
|
||
```
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Microplugins
|
||
|
||
The next step: giving code-based appids (limited) access to the XKS core
|
||
|
||
* Accessing top-level session metadata
|
||
* Throwing common events
|
||
* Contributing metadata for databasing
|
||
|
||
The goal: higher level of agility with lower learning curve
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Microplugins
|
||
|
||
Example: accessing session metadata
|
||
|
||
```
|
||
fingerprint('maps/google/server1') =
|
||
http_host('.google.') and url('/maps?')
|
||
: c++
|
||
main : {{
|
||
return (SESSION["to_ip"] == "123.45.67.1");
|
||
}};
|
||
```
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Microplugins
|
||
|
||
Example: throwing a document_metadata event
|
||
|
||
```
|
||
fingerprint('maps/google/contrived') =
|
||
http_host('.google.') and url('/maps?')
|
||
: c++
|
||
main : {{
|
||
xks::doc_meta_t dm;
|
||
dm.filename = "google.txt";
|
||
dm.author = "Google, Inc.";
|
||
xks::document_metadata(dm);
|
||
}};
|
||
```
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
---
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||
|
||
## Microplugins
|
||
|
||
Example: contributing metadata to HTTP Activity
|
||
|
||
```
|
||
fingerprint('maps/google/search') =
|
||
http_host('.google.') and url('/maps?')
|
||
: c++
|
||
extractors : {{
|
||
q = /[&?]q=([^&]+)/;
|
||
}}
|
||
main : {{
|
||
if(q) {
|
||
DB["http_parser"]["search_terms"] = xks::urldecode(q[0]);
|
||
DB.apply();
|
||
return true;
|
||
}
|
||
}};
|
||
```
|
||
|
||
TOP SECRET//COMINT//REL TO USA, FVEY
|