first commit
This commit is contained in:
commit
0d90363116
6510 changed files with 679146 additions and 0 deletions
67
data/xks-application-ids/metadata.json
Normal file
67
data/xks-application-ids/metadata.json
Normal file
|
|
@ -0,0 +1,67 @@
|
|||
{
|
||||
"title": "XKEYSCORE Appids & Fingerprints",
|
||||
"type": "REPORTED",
|
||||
"description": "A TOP SECRET XKeyscore developer training presentation describing the complete AppID and fingerprint programming language — the system NSA analysts and developers use to write pattern-matching rules that automatically classify every piece of intercepted internet traffic by application type, protocol, encryption method and target behaviour — including syntax for keyword matching, regular expressions, binary patterns, context-sensitive scanning and embedded C++ code execution against live intercepted sessions.",
|
||||
"long_description": "This is a TOP SECRET//COMINT//REL TO USA, FVEY XKeyscore technical training presentation from the xkeyscore@nsa development team. Unlike the analyst-facing XKeyscore documents that teach how to query stored data, this presentation teaches developers how to write the pattern-matching rules that classify intercepted internet traffic in real time as it flows through XKeyscore's processing pipeline. Every intercepted session receives one AppID (the most specific matching application identifier) and all matching fingerprints, which are then stored in the database and used for querying, forwarding and alerting.\n\nThe AppID/fingerprint system uses a C-like syntax with two primary functions. AppIDs classify traffic by application type using a hierarchical naming convention (e.g., chat/yahoo/incoming, mail/smtp/to_server, voip/sip/invite) with priority levels from 1.0 (highest) to 9.9 (lowest) determining which classification wins when multiple signatures match. Fingerprints tag all matching patterns without priority competition and are stored alongside the winning AppID. The basic pattern matching supports Boolean logic (and, or, not), raw text keywords with minimum three-character length, regular expressions, binary/hex patterns, case sensitivity flags and UTF-16 matching. Built-in functions provide matching against IP addresses, ports, protocols, MAC addresses, email addresses (with DECODEORDAIN-style permutation) and positional constraints. Chainwords allow reusable pattern variables — predefined chainwords include $tcp, $udp, $http, $ssl, $vbulletin and $user_agent among others.\n\nContext-sensitive scanning is a key capability, allowing patterns to be evaluated only within specific extracted fields rather than across the entire session. Sample contexts include html_title, url, http_host, http_referer, http_cookie, user_agent, web_search, email_body, chat_body, doc_author, doc_org and filename — meaning a fingerprint can target only Google Maps requests appearing in the HTTP Host header rather than matching any mention of maps.google.com anywhere in a session. This improves both accuracy and performance.\n\nThe advanced syntax sections describe three progressively more powerful capabilities. Code-based AppIDs combine a keyword/regex trigger with embedded C++ code for secondary validation — useful for verifying packet structure, checksums, length fields and encoded content that cannot be matched with pattern syntax alone. The examples show OSPF protocol verification, VPN packet inspection examining specific byte positions, and Office document filename matching. Extractors provide simplified regex-based metadata extraction with named capture groups. Flex support enables lex-style tokenised pattern matching against session content. Microplugins represent the most advanced capability — giving code-based AppIDs limited access to XKeyscore's core, including reading session metadata (IP addresses, ports), throwing events (document_metadata events that feed into the document metadata search), and contributing derived fields to database tables like HTTP Activity (for example, extracting and URL-decoding Google Maps search terms for storage in the search_terms field).\n\nThe real-world example slide shows a captured Arabic-language vBulletin forum private message containing a complete \"ASRAR El Mojahedeen v2.0 Encrypted Message\" — the Mujahideen Secrets 2.0 encryption tool — demonstrating the operational context in which these fingerprints operate. The distribution slide confirms that AppID and fingerprint definitions are pushed across the entire global XKeyscore network every hour, with changes taking effect within two hours of check-in. The validation tooling shows AppID definitions loaded from configuration files including anonymizer.appid, bulletin_board.appid, tao_vpn.appid, terminal.appid and voip.appid, with unit testing against sample traffic captures.",
|
||||
"tags": [
|
||||
"AppID",
|
||||
"fingerprints",
|
||||
"pattern matching",
|
||||
"C++",
|
||||
"microplugins",
|
||||
"chainword",
|
||||
"context-sensitive scanning",
|
||||
"regular expression",
|
||||
"binary pattern",
|
||||
"session metadata",
|
||||
"HTTP Activity",
|
||||
"document metadata",
|
||||
"SMTP",
|
||||
"ICQ",
|
||||
"OSPF",
|
||||
"Skinny",
|
||||
"VPNs",
|
||||
"Kazaa",
|
||||
"Google Maps",
|
||||
"vBulletin",
|
||||
"Mujahideen Secrets"
|
||||
],
|
||||
"document": {
|
||||
"creator": "NSA",
|
||||
"date_created": "",
|
||||
"classification": [
|
||||
"TOP SECRET",
|
||||
"COMINT"
|
||||
],
|
||||
"dissemination": [
|
||||
"REL TO USA, FVEY"
|
||||
],
|
||||
"banner": "TOP SECRET//COMINT//REL TO USA, FVEY",
|
||||
"genre": "slides",
|
||||
"programs": [
|
||||
"XKEYSCORE",
|
||||
"DECODEORDAIN"
|
||||
],
|
||||
"countries": [],
|
||||
"people": []
|
||||
},
|
||||
"publication": [
|
||||
{
|
||||
"publisher": "The Intercept",
|
||||
"date_published": "2015-07-01",
|
||||
"title": "XKEYSCORE: NSA’s Google for the World’s Private Communications",
|
||||
"link": "https://theintercept.com/2015/07/01/nsas-google-worlds-private-communications/",
|
||||
"reporters": [
|
||||
"Marquis-Boire, Morgan",
|
||||
"Greenwald, Glenn",
|
||||
"Lee, Micah"
|
||||
],
|
||||
"contributors": [],
|
||||
"doc_source": "https://theintercept.com/document/2015/07/01/xks-application-ids/",
|
||||
"doc_url": "https://s3.documentcloud.org/documents/2116302/xks-application-ids.pdf"
|
||||
}
|
||||
],
|
||||
"aka_id": [],
|
||||
"pdf_sha256": "2e15431de13a51d2c49002a9fca5ab411dd646f36e66549847267f6f76e7e53f",
|
||||
"pages": 42
|
||||
}
|
||||
846
data/xks-application-ids/xks-application-ids.md
Normal file
846
data/xks-application-ids/xks-application-ids.md
Normal file
|
|
@ -0,0 +1,846 @@
|
|||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
# XKEYSCORE Appids & Fingerprints
|
||||
|
||||
xkeyscore@nsa
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Basic Syntax
|
||||
|
||||
* Syntax is similar to C:
|
||||
|
||||
`function('name', level, <optional info> ) = 'search terms and patterns';`
|
||||
|
||||
* Two main functions -- appid and fingerprint:
|
||||
|
||||
`appid('chat/icq', 8.5, wireshark='icq', chatproc='ICQ') = /[^o]icq/c and $icq;`
|
||||
|
||||
`fingerprint('fingerprint/phone/nokia/generic') = 'user-agent: nokia' or 'profile: http://nds.nokia.com/uaprof/n';`
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Naming Conventions
|
||||
|
||||
Appids are named using a pseudo directory
|
||||
convention:
|
||||
|
||||
`/application_type/sub_type/name`
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Levels
|
||||
|
||||
Levels are 1.0 – 9.9 with lower numbers meaning higher priority.
|
||||
|
||||
This allows multiple signatures to match a piece of traffic, but only the most specific appid will be applied. For example:
|
||||
|
||||
`appid('chat', 9.9) = ...`
|
||||
`appid('chat/yahoo', 9.8) = ...`
|
||||
`appid('chat/yahoo/incoming', 9.7 = ...`
|
||||
|
||||
If a session matches all three signatures, the appid will be 'chat/yahoo/incoming' since that has the best priority.
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Application Type
|
||||
|
||||
Third parameter is the _application type_; if missing, we use the appid name up to the first slash as the type
|
||||
|
||||
`appid('http/response', 9.2, 'web') = ...`
|
||||
|
||||
`appid('chat/yahoo/incoming', 9.1) = ...`
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
--
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Basic Search Patterns
|
||||
|
||||
XKEYSCORE supports Boolean operations and regular expressions
|
||||
|
||||
Raw text must be encapsulated between single quotes
|
||||
|
||||
* 'search term'
|
||||
|
||||
Terms can be combined with Boolean logic
|
||||
|
||||
* 'search term' and 'another term' and not 'defeat term'
|
||||
* 'search term' or 'another term'
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Example
|
||||
|
||||
```
|
||||
appid('voip/sip/IMS', 6.0, wireshark='sip') =
|
||||
('via: sip' or 'v: sip') and 'cseq:' and (
|
||||
'p-access-network-info:' or
|
||||
'p-called-party-id:' or
|
||||
'p-charging-vector:' or
|
||||
'p-charging-vector-addresses:' or
|
||||
'p-media-authorization:' or
|
||||
'security-verify:' or
|
||||
'proxy-authorization:' and 'scscf' or
|
||||
'path:' and 'pcscf' or
|
||||
'path:' and 'scscf'
|
||||
);
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Binary and Regex Patterns
|
||||
|
||||
Binary patterns can be represented by putting a \x in front of each value:
|
||||
|
||||
`'\xff\xff\x00\x02'`
|
||||
|
||||
Or use the hex function:
|
||||
|
||||
`hex('ff:ff:00:02')`
|
||||
|
||||
Use slashes to enclose regular expressions:
|
||||
|
||||
`/[^a-zA-Z0-9]BTE/`
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Case sensitivity
|
||||
|
||||
Keywords and regular expressions are NOT case sensitive by default.
|
||||
|
||||
Append a 'c' to request case-sensitive evaluation:
|
||||
|
||||
`'keyword'c`
|
||||
`/regex/c`
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Keyword Length
|
||||
|
||||
Keywords must be at least 3 characters or they will never hit. This minimum is increased to 4 at some sites for performance reasons.
|
||||
|
||||
Regular expressions must include a fixed "anchor" meeting the minimum keyword length.
|
||||
|
||||
Bad: `/[A-Z]{3}-[0-9]{3,5}/`
|
||||
OK: `/ABC-[0-9]{3,5}/`
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Appid vs Fingerprint
|
||||
|
||||
Each session gets one appid -- lowest level wins. It gets databased in the 'application' field.
|
||||
|
||||
All matching fingerprints are stored in the 'fingerprint' field. Level is ignored and can be omitted from fingerprint definitions.
|
||||
|
||||
```
|
||||
Application Type*:
|
||||
Application Info*:
|
||||
Application:
|
||||
AppID (+Fingerprints)* [fulltext]: [Field Builder]
|
||||
|
||||
Winning appid
|
||||
Winning appid + all fingerprints
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Example
|
||||
|
||||
```
|
||||
appid('mail/yahoo', 9.0) = 'Host: mail.yahoo';
|
||||
appid('mail/yahoo/login, 8.0) = 'Host: mail.yahoo' and '/login';
|
||||
|
||||
fingerprint('mail/arabic') = 'mail' and /language[:=] ?ar/;
|
||||
fingerprint('mail/yahoo/ymbm') = 'Host: mail.yahoo' and 'YMBM='c;
|
||||
|
||||
GET /login.html HTTP/1.1
|
||||
Referer: http://us.f359.mail.yahoo.com/ym/ShowLetter
|
||||
Accept-Language: ar
|
||||
Accept-Encoding: gzip, deflate
|
||||
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
|
||||
Host: mail.yahoo.com
|
||||
Connection: Keep-Alive
|
||||
Cookie: B=fn50ehd2612o2&b=3&s=rp; YMBM=d=&v=1;
|
||||
```
|
||||
|
||||
Application: `mail/yahoo/login`
|
||||
Fingerprint: `mail/yahoo/login mail/arabic mail/yahoo/ymbm`
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Real world example
|
||||
|
||||
```
|
||||
لوحة التحكم
|
||||
PM 08:04, 2008-18-02
|
||||
Mar 2007
|
||||
1,199
|
||||
tawab
|
||||
الإعدادات والخيارات
|
||||
تعديل التوقيع
|
||||
البريد الالكتروني / كلمة المرور
|
||||
تعديل الملف الشخصي
|
||||
تعديل الخيارات
|
||||
الرسائل الخاصة
|
||||
قائمة الرسائل
|
||||
ارسال رسالة جديدة
|
||||
رسالة خاصة: تفضل
|
||||
غير متواجد حالياً
|
||||
نائب المشرف
|
||||
افتراضي تفضل
|
||||
تتبع الرسائل
|
||||
تعديل المجلدات
|
||||
مواضيع مشترك بها
|
||||
قائمة الاشتراكات
|
||||
تعديل المجلدات
|
||||
الرقابة
|
||||
المواضيع المحذوفة
|
||||
المشاركات المحذوفة
|
||||
المواضيع المراقبة
|
||||
المشاركات المراقبة
|
||||
المتنوعات
|
||||
البذكر بالحدث
|
||||
قوائم الأصدقاء / التجاهل
|
||||
الملفات المرفقة
|
||||
تاريخ التسجيل : 2007 Mar
|
||||
المشاركات : 1,199
|
||||
|
||||
### Begin ASRAR EI Mojahedeen v2.0 Encrypted Message ###
|
||||
r/RgTzT/ATRhN2E1Zjg1OWQyNWRjMmE2ZTdINzZmZDhI0DUxZWZhMDQ1MjYwMjVIZGUO
|
||||
ZGYwMjdkMmJmNTA4ZDY2Yjk0MGU2NGNYjg6MzNjZTc6MThjY2 Y1ZmY6MTgzZDIkYjhjMTE
|
||||
xOGYzYjc1ZDdiMDAxNTQzZmVINDVIY2YyMGJjYjU20DkyYjdmYjFjYjAzMWM6ZDQ20WFIMzg
|
||||
4NThhM211Mjc5ODkzZGNh0GRmNWJmNjVIZjQ0MjMxNDM4MDIyOTg1MmRjMGJINGNKYTN
|
||||
KYTQ4MzMxZjRIN2FiNjI3MjE1NGI3MTA3ZDQ4NWRmYzMyOTUzZjZIMjg3NjQ10GQ4MTA3N
|
||||
TU2N2ZkN2ZjYzUzYzYyMjFIODAwN2VkM2U5MTZINDY2MmM2ZTV IYjQ2YzI0OGQ20DUxNW
|
||||
VkMjI2MWVINDAyOGI0MThkMTdhNTY1YzIxMDgyOGZIM2IwZWZj MDgwM2U4MzNINDg1OD
|
||||
UxZTc4ODc1MTY2M210NjU5ZjBhZjVhNjk00TlhNGExOThmYWVI NmFIZjlyNmMwZDA3MDMO
|
||||
NjJkZDhhMml4ZmRhYjc3NmZINDFk0DkyYjBhYjY3MDQ10GVIMj dhYmUwZTlyNGIxYmQyZDIz
|
||||
ZjliM2E5ZGQ5NmNhZDQx0TM4NT|0Mjc3MzBIOWEwZWE1Njk3Yj gxY2VINTQ10WULnoi/VD
|
||||
ULIJTEuDJqne0GMRHesi/BPTnZj02yqbmKbFkIPjwMhe7FUhFAOw74S+i+PokOREo5XhdP+ y9
|
||||
Gul3juYTvrlE0xGx20sSfNS5kfRXXH1DaTnb7Oyufe9r6mMIQ6
|
||||
1
|
||||
e6E0SRUIdU6YVupz0hhgd4Dof
|
||||
SBbFR30vgOS+pUxDYgmE0r/RA+fYi47tuHQMh+dynZqQspNdmRUmkjEpFqF03sPHS/10injqo
|
||||
e1GsfB+xn52XE2q/WdnU+4XjWnl/isVNAjv2nsL+s2TG1lHbgocmpQoxy0B0SXPcRv/+2JekV37
|
||||
XyONZk9YH+DV3aWYPXt+ym+wG0XNTqPHIU1JWAZqI2NK/cSXt9DMtCtcb8czRj6G9/XvJ9
|
||||
Eny7t06xPd9BGio9M+3QuUkZHLEmJiAvgvB6R/X/3whBqk6zMHQLfo+VJcX9umW5mRtgCjzS
|
||||
PW6lzzFCGt/B4SK4PxT52ZC0B2kWD8VMyNffrlsTG4XUesgx47Nd5xML8w5pj/fZwKNK+EfKIP
|
||||
==Z1 ow29A9N3uLIXBX62LhOyj/1iqfJ2FNR7AIONSEjwKoggVmkxDiuGaQi+TurpxBgat1g
|
||||
### End ASRAR EI Mojahedeen v2.0 Encrypted Message ###
|
||||
|
||||
Displaying 1 items Hidden fields
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Distribution
|
||||
|
||||
* Appids and fingerprints are distributed across the XKEYSCORE network every hour
|
||||
* Changes will take effect within 2 hours of check-in
|
||||
* Current definitions are available on the website:
|
||||
|
||||
http://xkeyscore.r1.r.nsa/documents/appid.html
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Intermediate Syntax
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Append Option
|
||||
|
||||
You can append derived metadata fields onto the end of an appid:
|
||||
|
||||
```
|
||||
appid('p2p/kazaa', 7.7, append='mime_type') =
|
||||
Ipos('x-kazaa') and not $http;
|
||||
```
|
||||
|
||||
This will result in an appid like 'p2p/kazaa/image/jpeg'.
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Built in functions
|
||||
|
||||
| Function | Description |
|
||||
| :--------------------- | :----------- |
|
||||
| `ip(expr)` | Matches against an IP Address looks in to address and from address in the session headere<br>`• ip('10.10.10.1');` |
|
||||
| `toport(expr)` | Matches against the Destination/To port. Note this must be a numeric representation of a port.<br> `• toport(1920);` |
|
||||
| `fromport( expr)` | Matches against the Source/From port. Note this must be a numeric representation of a port.<br> `• fromport(80);` |
|
||||
| `port(expr)` | Matches against the either port. Note this must be a numeric representation of a port.<br> `• port(6667);` |
|
||||
| `next_protocol(expr)` | Matches against the integer version of the next protocol.<br> `• next_protocol(250);` |
|
||||
| `protocol('text')` | Will only work for IP next protocol names as defined in the IANA next protocol numbers document<br> `• protocol('tcp');` |
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Built in functions
|
||||
|
||||
| Function | Description |
|
||||
| :------------------ | :------------------------------------------------- |
|
||||
| `email_address(sel)` | permutes just like strong_selector (just like DECODEORDAIN |
|
||||
| `mac_address(addr)` | Tasks a mac address |
|
||||
| `smac(addr)` | |
|
||||
| `dmac(addr)` | |
|
||||
| `ip(addr)` | tasks this IP address (either to or from) |
|
||||
| `from_ip(addr)` | tasks this IP address only when it is the originator |
|
||||
| `to_ip(addr)` | tasks this IP address only when it is the destination |
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## More built in functions
|
||||
|
||||
| Function | Description |
|
||||
| :------------------ | :----------- |
|
||||
| `first(expr)` | Matches against a pattern at the beginning of the session |
|
||||
| `Ipos(expr)` | Matches against a pattern at the beginning of each line (\n) |
|
||||
| `pos( expr)` | expression occurs at offset X in the session<br>`• pos('Hello') == 5,`<br>`• pos(/Good.*Grief/) <= 10` |
|
||||
| `between(expr)` | `• between('Hello', 'World', 10, 100)`<br>Separation between 'Hello' and 'World' is greater than or equal to 10 bytes and less than or equal to 100 bytes<br>This is the same as using the following regular expression:<br>`• /Hello.{10,100}World/` |
|
||||
| `'term'c` | Does a case sensitive match of the term |
|
||||
| `'term'u` | Treats the term as UTF-16 |
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Example
|
||||
|
||||
`appid('voip/skinny(port2000)', 9.9, wireshark='skinny') = port(2000);`
|
||||
|
||||
`appid('voip/skinny/keep-alive', 3.0, wireshark='skinny') = toport(2000) and first('\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00');`
|
||||
|
||||
`appid('voip/skinny/keep-alive-ack', 3.0, wireshark='skinny') = fromport(2000) and first('\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00');`
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Example
|
||||
|
||||
```
|
||||
appid('mail/smtp/to_server', 8.5, direction=$from_server,
|
||||
wireshark='smtp') =
|
||||
toport(25) and
|
||||
( first('helo') or
|
||||
first('ehlo') or
|
||||
first('data') or
|
||||
(Ipos('To: 'c) and Ipos('From:'c)) or
|
||||
Ipos('QUIT'c) or
|
||||
Ipos('mail from:') or
|
||||
Ipos('rcpt to:') );
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## CHAINWORDs
|
||||
|
||||
You can assign a pattern to a variable (CHAINWORD) and reuse the variable in many patterns.
|
||||
|
||||
`$sip = 'via: sip' and 'cseq:' and 'SIP/2'c;`
|
||||
|
||||
Now we can use this variable in future definitions:
|
||||
|
||||
`appid('voip/sip', 7.2) = $sip;`
|
||||
|
||||
`appid('voip/sip/invite', 6.9) = $sip and 'INVITE';`
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Predefined Chainwords
|
||||
|
||||
There are a number of chainwords predefined for convenience:
|
||||
|
||||
* `$tcp`
|
||||
* `$udp`
|
||||
* `$icmp`
|
||||
* `$sctp`
|
||||
* `$rpc`
|
||||
* `$arp`
|
||||
* `$ssl`
|
||||
* `$http_cmd`
|
||||
* `$http`
|
||||
* `$http_get`
|
||||
* `$http_put`
|
||||
* `$http_post`
|
||||
* `$http_delete`
|
||||
* `$http_trace`
|
||||
* `$http_head`
|
||||
* `$http_options`
|
||||
* `$http_partial`
|
||||
* `$vbulletin`
|
||||
* `$mime_type`
|
||||
* `$user_agent`
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Example
|
||||
|
||||
`$icq = 'ICQ'c and $http and not (port(80) or $html_body or $http_cmd);`
|
||||
|
||||
`appid('chat/icq', 8.5, wireshark='icq', chatproc='ICQ') = /[^o]icq/c and $icq;`
|
||||
|
||||
`appid('chat/icq', 9.0, wireshark='icq', chatproc='ICQ') = first('icq') and not port(25);`
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Context sensitivity
|
||||
|
||||
Expressions are evaluated only with a certain context instead of across the session as a whole.
|
||||
|
||||
`html_title('Yahoo! Mail' or 'Yahoo! Address Book')`
|
||||
`... only hits if those keywords are seen within the title of a web page`
|
||||
|
||||
`http_host('maps.google.com')`
|
||||
`... only hits within the "Host:" HTTP header`
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Context sensitivity
|
||||
|
||||
Why use context-sensitive scanning?
|
||||
|
||||
* More intuitive - you can say what you mean
|
||||
* More accurate - if 'maps.google.com' is mentioned in a blog post, you don't want to try processing it as a Google Maps session
|
||||
* Better performance for XKEYSCORE
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Context sensitivity
|
||||
|
||||
Sample contexts:
|
||||
|
||||
`html_title`
|
||||
`url`
|
||||
`http_host`
|
||||
`http_referer`
|
||||
`http_cookie`
|
||||
`http_server`
|
||||
`user_agent`
|
||||
`web_search`
|
||||
`to_cc`
|
||||
`from_cc`
|
||||
`filename`
|
||||
`file_ext`
|
||||
`doc_title`
|
||||
`doc_subject`
|
||||
`doc_author`
|
||||
`doc_org`
|
||||
`doc_hash`
|
||||
`doc_body`
|
||||
`email_body`
|
||||
`chat_body`
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Example
|
||||
|
||||
```
|
||||
appid('finance/currency_conversion/generic', 8.0) =
|
||||
html_title('currency' and ('exchange' or 'conver')) or
|
||||
http_server('currency' or 'x-rates.com');`
|
||||
|
||||
appid('finance/currency_conversion/xe', 8.0) =
|
||||
http_host('xe.com') or
|
||||
html_title(/^XE -/c or 'XE.com'c);
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Appid utility
|
||||
|
||||
appid options:
|
||||
`--help` this help message
|
||||
`--list-all` list all the application/fingerprint names and levels
|
||||
`--list-appids` list all the application names (no fingerprints)
|
||||
`--list-fingerprints` list all the application names (no appids)
|
||||
`--list-types` list all the application types
|
||||
`--list-levels` list all the application levels
|
||||
`--unit-test` perform unit tests with data in the heirachy 'datadir', with files matching 'filespec'
|
||||
`--quiet` don't print any load messages
|
||||
`--appid_fname arg` location of appid.cfg
|
||||
`--input-file arg` input file to test
|
||||
`--datadir arg` The test data directory. Defaults to $(XSCORE_TEST_DATA_DIR)/appids
|
||||
`--filespec arg (=.*\.u124)` A regular expression to match against files to check
|
||||
`--noexit arg (=0)` do not stop on the first error
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Appid Validation
|
||||
|
||||
```
|
||||
appid sample.u124
|
||||
Loading appids
|
||||
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/appid_definitions.cfg
|
||||
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/anonymizer.appid
|
||||
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/bulletin_board.appid
|
||||
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/tao_vpn.appid
|
||||
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/tdmoip.appid
|
||||
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/terminal.appid
|
||||
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/voip.appid
|
||||
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/appid_definitions.cfg
|
||||
Finished loading appids
|
||||
Filename: sample.u124
|
||||
|
||||
Appid: encryption/https
|
||||
|
||||
Total Size: 19.36Kbits
|
||||
Total Time: 0.01secs
|
||||
Rate: 1.936Mbits/s
|
||||
Overall performance:
|
||||
Total Time: 0.01secs
|
||||
Total Bits: 0.01936Mbits
|
||||
Overall Rate: 1.936Mbits/s
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Advanced Syntax
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Code-based appids
|
||||
|
||||
Keywords and regular expressions don't work for everything
|
||||
|
||||
* Looking down columns in packet data
|
||||
* Checksums
|
||||
* Decoding (urlencoding, base64, gzip, etc.)
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Code-based appids
|
||||
|
||||
Basic idea:
|
||||
|
||||
1. Preliminary "trigger" using standard keywords and regular expressions
|
||||
|
||||
2. Secondary test using a snippet of C++ code
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Code-based appids
|
||||
|
||||
Example -- verifying a length field:
|
||||
|
||||
```
|
||||
appid('netmanagement/ospf', 2, wireshark='ospf') =
|
||||
protocol('ospf')
|
||||
: c++ {{
|
||||
|
||||
if (size() < 4)
|
||||
return false;
|
||||
|
||||
const uint8_t *data = begin();
|
||||
return (data[3]==size());
|
||||
}};
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Code-based appids
|
||||
|
||||
Example -- packet data:
|
||||
|
||||
```
|
||||
fingerprint('vpn/xxp_example') =
|
||||
'Next Protocol 250'
|
||||
: c++ {{
|
||||
packet_t pkt;
|
||||
int count = 0;
|
||||
while ((pkt = get_packet()) && count < 20) {
|
||||
++count;
|
||||
if (pkt.size < 16)
|
||||
return false;
|
||||
|
||||
if (pkt.data[4] != 0xCC ||
|
||||
pkt.data[5] != 0x45 ||
|
||||
pkt.data[15] != 0x72)
|
||||
return false;
|
||||
}
|
||||
return (count > 0);
|
||||
}};
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Context-sensitive code
|
||||
|
||||
Example -- code-based check on certain extracted files:
|
||||
|
||||
```
|
||||
fingerprint('crazy/office') =
|
||||
extracted_file('doc' or 'xls' or 'ppt' : c++ {{
|
||||
return xks::filename().find("crazy") != std::string::npos;
|
||||
}});
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Extractors
|
||||
|
||||
Simplified regex-based metadata extraction
|
||||
|
||||
```
|
||||
fingerprint('maps/google/example') =
|
||||
http_host('.google.') and url('/maps?')
|
||||
: c++
|
||||
extractors : {{
|
||||
ka = /Keep-Alive: (\d+)/;
|
||||
accept[] = /Accept-([^:]+): ([\w-]+)/;
|
||||
}}
|
||||
main : {{
|
||||
if(ka && ka[0] == "300") {
|
||||
for(size_t i = 0; i < accept.size(); ++i)
|
||||
if(accept[i][0] == "Encoding" && accept[i][1] == "gzip")
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}};
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Flex
|
||||
|
||||
Support for flex-based pattern matching
|
||||
|
||||
```
|
||||
fingerprint('maps/google/firefox') =
|
||||
http_host('.google.') and url('/maps?')
|
||||
: c++
|
||||
flex : {{
|
||||
USER_AGENT_CHAR [^\r\n]
|
||||
%%
|
||||
"User-Agent: "{USER_AGENT_CHAR}+ {
|
||||
std::string agent(yytext);
|
||||
if(agent.find("Firefox") != std::string::npos)
|
||||
return true;
|
||||
}
|
||||
}};
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Microplugins
|
||||
|
||||
The next step: giving code-based appids (limited) access to the XKS core
|
||||
|
||||
* Accessing top-level session metadata
|
||||
* Throwing common events
|
||||
* Contributing metadata for databasing
|
||||
|
||||
The goal: higher level of agility with lower learning curve
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Microplugins
|
||||
|
||||
Example: accessing session metadata
|
||||
|
||||
```
|
||||
fingerprint('maps/google/server1') =
|
||||
http_host('.google.') and url('/maps?')
|
||||
: c++
|
||||
main : {{
|
||||
return (SESSION["to_ip"] == "123.45.67.1");
|
||||
}};
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Microplugins
|
||||
|
||||
Example: throwing a document_metadata event
|
||||
|
||||
```
|
||||
fingerprint('maps/google/contrived') =
|
||||
http_host('.google.') and url('/maps?')
|
||||
: c++
|
||||
main : {{
|
||||
xks::doc_meta_t dm;
|
||||
dm.filename = "google.txt";
|
||||
dm.author = "Google, Inc.";
|
||||
xks::document_metadata(dm);
|
||||
}};
|
||||
```
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
---
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
|
||||
## Microplugins
|
||||
|
||||
Example: contributing metadata to HTTP Activity
|
||||
|
||||
```
|
||||
fingerprint('maps/google/search') =
|
||||
http_host('.google.') and url('/maps?')
|
||||
: c++
|
||||
extractors : {{
|
||||
q = /[&?]q=([^&]+)/;
|
||||
}}
|
||||
main : {{
|
||||
if(q) {
|
||||
DB["http_parser"]["search_terms"] = xks::urldecode(q[0]);
|
||||
DB.apply();
|
||||
return true;
|
||||
}
|
||||
}};
|
||||
```
|
||||
|
||||
TOP SECRET//COMINT//REL TO USA, FVEY
|
||||
Loading…
Add table
Add a link
Reference in a new issue