first commit

This commit is contained in:
libroot 2026-11-19 04:39:11 +02:00
commit 0d90363116
Signed by: libroot
GPG key ID: 2AC68C029EBE9ACE
6510 changed files with 679146 additions and 0 deletions

View file

@ -0,0 +1,67 @@
{
"title": "XKEYSCORE Appids & Fingerprints",
"type": "REPORTED",
"description": "A TOP SECRET XKeyscore developer training presentation describing the complete AppID and fingerprint programming language — the system NSA analysts and developers use to write pattern-matching rules that automatically classify every piece of intercepted internet traffic by application type, protocol, encryption method and target behaviour — including syntax for keyword matching, regular expressions, binary patterns, context-sensitive scanning and embedded C++ code execution against live intercepted sessions.",
"long_description": "This is a TOP SECRET//COMINT//REL TO USA, FVEY XKeyscore technical training presentation from the xkeyscore@nsa development team. Unlike the analyst-facing XKeyscore documents that teach how to query stored data, this presentation teaches developers how to write the pattern-matching rules that classify intercepted internet traffic in real time as it flows through XKeyscore's processing pipeline. Every intercepted session receives one AppID (the most specific matching application identifier) and all matching fingerprints, which are then stored in the database and used for querying, forwarding and alerting.\n\nThe AppID/fingerprint system uses a C-like syntax with two primary functions. AppIDs classify traffic by application type using a hierarchical naming convention (e.g., chat/yahoo/incoming, mail/smtp/to_server, voip/sip/invite) with priority levels from 1.0 (highest) to 9.9 (lowest) determining which classification wins when multiple signatures match. Fingerprints tag all matching patterns without priority competition and are stored alongside the winning AppID. The basic pattern matching supports Boolean logic (and, or, not), raw text keywords with minimum three-character length, regular expressions, binary/hex patterns, case sensitivity flags and UTF-16 matching. Built-in functions provide matching against IP addresses, ports, protocols, MAC addresses, email addresses (with DECODEORDAIN-style permutation) and positional constraints. Chainwords allow reusable pattern variables — predefined chainwords include $tcp, $udp, $http, $ssl, $vbulletin and $user_agent among others.\n\nContext-sensitive scanning is a key capability, allowing patterns to be evaluated only within specific extracted fields rather than across the entire session. Sample contexts include html_title, url, http_host, http_referer, http_cookie, user_agent, web_search, email_body, chat_body, doc_author, doc_org and filename — meaning a fingerprint can target only Google Maps requests appearing in the HTTP Host header rather than matching any mention of maps.google.com anywhere in a session. This improves both accuracy and performance.\n\nThe advanced syntax sections describe three progressively more powerful capabilities. Code-based AppIDs combine a keyword/regex trigger with embedded C++ code for secondary validation — useful for verifying packet structure, checksums, length fields and encoded content that cannot be matched with pattern syntax alone. The examples show OSPF protocol verification, VPN packet inspection examining specific byte positions, and Office document filename matching. Extractors provide simplified regex-based metadata extraction with named capture groups. Flex support enables lex-style tokenised pattern matching against session content. Microplugins represent the most advanced capability — giving code-based AppIDs limited access to XKeyscore's core, including reading session metadata (IP addresses, ports), throwing events (document_metadata events that feed into the document metadata search), and contributing derived fields to database tables like HTTP Activity (for example, extracting and URL-decoding Google Maps search terms for storage in the search_terms field).\n\nThe real-world example slide shows a captured Arabic-language vBulletin forum private message containing a complete \"ASRAR El Mojahedeen v2.0 Encrypted Message\" — the Mujahideen Secrets 2.0 encryption tool — demonstrating the operational context in which these fingerprints operate. The distribution slide confirms that AppID and fingerprint definitions are pushed across the entire global XKeyscore network every hour, with changes taking effect within two hours of check-in. The validation tooling shows AppID definitions loaded from configuration files including anonymizer.appid, bulletin_board.appid, tao_vpn.appid, terminal.appid and voip.appid, with unit testing against sample traffic captures.",
"tags": [
"AppID",
"fingerprints",
"pattern matching",
"C++",
"microplugins",
"chainword",
"context-sensitive scanning",
"regular expression",
"binary pattern",
"session metadata",
"HTTP Activity",
"document metadata",
"SMTP",
"ICQ",
"OSPF",
"Skinny",
"VPNs",
"Kazaa",
"Google Maps",
"vBulletin",
"Mujahideen Secrets"
],
"document": {
"creator": "NSA",
"date_created": "",
"classification": [
"TOP SECRET",
"COMINT"
],
"dissemination": [
"REL TO USA, FVEY"
],
"banner": "TOP SECRET//COMINT//REL TO USA, FVEY",
"genre": "slides",
"programs": [
"XKEYSCORE",
"DECODEORDAIN"
],
"countries": [],
"people": []
},
"publication": [
{
"publisher": "The Intercept",
"date_published": "2015-07-01",
"title": "XKEYSCORE: NSA’s Google for the World’s Private Communications",
"link": "https://theintercept.com/2015/07/01/nsas-google-worlds-private-communications/",
"reporters": [
"Marquis-Boire, Morgan",
"Greenwald, Glenn",
"Lee, Micah"
],
"contributors": [],
"doc_source": "https://theintercept.com/document/2015/07/01/xks-application-ids/",
"doc_url": "https://s3.documentcloud.org/documents/2116302/xks-application-ids.pdf"
}
],
"aka_id": [],
"pdf_sha256": "2e15431de13a51d2c49002a9fca5ab411dd646f36e66549847267f6f76e7e53f",
"pages": 42
}

View file

@ -0,0 +1,846 @@
TOP SECRET//COMINT//REL TO USA, FVEY
# XKEYSCORE Appids & Fingerprints
xkeyscore@nsa
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Basic Syntax
* Syntax is similar to C:
`function('name', level, <optional info> ) = 'search terms and patterns';`
* Two main functions -- appid and fingerprint:
`appid('chat/icq', 8.5, wireshark='icq', chatproc='ICQ') = /[^o]icq/c and $icq;`
`fingerprint('fingerprint/phone/nokia/generic') = 'user-agent: nokia' or 'profile: http://nds.nokia.com/uaprof/n';`
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Naming Conventions
Appids are named using a pseudo directory
convention:
`/application_type/sub_type/name`
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Levels
Levels are 1.0 – 9.9 with lower numbers meaning higher priority.
This allows multiple signatures to match a piece of traffic, but only the most specific appid will be applied. For example:
`appid('chat', 9.9) = ...`
`appid('chat/yahoo', 9.8) = ...`
`appid('chat/yahoo/incoming', 9.7 = ...`
If a session matches all three signatures, the appid will be 'chat/yahoo/incoming' since that has the best priority.
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Application Type
Third parameter is the _application type_; if missing, we use the appid name up to the first slash as the type
`appid('http/response', 9.2, 'web') = ...`
`appid('chat/yahoo/incoming', 9.1) = ...`
TOP SECRET//COMINT//REL TO USA, FVEY
--
TOP SECRET//COMINT//REL TO USA, FVEY
## Basic Search Patterns
XKEYSCORE supports Boolean operations and regular expressions
Raw text must be encapsulated between single quotes
* 'search term'
Terms can be combined with Boolean logic
* 'search term' and 'another term' and not 'defeat term'
* 'search term' or 'another term'
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Example
```
appid('voip/sip/IMS', 6.0, wireshark='sip') =
('via: sip' or 'v: sip') and 'cseq:' and (
'p-access-network-info:' or
'p-called-party-id:' or
'p-charging-vector:' or
'p-charging-vector-addresses:' or
'p-media-authorization:' or
'security-verify:' or
'proxy-authorization:' and 'scscf' or
'path:' and 'pcscf' or
'path:' and 'scscf'
);
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Binary and Regex Patterns
Binary patterns can be represented by putting a \x in front of each value:
`'\xff\xff\x00\x02'`
Or use the hex function:
`hex('ff:ff:00:02')`
Use slashes to enclose regular expressions:
`/[^a-zA-Z0-9]BTE/`
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Case sensitivity
Keywords and regular expressions are NOT case sensitive by default.
Append a 'c' to request case-sensitive evaluation:
`'keyword'c`
`/regex/c`
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Keyword Length
Keywords must be at least 3 characters or they will never hit. This minimum is increased to 4 at some sites for performance reasons.
Regular expressions must include a fixed "anchor" meeting the minimum keyword length.
Bad: `/[A-Z]{3}-[0-9]{3,5}/`
OK: `/ABC-[0-9]{3,5}/`
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Appid vs Fingerprint
Each session gets one appid -- lowest level wins. It gets databased in the 'application' field.
All matching fingerprints are stored in the 'fingerprint' field. Level is ignored and can be omitted from fingerprint definitions.
```
Application Type*:
Application Info*:
Application:
AppID (+Fingerprints)* [fulltext]: [Field Builder]
Winning appid
Winning appid + all fingerprints
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Example
```
appid('mail/yahoo', 9.0) = 'Host: mail.yahoo';
appid('mail/yahoo/login, 8.0) = 'Host: mail.yahoo' and '/login';
fingerprint('mail/arabic') = 'mail' and /language[:=] ?ar/;
fingerprint('mail/yahoo/ymbm') = 'Host: mail.yahoo' and 'YMBM='c;
GET /login.html HTTP/1.1
Referer: http://us.f359.mail.yahoo.com/ym/ShowLetter
Accept-Language: ar
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Host: mail.yahoo.com
Connection: Keep-Alive
Cookie: B=fn50ehd2612o2&b=3&s=rp; YMBM=d=&v=1;
```
Application: `mail/yahoo/login`
Fingerprint: `mail/yahoo/login mail/arabic mail/yahoo/ymbm`
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Real world example
```
لوحة التحكم
PM 08:04, 2008-18-02
Mar 2007
1,199
tawab
الإعدادات والخيارات
تعديل التوقيع
البريد الالكتروني / كلمة المرور
تعديل الملف الشخصي
تعديل الخيارات
الرسائل الخاصة
قائمة الرسائل
ارسال رسالة جديدة
رسالة خاصة: تفضل
غير متواجد حالياً
نائب المشرف
افتراضي تفضل
تتبع الرسائل
تعديل المجلدات
مواضيع مشترك بها
قائمة الاشتراكات
تعديل المجلدات
الرقابة
المواضيع المحذوفة
المشاركات المحذوفة
المواضيع المراقبة
المشاركات المراقبة
المتنوعات
البذكر بالحدث
قوائم الأصدقاء / التجاهل
الملفات المرفقة
تاريخ التسجيل : 2007 Mar
المشاركات : 1,199
### Begin ASRAR EI Mojahedeen v2.0 Encrypted Message ###
r/RgTzT/ATRhN2E1Zjg1OWQyNWRjMmE2ZTdINzZmZDhI0DUxZWZhMDQ1MjYwMjVIZGUO
ZGYwMjdkMmJmNTA4ZDY2Yjk0MGU2NGNYjg6MzNjZTc6MThjY2 Y1ZmY6MTgzZDIkYjhjMTE
xOGYzYjc1ZDdiMDAxNTQzZmVINDVIY2YyMGJjYjU20DkyYjdmYjFjYjAzMWM6ZDQ20WFIMzg
4NThhM211Mjc5ODkzZGNh0GRmNWJmNjVIZjQ0MjMxNDM4MDIyOTg1MmRjMGJINGNKYTN
KYTQ4MzMxZjRIN2FiNjI3MjE1NGI3MTA3ZDQ4NWRmYzMyOTUzZjZIMjg3NjQ10GQ4MTA3N
TU2N2ZkN2ZjYzUzYzYyMjFIODAwN2VkM2U5MTZINDY2MmM2ZTV IYjQ2YzI0OGQ20DUxNW
VkMjI2MWVINDAyOGI0MThkMTdhNTY1YzIxMDgyOGZIM2IwZWZj MDgwM2U4MzNINDg1OD
UxZTc4ODc1MTY2M210NjU5ZjBhZjVhNjk00TlhNGExOThmYWVI NmFIZjlyNmMwZDA3MDMO
NjJkZDhhMml4ZmRhYjc3NmZINDFk0DkyYjBhYjY3MDQ10GVIMj dhYmUwZTlyNGIxYmQyZDIz
ZjliM2E5ZGQ5NmNhZDQx0TM4NT|0Mjc3MzBIOWEwZWE1Njk3Yj gxY2VINTQ10WULnoi/VD
ULIJTEuDJqne0GMRHesi/BPTnZj02yqbmKbFkIPjwMhe7FUhFAOw74S+i+PokOREo5XhdP+ y9
Gul3juYTvrlE0xGx20sSfNS5kfRXXH1DaTnb7Oyufe9r6mMIQ6
1
e6E0SRUIdU6YVupz0hhgd4Dof
SBbFR30vgOS+pUxDYgmE0r/RA+fYi47tuHQMh+dynZqQspNdmRUmkjEpFqF03sPHS/10injqo
e1GsfB+xn52XE2q/WdnU+4XjWnl/isVNAjv2nsL+s2TG1lHbgocmpQoxy0B0SXPcRv/+2JekV37
XyONZk9YH+DV3aWYPXt+ym+wG0XNTqPHIU1JWAZqI2NK/cSXt9DMtCtcb8czRj6G9/XvJ9
Eny7t06xPd9BGio9M+3QuUkZHLEmJiAvgvB6R/X/3whBqk6zMHQLfo+VJcX9umW5mRtgCjzS
PW6lzzFCGt/B4SK4PxT52ZC0B2kWD8VMyNffrlsTG4XUesgx47Nd5xML8w5pj/fZwKNK+EfKIP
==Z1 ow29A9N3uLIXBX62LhOyj/1iqfJ2FNR7AIONSEjwKoggVmkxDiuGaQi+TurpxBgat1g
### End ASRAR EI Mojahedeen v2.0 Encrypted Message ###
Displaying 1 items Hidden fields
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Distribution
* Appids and fingerprints are distributed across the XKEYSCORE network every hour
* Changes will take effect within 2 hours of check-in
* Current definitions are available on the website:
http://xkeyscore.r1.r.nsa/documents/appid.html
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Intermediate Syntax
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Append Option
You can append derived metadata fields onto the end of an appid:
```
appid('p2p/kazaa', 7.7, append='mime_type') =
Ipos('x-kazaa') and not $http;
```
This will result in an appid like 'p2p/kazaa/image/jpeg'.
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Built in functions
| Function | Description |
| :--------------------- | :----------- |
| `ip(expr)` | Matches against an IP Address looks in to address and from address in the session headere<br>`• ip('10.10.10.1');` |
| `toport(expr)` | Matches against the Destination/To port. Note this must be a numeric representation of a port.<br> `• toport(1920);` |
| `fromport( expr)` | Matches against the Source/From port. Note this must be a numeric representation of a port.<br> `• fromport(80);` |
| `port(expr)` | Matches against the either port. Note this must be a numeric representation of a port.<br> `• port(6667);` |
| `next_protocol(expr)` | Matches against the integer version of the next protocol.<br> `• next_protocol(250);` |
| `protocol('text')` | Will only work for IP next protocol names as defined in the IANA next protocol numbers document<br> `• protocol('tcp');` |
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Built in functions
| Function | Description |
| :------------------ | :------------------------------------------------- |
| `email_address(sel)` | permutes just like strong_selector (just like DECODEORDAIN |
| `mac_address(addr)` | Tasks a mac address |
| `smac(addr)` | |
| `dmac(addr)` | |
| `ip(addr)` | tasks this IP address (either to or from) |
| `from_ip(addr)` | tasks this IP address only when it is the originator |
| `to_ip(addr)` | tasks this IP address only when it is the destination |
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## More built in functions
| Function | Description |
| :------------------ | :----------- |
| `first(expr)` | Matches against a pattern at the beginning of the session |
| `Ipos(expr)` | Matches against a pattern at the beginning of each line (\n) |
| `pos( expr)` | expression occurs at offset X in the session<br>`• pos('Hello') == 5,`<br>`• pos(/Good.*Grief/) <= 10` |
| `between(expr)` | `• between('Hello', 'World', 10, 100)`<br>Separation between 'Hello' and 'World' is greater than or equal to 10 bytes and less than or equal to 100 bytes<br>This is the same as using the following regular expression:<br>`• /Hello.{10,100}World/` |
| `'term'c` | Does a case sensitive match of the term |
| `'term'u` | Treats the term as UTF-16 |
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Example
`appid('voip/skinny(port2000)', 9.9, wireshark='skinny') = port(2000);`
`appid('voip/skinny/keep-alive', 3.0, wireshark='skinny') = toport(2000) and first('\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00');`
`appid('voip/skinny/keep-alive-ack', 3.0, wireshark='skinny') = fromport(2000) and first('\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00');`
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Example
```
appid('mail/smtp/to_server', 8.5, direction=$from_server,
wireshark='smtp') =
toport(25) and
( first('helo') or
first('ehlo') or
first('data') or
(Ipos('To: 'c) and Ipos('From:'c)) or
Ipos('QUIT'c) or
Ipos('mail from:') or
Ipos('rcpt to:') );
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## CHAINWORDs
You can assign a pattern to a variable (CHAINWORD) and reuse the variable in many patterns.
`$sip = 'via: sip' and 'cseq:' and 'SIP/2'c;`
Now we can use this variable in future definitions:
`appid('voip/sip', 7.2) = $sip;`
`appid('voip/sip/invite', 6.9) = $sip and 'INVITE';`
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Predefined Chainwords
There are a number of chainwords predefined for convenience:
* `$tcp`
* `$udp`
* `$icmp`
* `$sctp`
* `$rpc`
* `$arp`
* `$ssl`
* `$http_cmd`
* `$http`
* `$http_get`
* `$http_put`
* `$http_post`
* `$http_delete`
* `$http_trace`
* `$http_head`
* `$http_options`
* `$http_partial`
* `$vbulletin`
* `$mime_type`
* `$user_agent`
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Example
`$icq = 'ICQ'c and $http and not (port(80) or $html_body or $http_cmd);`
`appid('chat/icq', 8.5, wireshark='icq', chatproc='ICQ') = /[^o]icq/c and $icq;`
`appid('chat/icq', 9.0, wireshark='icq', chatproc='ICQ') = first('icq') and not port(25);`
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Context sensitivity
Expressions are evaluated only with a certain context instead of across the session as a whole.
`html_title('Yahoo! Mail' or 'Yahoo! Address Book')`
`... only hits if those keywords are seen within the title of a web page`
`http_host('maps.google.com')`
`... only hits within the "Host:" HTTP header`
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Context sensitivity
Why use context-sensitive scanning?
* More intuitive - you can say what you mean
* More accurate - if 'maps.google.com' is mentioned in a blog post, you don't want to try processing it as a Google Maps session
* Better performance for XKEYSCORE
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Context sensitivity
Sample contexts:
`html_title`
`url`
`http_host`
`http_referer`
`http_cookie`
`http_server`
`user_agent`
`web_search`
`to_cc`
`from_cc`
`filename`
`file_ext`
`doc_title`
`doc_subject`
`doc_author`
`doc_org`
`doc_hash`
`doc_body`
`email_body`
`chat_body`
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Example
```
appid('finance/currency_conversion/generic', 8.0) =
html_title('currency' and ('exchange' or 'conver')) or
http_server('currency' or 'x-rates.com');`
appid('finance/currency_conversion/xe', 8.0) =
http_host('xe.com') or
html_title(/^XE -/c or 'XE.com'c);
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Appid utility
appid options:
`--help` this help message
`--list-all` list all the application/fingerprint names and levels
`--list-appids` list all the application names (no fingerprints)
`--list-fingerprints` list all the application names (no appids)
`--list-types` list all the application types
`--list-levels` list all the application levels
`--unit-test` perform unit tests with data in the heirachy 'datadir', with files matching 'filespec'
`--quiet` don't print any load messages
`--appid_fname arg` location of appid.cfg
`--input-file arg` input file to test
`--datadir arg` The test data directory. Defaults to $(XSCORE_TEST_DATA_DIR)/appids
`--filespec arg (=.*\.u124)` A regular expression to match against files to check
`--noexit arg (=0)` do not stop on the first error
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Appid Validation
```
appid sample.u124
Loading appids
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/appid_definitions.cfg
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/anonymizer.appid
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/bulletin_board.appid
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/tao_vpn.appid
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/tdmoip.appid
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/terminal.appid
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/voip.appid
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/appid_definitions.cfg
Finished loading appids
Filename: sample.u124
Appid: encryption/https
Total Size: 19.36Kbits
Total Time: 0.01secs
Rate: 1.936Mbits/s
Overall performance:
Total Time: 0.01secs
Total Bits: 0.01936Mbits
Overall Rate: 1.936Mbits/s
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Advanced Syntax
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Code-based appids
Keywords and regular expressions don't work for everything
* Looking down columns in packet data
* Checksums
* Decoding (urlencoding, base64, gzip, etc.)
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Code-based appids
Basic idea:
1. Preliminary "trigger" using standard keywords and regular expressions
2. Secondary test using a snippet of C++ code
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Code-based appids
Example -- verifying a length field:
```
appid('netmanagement/ospf', 2, wireshark='ospf') =
protocol('ospf')
: c++ {{
if (size() < 4)
return false;
const uint8_t *data = begin();
return (data[3]==size());
}};
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Code-based appids
Example -- packet data:
```
fingerprint('vpn/xxp_example') =
'Next Protocol 250'
: c++ {{
packet_t pkt;
int count = 0;
while ((pkt = get_packet()) && count < 20) {
++count;
if (pkt.size < 16)
return false;
if (pkt.data[4] != 0xCC ||
pkt.data[5] != 0x45 ||
pkt.data[15] != 0x72)
return false;
}
return (count > 0);
}};
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Context-sensitive code
Example -- code-based check on certain extracted files:
```
fingerprint('crazy/office') =
extracted_file('doc' or 'xls' or 'ppt' : c++ {{
return xks::filename().find("crazy") != std::string::npos;
}});
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Extractors
Simplified regex-based metadata extraction
```
fingerprint('maps/google/example') =
http_host('.google.') and url('/maps?')
: c++
extractors : {{
ka = /Keep-Alive: (\d+)/;
accept[] = /Accept-([^:]+): ([\w-]+)/;
}}
main : {{
if(ka && ka[0] == "300") {
for(size_t i = 0; i < accept.size(); ++i)
if(accept[i][0] == "Encoding" && accept[i][1] == "gzip")
return true;
}
return false;
}};
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Flex
Support for flex-based pattern matching
```
fingerprint('maps/google/firefox') =
http_host('.google.') and url('/maps?')
: c++
flex : {{
USER_AGENT_CHAR [^\r\n]
%%
"User-Agent: "{USER_AGENT_CHAR}+ {
std::string agent(yytext);
if(agent.find("Firefox") != std::string::npos)
return true;
}
}};
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Microplugins
The next step: giving code-based appids (limited) access to the XKS core
* Accessing top-level session metadata
* Throwing common events
* Contributing metadata for databasing
The goal: higher level of agility with lower learning curve
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Microplugins
Example: accessing session metadata
```
fingerprint('maps/google/server1') =
http_host('.google.') and url('/maps?')
: c++
main : {{
return (SESSION["to_ip"] == "123.45.67.1");
}};
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Microplugins
Example: throwing a document_metadata event
```
fingerprint('maps/google/contrived') =
http_host('.google.') and url('/maps?')
: c++
main : {{
xks::doc_meta_t dm;
dm.filename = "google.txt";
dm.author = "Google, Inc.";
xks::document_metadata(dm);
}};
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Microplugins
Example: contributing metadata to HTTP Activity
```
fingerprint('maps/google/search') =
http_host('.google.') and url('/maps?')
: c++
extractors : {{
q = /[&?]q=([^&]+)/;
}}
main : {{
if(q) {
DB["http_parser"]["search_terms"] = xks::urldecode(q[0]);
DB.apply();
return true;
}
}};
```
TOP SECRET//COMINT//REL TO USA, FVEY