snowchive-meta/data/xks-application-ids/xks-application-ids.md
2026-11-19 04:39:11 +02:00

21 KiB
Raw Blame History

TOP SECRET//COMINT//REL TO USA, FVEY

XKEYSCORE Appids & Fingerprints

xkeyscore@nsa

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Basic Syntax

  • Syntax is similar to C:

    function('name', level, <optional info> ) = 'search terms and patterns';

  • Two main functions -- appid and fingerprint:

    appid('chat/icq', 8.5, wireshark='icq', chatproc='ICQ') = /[^o]icq/c and $icq;

    fingerprint('fingerprint/phone/nokia/generic') = 'user-agent: nokia' or 'profile: http://nds.nokia.com/uaprof/n';

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Naming Conventions

Appids are named using a pseudo directory convention:

/application_type/sub_type/name

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Levels

Levels are 1.0 – 9.9 with lower numbers meaning higher priority.

This allows multiple signatures to match a piece of traffic, but only the most specific appid will be applied. For example:

appid('chat', 9.9) = ... appid('chat/yahoo', 9.8) = ... appid('chat/yahoo/incoming', 9.7 = ...

If a session matches all three signatures, the appid will be 'chat/yahoo/incoming' since that has the best priority.

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Application Type

Third parameter is the application type; if missing, we use the appid name up to the first slash as the type

appid('http/response', 9.2, 'web') = ...

appid('chat/yahoo/incoming', 9.1) = ...

TOP SECRET//COMINT//REL TO USA, FVEY

--

TOP SECRET//COMINT//REL TO USA, FVEY

Basic Search Patterns

XKEYSCORE supports Boolean operations and regular expressions

Raw text must be encapsulated between single quotes

  • 'search term'

Terms can be combined with Boolean logic

  • 'search term' and 'another term' and not 'defeat term'
  • 'search term' or 'another term'

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Example

appid('voip/sip/IMS', 6.0, wireshark='sip') =
 ('via: sip' or 'v: sip') and 'cseq:' and (
  'p-access-network-info:' or
  'p-called-party-id:' or
  'p-charging-vector:' or
  'p-charging-vector-addresses:' or
  'p-media-authorization:' or
  'security-verify:' or
  'proxy-authorization:' and 'scscf' or
  'path:' and 'pcscf' or
  'path:' and 'scscf'
);

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Binary and Regex Patterns

Binary patterns can be represented by putting a \x in front of each value:

'\xff\xff\x00\x02'

Or use the hex function:

hex('ff:ff:00:02')

Use slashes to enclose regular expressions:

/[^a-zA-Z0-9]BTE/

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Case sensitivity

Keywords and regular expressions are NOT case sensitive by default.

Append a 'c' to request case-sensitive evaluation:

'keyword'c /regex/c

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Keyword Length

Keywords must be at least 3 characters or they will never hit. This minimum is increased to 4 at some sites for performance reasons.

Regular expressions must include a fixed "anchor" meeting the minimum keyword length.

Bad: /[A-Z]{3}-[0-9]{3,5}/ OK: /ABC-[0-9]{3,5}/

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Appid vs Fingerprint

Each session gets one appid -- lowest level wins. It gets databased in the 'application' field.

All matching fingerprints are stored in the 'fingerprint' field. Level is ignored and can be omitted from fingerprint definitions.

Application Type*:
Application Info*:
Application:
AppID (+Fingerprints)* [fulltext]: [Field Builder]

Winning appid
Winning appid + all fingerprints

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Example

appid('mail/yahoo', 9.0) = 'Host: mail.yahoo';
appid('mail/yahoo/login, 8.0) = 'Host: mail.yahoo' and '/login';

fingerprint('mail/arabic') = 'mail' and /language[:=] ?ar/;
fingerprint('mail/yahoo/ymbm') = 'Host: mail.yahoo' and 'YMBM='c;

GET /login.html HTTP/1.1
Referer: http://us.f359.mail.yahoo.com/ym/ShowLetter
Accept-Language: ar
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Host: mail.yahoo.com
Connection: Keep-Alive
Cookie: B=fn50ehd2612o2&b=3&s=rp; YMBM=d=&v=1;

Application: mail/yahoo/login Fingerprint: mail/yahoo/login mail/arabic mail/yahoo/ymbm

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Real world example

لوحة التحكم
PM 08:04, 2008-18-02
Mar 2007
1,199
tawab
الإعدادات والخيارات
تعديل التوقيع
البريد الالكتروني / كلمة المرور
تعديل الملف الشخصي
تعديل الخيارات
الرسائل الخاصة
قائمة الرسائل
ارسال رسالة جديدة
رسالة خاصة: تفضل
غير متواجد حالياً
نائب المشرف
افتراضي تفضل
تتبع الرسائل
تعديل المجلدات
مواضيع مشترك بها
قائمة الاشتراكات
تعديل المجلدات
الرقابة
المواضيع المحذوفة
المشاركات المحذوفة
المواضيع المراقبة
المشاركات المراقبة
المتنوعات
البذكر بالحدث
قوائم الأصدقاء / التجاهل
الملفات المرفقة
تاريخ التسجيل : 2007 Mar
المشاركات : 1,199

### Begin ASRAR EI Mojahedeen v2.0 Encrypted Message ###
r/RgTzT/ATRhN2E1Zjg1OWQyNWRjMmE2ZTdINzZmZDhI0DUxZWZhMDQ1MjYwMjVIZGUO
ZGYwMjdkMmJmNTA4ZDY2Yjk0MGU2NGNYjg6MzNjZTc6MThjY2 Y1ZmY6MTgzZDIkYjhjMTE
xOGYzYjc1ZDdiMDAxNTQzZmVINDVIY2YyMGJjYjU20DkyYjdmYjFjYjAzMWM6ZDQ20WFIMzg
4NThhM211Mjc5ODkzZGNh0GRmNWJmNjVIZjQ0MjMxNDM4MDIyOTg1MmRjMGJINGNKYTN
KYTQ4MzMxZjRIN2FiNjI3MjE1NGI3MTA3ZDQ4NWRmYzMyOTUzZjZIMjg3NjQ10GQ4MTA3N
TU2N2ZkN2ZjYzUzYzYyMjFIODAwN2VkM2U5MTZINDY2MmM2ZTV IYjQ2YzI0OGQ20DUxNW
VkMjI2MWVINDAyOGI0MThkMTdhNTY1YzIxMDgyOGZIM2IwZWZj MDgwM2U4MzNINDg1OD
UxZTc4ODc1MTY2M210NjU5ZjBhZjVhNjk00TlhNGExOThmYWVI NmFIZjlyNmMwZDA3MDMO
NjJkZDhhMml4ZmRhYjc3NmZINDFk0DkyYjBhYjY3MDQ10GVIMj dhYmUwZTlyNGIxYmQyZDIz
ZjliM2E5ZGQ5NmNhZDQx0TM4NT|0Mjc3MzBIOWEwZWE1Njk3Yj gxY2VINTQ10WULnoi/VD
ULIJTEuDJqne0GMRHesi/BPTnZj02yqbmKbFkIPjwMhe7FUhFAOw74S+i+PokOREo5XhdP+ y9
Gul3juYTvrlE0xGx20sSfNS5kfRXXH1DaTnb7Oyufe9r6mMIQ6
1
e6E0SRUIdU6YVupz0hhgd4Dof
SBbFR30vgOS+pUxDYgmE0r/RA+fYi47tuHQMh+dynZqQspNdmRUmkjEpFqF03sPHS/10injqo
e1GsfB+xn52XE2q/WdnU+4XjWnl/isVNAjv2nsL+s2TG1lHbgocmpQoxy0B0SXPcRv/+2JekV37
XyONZk9YH+DV3aWYPXt+ym+wG0XNTqPHIU1JWAZqI2NK/cSXt9DMtCtcb8czRj6G9/XvJ9
Eny7t06xPd9BGio9M+3QuUkZHLEmJiAvgvB6R/X/3whBqk6zMHQLfo+VJcX9umW5mRtgCjzS
PW6lzzFCGt/B4SK4PxT52ZC0B2kWD8VMyNffrlsTG4XUesgx47Nd5xML8w5pj/fZwKNK+EfKIP
==Z1 ow29A9N3uLIXBX62LhOyj/1iqfJ2FNR7AIONSEjwKoggVmkxDiuGaQi+TurpxBgat1g
### End ASRAR EI Mojahedeen v2.0 Encrypted Message ###

Displaying 1 items Hidden fields

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Distribution

  • Appids and fingerprints are distributed across the XKEYSCORE network every hour
  • Changes will take effect within 2 hours of check-in
  • Current definitions are available on the website:

http://xkeyscore.r1.r.nsa/documents/appid.html

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Intermediate Syntax

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Append Option

You can append derived metadata fields onto the end of an appid:

appid('p2p/kazaa', 7.7, append='mime_type') =
     Ipos('x-kazaa') and not $http;

This will result in an appid like 'p2p/kazaa/image/jpeg'.

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Built in functions

Function Description
ip(expr) Matches against an IP Address looks in to address and from address in the session headere
• ip('10.10.10.1');
toport(expr) Matches against the Destination/To port. Note this must be a numeric representation of a port.
• toport(1920);
fromport( expr) Matches against the Source/From port. Note this must be a numeric representation of a port.
• fromport(80);
port(expr) Matches against the either port. Note this must be a numeric representation of a port.
• port(6667);
next_protocol(expr) Matches against the integer version of the next protocol.
• next_protocol(250);
protocol('text') Will only work for IP next protocol names as defined in the IANA next protocol numbers document
• protocol('tcp');

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Built in functions

Function Description
email_address(sel) permutes just like strong_selector (just like DECODEORDAIN
mac_address(addr) Tasks a mac address
smac(addr)
dmac(addr)
ip(addr) tasks this IP address (either to or from)
from_ip(addr) tasks this IP address only when it is the originator
to_ip(addr) tasks this IP address only when it is the destination

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

More built in functions

Function Description
first(expr) Matches against a pattern at the beginning of the session
Ipos(expr) Matches against a pattern at the beginning of each line (\n)
pos( expr) expression occurs at offset X in the session
• pos('Hello') == 5,
• pos(/Good.*Grief/) <= 10
between(expr) • between('Hello', 'World', 10, 100)
Separation between 'Hello' and 'World' is greater than or equal to 10 bytes and less than or equal to 100 bytes
This is the same as using the following regular expression:
• /Hello.{10,100}World/
'term'c Does a case sensitive match of the term
'term'u Treats the term as UTF-16

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Example

appid('voip/skinny(port2000)', 9.9, wireshark='skinny') = port(2000);

appid('voip/skinny/keep-alive', 3.0, wireshark='skinny') = toport(2000) and first('\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00');

appid('voip/skinny/keep-alive-ack', 3.0, wireshark='skinny') = fromport(2000) and first('\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00');

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Example

appid('mail/smtp/to_server', 8.5, direction=$from_server,
  wireshark='smtp') =
    toport(25) and
    ( first('helo') or
      first('ehlo') or
      first('data') or
      (Ipos('To: 'c) and Ipos('From:'c)) or
      Ipos('QUIT'c) or
      Ipos('mail from:') or
      Ipos('rcpt to:') );

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

CHAINWORDs

You can assign a pattern to a variable (CHAINWORD) and reuse the variable in many patterns.

$sip = 'via: sip' and 'cseq:' and 'SIP/2'c;

Now we can use this variable in future definitions:

appid('voip/sip', 7.2) = $sip;

appid('voip/sip/invite', 6.9) = $sip and 'INVITE';

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Predefined Chainwords

There are a number of chainwords predefined for convenience:

  • $tcp
  • $udp
  • $icmp
  • $sctp
  • $rpc
  • $arp
  • $ssl
  • $http_cmd
  • $http
  • $http_get
  • $http_put
  • $http_post
  • $http_delete
  • $http_trace
  • $http_head
  • $http_options
  • $http_partial
  • $vbulletin
  • $mime_type
  • $user_agent

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Example

$icq = 'ICQ'c and $http and not (port(80) or $html_body or $http_cmd);

appid('chat/icq', 8.5, wireshark='icq', chatproc='ICQ') = /[^o]icq/c and $icq;

appid('chat/icq', 9.0, wireshark='icq', chatproc='ICQ') = first('icq') and not port(25);

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Context sensitivity

Expressions are evaluated only with a certain context instead of across the session as a whole.

html_title('Yahoo! Mail' or 'Yahoo! Address Book') ... only hits if those keywords are seen within the title of a web page

http_host('maps.google.com') ... only hits within the "Host:" HTTP header

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Context sensitivity

Why use context-sensitive scanning?

  • More intuitive - you can say what you mean
  • More accurate - if 'maps.google.com' is mentioned in a blog post, you don't want to try processing it as a Google Maps session
  • Better performance for XKEYSCORE

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Context sensitivity

Sample contexts:

html_title url http_host http_referer http_cookie http_server user_agent web_search to_cc from_cc filename file_ext doc_title doc_subject doc_author doc_org doc_hash doc_body email_body chat_body

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Example

appid('finance/currency_conversion/generic', 8.0) =
  html_title('currency' and ('exchange' or 'conver')) or
  http_server('currency' or 'x-rates.com');`

appid('finance/currency_conversion/xe', 8.0) =
  http_host('xe.com') or
  html_title(/^XE -/c or 'XE.com'c);

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Appid utility

appid options: --help this help message --list-all list all the application/fingerprint names and levels --list-appids list all the application names (no fingerprints) --list-fingerprints list all the application names (no appids) --list-types list all the application types --list-levels list all the application levels --unit-test perform unit tests with data in the heirachy 'datadir', with files matching 'filespec' --quiet don't print any load messages --appid_fname arg location of appid.cfg --input-file arg input file to test --datadir arg The test data directory. Defaults to $(XSCORE_TEST_DATA_DIR)/appids --filespec arg (=.*\.u124) A regular expression to match against files to check --noexit arg (=0) do not stop on the first error

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Appid Validation

appid sample.u124
Loading appids
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/appid_definitions.cfg
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/anonymizer.appid
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/bulletin_board.appid
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/tao_vpn.appid
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/tdmoip.appid
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/terminal.appid
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/voip.appid
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/appid_definitions.cfg
Finished loading appids
Filename: sample.u124

Appid: encryption/https

Total Size: 19.36Kbits
Total Time: 0.01secs
Rate: 1.936Mbits/s
Overall performance:
Total Time: 0.01secs
Total Bits: 0.01936Mbits
Overall Rate: 1.936Mbits/s

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Advanced Syntax

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Code-based appids

Keywords and regular expressions don't work for everything

  • Looking down columns in packet data
  • Checksums
  • Decoding (urlencoding, base64, gzip, etc.)

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Code-based appids

Basic idea:

  1. Preliminary "trigger" using standard keywords and regular expressions

  2. Secondary test using a snippet of C++ code

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Code-based appids

Example -- verifying a length field:

appid('netmanagement/ospf', 2, wireshark='ospf') =
  protocol('ospf')
  : c++ {{

    if (size() < 4)
      return false;

    const uint8_t *data = begin();
    return (data[3]==size());
  }};

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Code-based appids

Example -- packet data:

fingerprint('vpn/xxp_example') =
  'Next Protocol 250'
    : c++ {{
    packet_t pkt;
    int count = 0;
    while ((pkt = get_packet()) && count < 20) {
      ++count;
      if (pkt.size < 16)
        return false;
    
    if (pkt.data[4] != 0xCC ||
      pkt.data[5] != 0x45 ||
      pkt.data[15] != 0x72)
     return false;
    }
   return (count > 0);
  }};

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Context-sensitive code

Example -- code-based check on certain extracted files:

fingerprint('crazy/office') =
  extracted_file('doc' or 'xls' or 'ppt' : c++ {{
    return xks::filename().find("crazy") != std::string::npos;
  }});

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Extractors

Simplified regex-based metadata extraction

fingerprint('maps/google/example') =
  http_host('.google.') and url('/maps?')
  : c++
  extractors : {{
    ka = /Keep-Alive: (\d+)/;
    accept[] = /Accept-([^:]+): ([\w-]+)/;
  }}
  main : {{
    if(ka && ka[0] == "300") {
      for(size_t i = 0; i < accept.size(); ++i)
        if(accept[i][0] == "Encoding" && accept[i][1] == "gzip")
          return true;
    }
    return false;
  }};

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Flex

Support for flex-based pattern matching

fingerprint('maps/google/firefox') =
  http_host('.google.') and url('/maps?')
  : c++
  flex : {{
    USER_AGENT_CHAR [^\r\n]
    %%
    "User-Agent: "{USER_AGENT_CHAR}+ {
      std::string agent(yytext);
      if(agent.find("Firefox") != std::string::npos)
        return true;
    }
  }};

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Microplugins

The next step: giving code-based appids (limited) access to the XKS core

  • Accessing top-level session metadata
  • Throwing common events
  • Contributing metadata for databasing

The goal: higher level of agility with lower learning curve

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Microplugins

Example: accessing session metadata

fingerprint('maps/google/server1') =
  http_host('.google.') and url('/maps?')
  : c++
  main : {{
    return (SESSION["to_ip"] == "123.45.67.1");
  }};

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Microplugins

Example: throwing a document_metadata event

fingerprint('maps/google/contrived') =
  http_host('.google.') and url('/maps?')
  : c++
  main : {{
    xks::doc_meta_t dm;
    dm.filename = "google.txt";
    dm.author = "Google, Inc.";
    xks::document_metadata(dm);
  }};

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Microplugins

Example: contributing metadata to HTTP Activity

fingerprint('maps/google/search') =
  http_host('.google.') and url('/maps?')
  : c++
  extractors : {{
    q = /[&?]q=([^&]+)/;
  }}
  main : {{
    if(q) {
      DB["http_parser"]["search_terms"] = xks::urldecode(q[0]);
      DB.apply();
      return true;
    }
}};

TOP SECRET//COMINT//REL TO USA, FVEY