snowchive-meta/data/Xks-Helper-Notes-Redacted/Xks-Helper-Notes-Redacted.md
2026-11-19 04:39:11 +02:00

10 KiB

TOP SECRET STRAP 1

XKEYSCORE HELPER NOTES

There are several new and updated features in this release of the XKEYSCORE Palantir helper:

  • Summary/Histogram import of data
  • Data sourcing for XKEYSCORE queries
  • Fixes for UI redraw bugs on query list refresh
  • Fixes for disappearing links

Summary import

This feature is intended to mirror the functionality in XKEYSCORE for creating histogram grids over a query. It allows for a large dataset to be reduced down in size considerably while still maintaining useful data. As an example this is a histogram grid view over a small query in XKEYSCORE, histogrammed by From IP, To IP and To Port:

Histogram Grid

Page 1 of 1 Clear Selection Export Displaying 1-39 of 39
| Filter | Fm IP      | To IP      | To Port | Count▾ |
|--------|------------|------------|---------|--------|
|        | [REDACTED] | [REDACTED] | 20644   | 19     |
|        | [REDACTED] | [REDACTED] | 8676    | 7      |
|        | [REDACTED] | [REDACTED] | 38960   | 6      |
|        | [REDACTED] | [REDACTED] | 25193   | 4      |
|        | [REDACTED] | [REDACTED] | 53      | 3      |
|        | [REDACTED] | [REDACTED] | 27977   | 2      |
|        | [REDACTED] | [REDACTED] | 32768   | 2      |
|        | [REDACTED] | [REDACTED] | 53      | 2      |
|        | [REDACTED] | [REDACTED] | 53      | 2      |
|        | [REDACTED] | [REDACTED] | 52835   | 2      |
|        | [REDACTED] | [REDACTED] | 47565   | 2      |
|        | [REDACTED] | [REDACTED] | 27977   | 2      |

As you can see, there are 19 entries for the top line here. In the old XKEYSCORE helper this would create 19 new events. While you still have the option of importing every row in an XKEYSCORE query as a new connection, the summary import lets you cut this down a little. Once logged in to the helper, choose the "Summarise by..." button:

History | Timeline | XKEYSCORE (LIVE)
------------------------------------------------------------------------------------------------------------
Add to graph
Refresh query list
Choose new node colour (First)
Choose new node colour (Last)
Summarise by...
[ ] Summarise results
[ ] Auto-merge links

Add to Gra... | Import as | Query Name  | Status   | Query Type  | Num Resu... | Num DBs  | Datetime S...  |              [ ]           | Classic   | http_parser | finished | http_parser | 2974        | 3 of 3   | 2011-03-24 ... |
[ ]           | Classic   | full_log    | finished | full_log    | 80          | 12 of 12 | 2011-03-22 ... |
[ ]           | Classic   | full_log    | finished | full_log    | 85          | 12 of 12 | 2011-03-21 ... |
[ ]           | Classic   | full_log    | finished | full_log    | 37          | 12 of 12 | 2011-03-21 ... |

1 of 6

[REDACTED]

TOP SECRET STRAP 1

TOP SECRET STRAP 1

To mirror the histogram grid performed on the data, I've chosen to include to_port:

Choose fields

Excluded              | Included
----------------------|-----------
fm_port               | to_port
casenotation          |
sigad                 |
fm_country_ip_        |
fm_city_ip_           |
to_country_ip_        |
to_city_ip_           |
session_length        |
appid_fingerprints_   |
filepath              |
query                 |
host                  |
http_type             |
browser               |
application_info      |
bluesmoke_id          |
intrusion_set         |
organisation          |
implant_id            |
implant_name          |
attribute_name        |
attribute_value       |
browser               |
application           |

Save Cancel

Note that when doing a summary import, summarisations will be done on source and destination IP in addition to any included fields.

2 of 6

[REDACTED]

TOP SECRET STRAP 1

TOP SECRET STRAP 1

Graph - Administrator Account 03/25/2011 12:02 GMT
Investigation Edit Select Preferences Windows Help
TS STRAP 1
Import Click here to search Search
Histogram Selection Filters
Type: Connection
Date Range: 2011/03/21 05:29:58.000
+00:00 to 2011/03/21 08:44:57.000 +00:00.
Properties

Type              | Values                   | Classification
Application       | P2p/bittorrent           | TS STRAP1
                  | P2p/bittorrent/bencoding | TS STRAP1
IP Address (From) | [REDACTED]               | TS STRAP1
IP Address (To)   | [REDACTED]               | TS STRAP1
Port (To)         | 20644                    | TS STRAP1
Quantity          | 19                       | TS STRAP1
Session Size      | 1764 B                   | TS STRAP1

Add Matches to Graph

Related Entities

Relationship   | Related    | Data | Data End | Test
Appears In (2) | [REDACTED] |
               | [REDACTED] |

Add Entities to Graph

Related Documents
No related documents.

Recent History
[REDACTED]
[REDACTED]

Current Selection
Connection (1)
[REDACTED]

History | Timeline | XKEYSCORE (LIVE)
------------------------------------------------------------------------------------------------------------
Add to graph
Refresh query list
Choose new node colour (First)
Choose new node colour (Last)
Summarise by...
[x] Summarise results
[ ] Auto-merge links

Add to Gra... | Import as | Query Name  | Status   | Query Type  | Num Resu... | Num DBs  | Datetime S...  |              [ ]           | Classic   | id by       | finished | http_parser | 2974        | 3 of 3   | 2011-03-24 ... |
[ ]           | Classic   | full_log    | finished | full_log    | 80          | 12 of 12 | 2011-03-22 ... |
[x]           | Classic   | full_log    | finished | full_log    | 85          | 12 of 12 | 2011-03-21 ... |
[ ]           | Classic   | full_log    | finished | full_log    | 37          | 12 of 12 | 2011-03-21 ... |

In this example, I removed all the other data from the input from the graph.

There are a few things to note from the results of this import:

  • Quantity records the number of results which matched that histogram criteria (In this case 19). This matches up with the XKEYSCORE histogram grid
  • Session size is a sum of all session sizes for this histogrammed piece of data. This allows you to see the total amount of data being sent from one IP to another, in this case also summarised by destination port.
  • "Application” shows the different fingerprints hit on for this summary event. For example, if X contacts Y and it is picked up by fingerprints foo on connection 1 and bar on connection 2, the summarised connection between X and Y will list "foo" and "bar" as applications
  • Time metadata is preserved in that you can view the first time this event occurred and the last.

3 of 6

[REDACTED]

TOP SECRET STRAP 1

TOP SECRET STRAP 1

Preferences for which fields to summarise by, whether you wish to summarise and whether you wish to automatically merge links between IP addresses and connections are saved per-user, so if you have a common histogram import then you don't need to re-select the fields to histogram on every time.

Data sourcing

Data imported into Palantir using the updated XKEYSCORE helper now has data sourcing.

There are a couple of places this can be seen, the most evident is the "Data sources" application within Palantir. So, when you open up Data Sourcing:


Search
+ Data Repository
+ Manually Entered Data
- XKEYSCORE
  + XKEYSCORE QUERY [REDACTED]

XKEYSCORE QUERY [REDACTED]
Data Source has 72 records, now showing:
32 Entities; 39 Events; 1 Document

Contents of XKEYSCORE QUERY [REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]

At the top level in this screenshot you can see there are three folders. The XKEYSCORE folder contains a list of the IP addresses and connection events associated with the query. Double clicking the document within this datasource opens up some metadata about the query run.

4 of 6

[REDACTED]

TOP SECRET STRAP 1

TOP SECRET STRAP 1

XKEYSCORE QUERY [REDACTED]
Data Source has 72 records, now showing:
32 Entities; 39 Events; 1 Document

Contents of XKEYSCORE QUERY [REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
back fwd path map search extract export +font -font shot prev next add show/hide find+tag opts

TS STRAP1
Hide Summary

XKEYSCORE QUERY [REDACTED]

Related | 
--------|
0       |
0       |
4       |
0       |

Object Info
Type: Document
Created by: Administrator Account
Created at: 2011/03/25 12:31:36+00:00
Classification: TS STRAP1

Who's Watching This?
No watchers
watch this object

Document | Properties | Related

XKEYSCORE QUERY [REDACTED]

Datetime: 2011-03-21 13:18:35
Output [REDACTED]
Query name [REDACTED]
DBs: 12 of 12
Hits: 85
Status: 100%

5 of 6

[REDACTED]

TOP SECRET STRAP 1

TOP SECRET STRAP 1

This information can also be accessed via an object imported into Palantir:

TS STRAP1

[REDACTED] More...

Related |
--------|
2       |
0       |
12      |
0       |

Type: Connection
Created by: Administrator Account
Created at: 2011/03/25 12:31:42+00:00
Date: 2011/03/21 05:29:58.000 +00:00 to 2011/03/21 08:44:57.000 +00:00

Who's Watching This?
No watchers
watch this object

Summary | Properties | Related | Notes & Media | History

Basic Info
Type: Connection
Label: [REDACTED]
Classification: TS STRAP1 | Sourcing....

Date Range
Begin: Mar 21, 2011 05:29:58.000 +00:00
End: Mar 21, 2011 08:44:57.000 +00:00

Location
Format: Lat/Lon
Latitude:
Longitude:

Add to Map
Sourcing...
View Permissions

Properties
Search for Properties Add

Type                 | Value                    | Classification
---------------------|--------------------------|-----------------
Application:         | P2p/bittorrent           | TS STRAP1
                     | P2p/bittorrent/bencoding | TS STRAP1
IP Address (From):   | [REDACTED]               | TS STRAP1
IP Address (To):     | [REDACTED]               | TS STRAP1
Port (To): 20644     | 20644                    | TS STRAP1
Quantity: 19         | 19                       | TS STRAP1
Session Size: 1764 B | 1764 B                   | TS STRAP1

Search Delete New Permissions

Property Type: Session Size
1764 B

Property Fields
Unit: B
Number: 1764
Role: None

Date
Metadata
Created: 03/25/2011 12:31 GMT
Last Modified: 03/25/2011 12:31 GMT

Data Sources
Data Sources: edit | view all documents
- (1) XKEYSCORE QUERY (tcariso 1)
  TS STRAP1

Q Search | X Delete | View Permissions

Number: 1764
IP Address (To): 46.21.87.220 | TS STRAP1

After multiple imports of XKEYSCORE data have been done within the same investigation the list of data sources also grows appropriately:

Data Sources - Administrator Account 03/25/2011 12:02 GMT
Investigation Edit Preferences Windows Help

+ Data Repository
+ Manually Entered Data
- XKEYSCORE
   + XKEYSCORE QUERY [REDACTED]
   + XKEYSCORE QUERY [REDACTED]

6 of 6

[REDACTED]

TOP SECRET STRAP 1