snowchive-meta/data/Elegant-Chaos-collect-it-all-exploit-it-all-plus/Elegant-Chaos-collect-it-all-exploit-it-all-plus.md
2026-11-19 04:39:11 +02:00

26 KiB
Raw Permalink Blame History

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

Elegant CHAOS

Project Leads: [REDACTED] Code Support: [REDACTED]

For more up-to-date information please visit this URL: [REDACTED]Elegant_Chaos For recent developments please contact the authors.

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

Small team = faster development

Specialities: [REDACTED]: project vision, keeping on track with partner sharing and end-to- end automation goals [REDACTED]: project vision, MySQL support, analytic perspective [REDACTED]: project organization, Whizbang/Cloudbase development, developers perspective [REDACTED]: code support for new data flows, MySQL, Whizbang development

Other contributors: Cloud Support Team [REDACTED] [REDACTED]


SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

Field Site Responsibilities

Sniff it All
Know it All
Collect it All
Process it All
Exploit it All
Partner it All

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

Sniff it all: Maximize receiving capabilities within our viewing arc Know it all: Survey enough to keep our finger on the pulse of the whole environment Collect it all: Maximize how many signals we can bring in the door simultaneously Process it all: Find the data in the signal Exploit it all: Find the intelligence in the signal Partner it all: Collaborate on techniques and share data with partners


SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

For Better Or Worse...

Collection access is increasing,

  • software modems (ASPHALT/A-PLUS)
  • new hardware/software solutions (STORMFORCE modems and DARKQUEST auto-survey)
  • new physical capacity (TORUS antennae)

Processing capabilities are increasing,

  • JCE, TINT, XKS Deep Dive

But size of analytic workforce is not!

  • more resources = more resource management

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

We have about 9100 signals in our view In 2008, we were only processing maybe 100-140 of these Director's edict to "collect it all" ASPHALT -> software modems processing low bit rate signals added a 300- signal capability STORMFORCE modems -> combination of hardware and software control increase capacity from 4 signals to 40 TORUS antenna -> 12 receivers (feeds) in place of 1

Now we are at ~500, moving quickly towards 1,000 Over the next year, we anticipate collecting 3,000 signals simultaneously

When a resource comes available, which signal do you place on collection???


SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

New Collection Posture

Sniff it All
   Torus increases physical access

Know it All
   Automated FORNSAT survey - DARKQUEST

Collect it All
   Increase volume of signals: ASPHALT/A-PLUS

Process it All
   Scale XKS and use MVR techniques

Exploit it All
   Analysis of data at scale: ELEGANTCHAOS

Partner it All
   Work with GCHQ, share with Misawa

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

Sniff it all: TORUS adds 12 new feeds to existing 20-some Know it all: DARKQUEST COMSAT development automates survey to capture all signals, at MHS, in a 2-week span. Collect it all: Increased modem capacity with software (APLUS) & STORMFORCE modems. Process it all: Scale XKS, consider Deep Dive XKS, use MVR techniques such as map/reduce and Cloudbase capabilities Exploit it all: Query focused datasets - Analysis of data at scale means automate, automate, automate. This is the motivation for ELEGANTCHAOS. Optimize automation to include: automate scoring of links based on current analytical priorities, and feeding prioritized list back to survey tools including DARKQUEST and modems. Secondary goals: transparent process; flexible scoring. Partner it all: Collaborate on techniques and share data with partners. Cloud solution = TINT; XKS scaling = JCE. First partner: GCHQ/Bude.


SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

ELEGANTCHAOS Goals

  • Goal: perform basic, time-sensitive analysis on all of MHS collection
  • Goal: create a prioritized list of signals (case notations) in our viewing arc
  • Goal: use this list to automatically drive collection as collection capabilities increase
  • Offshoot goal: create a product that analysts and collection managers can use to see into the system

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108


SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

ELEGANTCHAOS + Cloud

The MHS Cloud provides an excellent platform for this project:

  • data ingest, normalization, tagging
  • access to SIGINT data from various processors, from sustained mission + survey
  • access to a huge body of enrichment data
  • processing, storage, and web-hosting
    • considering decoupling these parts...

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108


SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

ELEGANTCHAOS In Context

COLLECTION
FRONT END RF Collection
STORM FORCE
DARK QUEST
ASPHALT

Collection Distribution
WC
XKS
METADATA
CLOUD
TURMOIL
METADATA
CLOUD
TARMAC
SLR
CLOUD

CLOUD

Multiple QFDs
EC GUI

ENRICHMENT
Open Source
Target Network
Service
IP Geo
System Status

Σ

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

TARMAC provides target activity, network space, XKS provides target activity, network space, technologies POPQUIZ provides malicious discovery across sessions using heuristic-type approaches.

Flexibility in terms of QFD's: derived from XKS, SLR, Popquiz, etc. Target activity Technology Geo-location Questions: can combine QFDs VPNs involving Ivory Coast? Paired links carrying VOIP or VPN? Malicious Activity on networks used by targets?

Experiment by doing Will challenge current tasking methods, hopefully make them easier ;-)


SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

EC System Overview

1 Data
SIGINT
MHS CLOUD SERVERS
enrichment

2 EC Code
QFD
questions
QFD
score log
QFD
QFD
QFD
QFD
QFD
QFD
QFD
3 GUI

MySql tables and code

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

This is a simplified view of the ELEGANTCHAOS machinery. Note: a key component that is not shown is the feedback loop into collection.

  1. SIGINT and enrichment data enters the system by being copied to the Cloud servers. This may take the form of a MAILORDER flow, a wget grab, or a database file transfer from another system.
  2. Data is processed. Some data is processed through the SIGDEV Cloud Stack, which formally validates/normalizes/tags the data. All SIGINT data goes through this process, as well as some of the enrichment data. The remaining enrichment data usually requires some minimal processing or reformatting.
  3. The results of step 2, whether pulled from the SIGDEV Cloud Stack via the WhizBang map/reduce API, or copied from an external source, are stored in Question Focused Datasets (QFDs). Some QFDs serve multiple analytic interests, and some analytic interests require an intersection of QFDs to evaluate.
  4. ELEGANTCHAOS MySql code pulls analytical questions from a database, queries the QFDs to find case notations satisfying each question, and writes scores to another database. These scoring databases at the heart of EC populate the GUI.

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

Data Sources (May 2011)

SIGINT Feeds

  • XKEYSCORE
  • ASDF (Turmoil LIVE)
  • SLR (TARMAC)
  • POPQUIZ (Turmoil DEV)
  • WEALTHYCLUSTER2

Enrichment Feeds

  • IPGeoTrap
  • TRAVELLINGWAVE Scores
  • BILBOBADGER Daily Summaries
  • Target Network Service list + CNO Target list
  • DRINKYBIRD monitoring info
  • GLOBETROTTER OH Geo
  • MASTERSHAKE Geo
  • Quantumable Case Notation list

Event counts over a 12-hour period. Total events: 335,663,981

TU/Live
TU/Dev
SLR
WC2
XKS

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108


SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

Analytic Questions (May 2011)

Target

  • Dictionary hits
  • Target Networks
    • TNS, TW, CNO
  • PLUS Reports, CRNs, etc.

Technology

  • [REDACTED]
  • VPNs
  • Twitter, Facebook, VoIP
  • CNO behavior

Miscellany

  • Modem Capacity
  • Paired Links
  • Quantumable

Location

  • IP-based
  • MAC-based
  • Geo-based
  • Surge Countries
    • Libya, Egypt, Afghanistan, Syria, Yemen, Ivory Coast, etc.

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

Don't forget combinations (paired links with VPNs, VPNs with target IP networks)


SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

Questions & Scoring

  • Each question is represented by a SQL query applied to one or more QFDs
  • QFDs are case notation-based repositories of signal information
    • eg, IPs and registries for all case notations
    • eg, category hits for all case notations
    • eg, GLOBETROTTER geos for all case notations
  • All questions are asked once per day across all case notations
  • Points are assigned to each question based on current analytic priorities
  • Points for any particular question are "active" for a window of time (eg, 1 day, 7 days, 30 days)
  • The sum of "active" points for a case notation, across all questions, forms the score

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108


SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

Interfaces

Different interfaces for different customers

  • ELEGANTCHAOS GUI
    • made for analysts to examine scores and the impact of the different questions
    • eventually, control over the algorithms may reside here
  • REST interface
    • made for programmatic query, precursor to auto tasking
  • DRINKYBIRD GUI
    • made for collection personnel to determine if resources are available, easy to view what's on cover

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108


SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

EC GUI: Case Notation View

Elegant CHAOS

All Questions || All CASNs || Survey CASNs

Show 10 entries [Refresh]
casen_id casen score direction notes first_heard last number_of_day
1 [REDACTED] 157040 forward [REDACTED] 2010-12-26 2011-03-17 25
101 [REDACTED] 55500 forward [REDACTED] 2010-12-26 2011-03-17 25
5275 [REDACTED] 33160 forward [REDACTED] 2011-03-16 2011-03-17 0
5288 [REDACTED] 31370 unknown [REDACTED] 2011-03-17 2011-03-17 0
5285 [REDACTED] 26890 unknown [REDACTED] 2011-03-17 2011-03-17 0
1110 [REDACTED] 17450 return [REDACTED] 2011-01-03 2011-03-17 17
5292 [REDACTED] 15330 unknown [REDACTED] 2011-03-17 2011-03-17 0
5277 [REDACTED] 14210 unknown [REDACTED] 2011-03-17 2011-03-17 0
5289 [REDACTED] 12930 unknown [REDACTED] 2011-03-17 2011-03-17 0
5283 [REDACTED] 12630 unknown [REDACTED] 2011-03-17 2011-03-17 0
Showing 1 to 10 of 3,888 entries
First Previous 1 2 3 4 5 Next Last

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

Continual assessment of bearers helps to determine when a bearer becomes less interesting, then it'd be possible to remove it from sustained collection.


SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

EC GUI: Case Notation View

Mozilla Firefox
File Edit View History Bookmarks Tools Help
bucket elegantchaos
Tag This!! $$ MHS AM Notes Puzz Tiny Base Java API 1.6 cheat MySQL MySQL Syntax Tools misc RB EChaos Cloud Web Prog Mett
(U) XKS ... http...desc GM ROY... XKEYSC... FORESIG... (U) Whiz... [REDACTED] cl4r2hd2... Safari Bo... Private N... Matatu ht...sc DRINKYBI... Mass Tra...

TOP SECRET // COMINT // REL TO USA, FVEY

Elegant CHAOS

Not seeing any results?

All Questions || All CASNs || Survey CASNs

Show 10 entries Refresh
casen_id casen score direction notes first_heard last_heard number_of_day
1 [REDACTED] 157540 forward [REDACTED] 2010-12-26 2011-03-17 25
103 [REDACTED] 30000 forward [REDACTED] 2010-12-20 2011-03-17 23
5275 [REDACTED] 33660 forward [REDACTED] 2011-03-16 2011-03-17 0
5288 [REDACTED] 31870 unknown [REDACTED] 2011-03-17 2011-03-17 0
5285 [REDACTED] 27398 unknown [REDACTED] 2011-03-17 2011-03-17 D
1110 [REDACTED] 18950 return [REDACTED] 2011-01-03 2011-03-17 17
5292 [REDACTED] 15830 unknown [REDACTED] 2011-03-17 2011-03-17 0
5277 [REDACTED] 14710 unknown [REDACTED] 2011-03-17 2011-03-17 D
5289 [REDACTED] 13430 unknown [REDACTED] 2011-03-17 2011-03-17 0
5283 [REDACTED] 13130 unknown [REDACTED] 2011-03-17 2011-03-17 0
Showing 1 to 10 of 4,106 entries

First Previous 1 2 3 4 5 Next Last
topic question weight question score first contributed
Egypt 1000 7000 2011-02-18
Qatar 100 200 2011-02-27
CNO Targets 500 500 2011-03-17
Paired Link 50 200 2011-02-23
Priority 3 CategoryHits 50 2400 2011-03-16
Jordan 1000 1000 2011-03-17
Afghanistan 50 50 2011-03-04
Libya 1000 7000 2011-02-18
Priority 4 CategoryHits 20 600 2011-03-26
Points earned over time

Select a case notation
Observe questions which affected its score

x Find 192 Previous Next Highlight all Match case Phrase not found
elegantchaos.bucket.r6.r.nsa FoxyProxy Patterns

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108


SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

EC GUI: Question View

TOP SECRET//SI/TK//REL TO USA, AUS, CAN, GBR, NZL

Elegant CHAOS

Not seeing any results?

All Questions || All CASNo || Survey CASNO

Show 111 entries [Refresh]
question_id topic weight days_between_run days_before_points_expire description active_score total_score expired_score first_active_
16 1000 1 30 9000 16000 6000 2011 02 18
1 TRAFFICTHIEF lit 1000 1 7 1000 points given each day for each case notation with a priority 1 (TRAFFICTHIEF) tip seen 28000 116000 118000 2011-03-15
17 Libya 1000 1 30 1000 points for each case notation with a Libyan IP address on the significant side of a link 117000 216000 99000 2011-02-17
13 Egypt 1000 1 30 Gives case notation 1000 points if the known significant/GAT side of the link has an IP address that resolves to Egypt 62000 172000 110000 2011-02-16
14 FORNSAT CRNo 500 / / 500 points for each case with a current FORNSAT CRN 185000 185000 145000 2011-01-16
12 Tasked Active Users 150 1 7 150 points given for a case notation if it has Marina Daily Active Counts greater than 0 in the BILBOBADGER Daily Summary 1350 16050 14700 2011-03-15

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

Important to remember, this is our attempt to best utilize increasing capacity. Sits between sustained/CRN collection and continual survey.

Provides ability to turn the dials on “hot topic” of the day. Can adjust the length of time an event remains interesting and affects prioritization. Some Questions will always be run. TT hit, confirmed target activity, [REDACTED], etc. Maintains history, so progress can be tracked. Are there more target hits, [REDACTED] paired VOIP signals, etc.


SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

EC GUI: Question View

Mozilla Firefox
File Edit View History Bookmarks Tools Help
bucket elegantchaos
Tag This!! $$ MHS AM Notes Puzz Tiny Base Java API 1.6 cheat MySQL MySQL Syntax Tools misc RB EChaos Cloud Web Prog Mett
(U) XKS ... http...desc GM ROY... XKEYSC... FORESIG... (U) Whiz... WhizBan... WhizBan... cl4r2hd2... Safari Bo... Private N... Matatu ht...sc DRINKYBI... Mass Tra...


TOP SECRET//SI//TK//REL TO USA, AUS, CAN, GBR, NZL

Elegant CHAOS

Show 10 entries
Refresh
question_id topic weight days_between_run days_before_points_expire description active_score total_score expired_score first_ac
CNO
23 CNO Targets 500 1 7 Gives case notation 500 points for each day (up to 7 most recent days) in which target traffic hit on CNO XKS fingerprints 44500 44500 2011-03-
Showing 1 to 1 of 1 entries
First Previous 1 Next Last

Question 23
Points Score Date
500 2590 2011-03
500 3080 2011-03
500 15830 2011-03
500 18950 2011-03
500 13430 2011-03
500 500 2011-03
500 820 2011-03
500 8210 2011-03
500 500 2011-03
500 1580 2011-03
500 13130 2011-03
500 600 2011-03
500 2730 2011-03
500 9080 2011-03
500 820 2011-03
500 3400 2011-03
500 1640 2011-03
500 570 2011-03
Points earned over time (chart)

Filter based on a topic; select a question
Observe case notations which scored

x Find 192 Previous Next Highlight all Match case Phrase not found
elegantchaos.bucket.r6.r.nsa FoxyProxy Patterns

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108


SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

Focus Areas: Custom Views?

Home || All Questions || All CASNs || Survey CASNs || ASPHALT CASNS || Custom SQL Queries

Show 50 entries [Refresh]
id name notes active_score first_run
2 geo focus on geographical area 885195 2011-02-23
3 target focus on target network or bad actor (via alert or dictionary hit) 528017 2011-01-19
5 other focus on signal parameters, resource availability, casn type, etc 33050 2011-03-22
1 technology focus on application or protocol 0 2011-03-30
4 compound questions with multiple focus areas 0 2011-03-30
Showing 1 to 5 of 5 entries
First Previous 1 Next Last
question topic active score weight first contributed last contributed
17 Libya 420000 1000 2011-02-23 2011-05-02
13 Egypt 187000 1000 2011-01-31 2011-05-02
25 Jordan 100000 1000 2011-03-17 2011-05-02
26 Syria 67000 1000 2011-03-21 2011-05-02
33 Syria GLOBETROTTER 42301 1000 2011-04-05 2011-05-02
6 Afghanistan 35900 50 2010-12-29 2011-05-02
32 Libya GLOBETROTTER 12661 100 2011-04-04 2011-05-02
24 Yemen 7000 1000 2011-03-17 2011-05-02
20 Qatar 6000 100 2011-03-03 2011-05-02
18 Bahrain 5100 100 2011-03-03 2011-05-02
34 Yemen GLOBETROTTER 1433 100 2011-04-05 2011-05-02
19 Oman 600 100 2011-03-03 2011-05-02
22 Algeria 100 100 2011-03-03 2011-05-02
21 Morocco 100 100 2011-03-03 2011-05-02
30 Libya Facebook 0 0 2011-03-30 2011-04-01
29 Libya Twitter 0 0 2011-03-30 2011-04-01
31 Libya TOR 0 0 2011-03-30 2011-04-01
9 Ivory Coast null 200 2010-01-05 2011-05-02

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108


SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

REST: “Auto-tasking”

  • ASPHALT
    • Updated list of prioritized casns

[REDACTED]

VENUSAFFECT

  • Scores in DRINKYBIRD
  • Using modem tasking sheet

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

Interface to ASPHALT is easy with the developers on site !! – CSV file with required case notations FTP'd Interface kludge into DRINKYBIRD – In-House network and tasking management GUI


SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

DRINKYBIRD

Elegant Chaos tasking priority

Survey Casen Casen Satellite Frequency Polarity Feed Rasin Data Rate EC Priority On Cover?
[REDACTED] [REDACTED] BB 12551 VER K3V IDIRECTOC 19994.703 7300 no
[REDACTED] [REDACTED] -- -- -- -- -- -- 5000 no
[REDACTED] [REDACTED] BB 12625.52 VER K3V IDIRECTOC NaN 4350 no
[REDACTED] [REDACTED] BB 12563.008 VER K3V WC1A 2048 4000 no
[REDACTED] [REDACTED] -- -- -- -- -- -- 3000 no
[REDACTED] [REDACTED] BB 12653.745 VER K3V IDIRECTOC -- 3000 no
[REDACTED] [REDACTED] -- -- -- -- -- -- 3000 no
[REDACTED] [REDACTED] -- -- -- -- -- -- 2200 no
[REDACTED] [REDACTED] -- -- -- -- -- -- 2200 no
[REDACTED] [REDACTED] BB 12658.442 VER K3V IDIRECTOC 7466.537 2000 no
[REDACTED] [REDACTED] -- -- -- -- -- -- 1200 no
[REDACTED] [REDACTED] NF 12580.063 HOR K3H IDIRECTOC 4525.582 12600 yes
[REDACTED] [REDACTED] 2C 11501.069 HOR K2H IDIRECTOC 3780 8200 yes
[REDACTED] [REDACTED] 5B 12666.495 VER K3V DVBS 21000 7050 yes

Survey Casen | Casen

EC Priority On Cover?
7300 no

DRINKYBIRD – Tasking Priority View

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108


SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

Libya Surge

PROBLEM

Which of the 1000's of signals surveyed have Libyan / Egyptian / Afghan networks on the VSAT-side?

SOLUTION

  • Pre-run analytics determine “significance”
  • Quick identification of 25 'LY' / 11 'EG' / 10 'AF' candidate signals
  • Combine other analytics: target hits, pairing, etc.
  • (Repeat for next country)

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108


SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

AMULET STELLAR

PROBLEM

Which case notations have traffic on IPs of interest that matches AST fingerprints?

SOLUTION

  • Create IP target set; create whitelist and blacklist of XKS fingerprints
  • Use Cloud capabilities to bridge between the set of all SIGINT events with matching fingerprints, and the target set
  • Add the scoring question to EC

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

  • Correlated IRC Botnet Activity to ASR Intrusion set (IRGC-QF Ramazan Corps Headquarters.)
  • Discovered New Victims in Iraq and Iran
  • Multiple targets associated with S2E tasked selectors
  • Discovered New Infrastructure
  • Engaged TAO for Vulnerability Assessment Evaluation
  • Discovered potential C2, update and exfiltration nodes
  • Evaluating 4th Party Collection Opportunities
  • Tipped: SSG, S2E, NTOC-G, TAO CCNE, documented in CROSSBONES.
  • No specific fingerprints highlight this as ASR activity
  • Develop new “Infrastructure Agnostic” ASR fingerprints
  • Use AES keys for decrypt and potential 4th party collection opportunity

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

Ongoing Work

  • New data feeds (FOGHORN, MATCHMAKER, ROADBED)
  • More fields from XKS (HTTP language, NetStrings)
  • XKS from MOONPENNY
  • Fine tuning of GUI for Link Characterization Analysts
  • NetStrings study
  • Better use of Cloud resources (Link Direction) (CCDP)
  • Detailed study of scoring methodology (math hire)
  • Close the auto-tasking loop (RSE)
  • Increase awareness and partnership with similar efforts
  • Training

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108


SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

Questions?

SECRET//REL TO USA, AUS, CAN, GBR, NZL//20320108

Is the enterprise considering SLR generation?