first commit

This commit is contained in:
libroot 2026-11-19 04:39:11 +02:00
commit 0d90363116
Signed by: libroot
GPG key ID: 2AC68C029EBE9ACE
6510 changed files with 679146 additions and 0 deletions

View file

@ -0,0 +1,573 @@
SECRET STRAP 1
APPLIED RESEARCH
# Contact chaining
Task leader: [REDACTED]
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]
Slide 15
---
SECRET STRAP 1
## Contact-chaining
* Start with known targets (comms identifiers)
* Expand data involving them to find new interesting comms identifiers
* B17 has worked on this for more than 8 years...
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]
Slide 16
---
SECRET STRAP 1
## Contact-chaining:
### Past prototypes
```
SNAPDRAGON
PRIME TIME
CARPAT
MoaG
SORTING FRIENDS
CHART BREAKER
```
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ o[REDACTED]
Slide 17
---
SECRET STRAP 1
## Contact-chaining:
### Drivers of current work
* Scoring, not just filtering
* Combining different algorithms
* C2C-specific techniques
* Graph representation – loss of information
* Far more rubbish in C2C
* Growth of events volumes
* Question-based, interactive analysis
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ o[REDACTED]
Slide 18
---
SECRET STRAP 1
## Combining scores:
### SORTING FRIENDS
* Clustering telephone numbers that have a strong relationship – extrapolating social relationships from events
* Direct communication
* Co-location relationships
* Timing and sequence
* Call backs
* Similar number
* Equipment sharing
* Calling profile similarity
* Etc., etc.
* (currently 12 relationships in total)
* Research prototype – SD can run... operational prototype will be through CHART BREAKER
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ o[REDACTED]
Slide 19
---
SECRET STRAP 1
## Contact-chaining, problem 1:
### Data-mart generation
* Can no longer feasibly build a TNNN data-mart for a set of targets
* Streaming data-mart creation solves the problem
* But... quite complex, loses information and requires stable target set
* Instead use scoring to reduce work...
```
T T
/ \ / \
1 30 1 90 1 50
/ | \ / \ \
2 2 2 2 2 2
```
CONTACT LENS
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ o[REDACTED]
Slide 20
---
SECRET STRAP 1
## Contact-chaining, problem 2:
### Scoring edges
* Represent complex events as a graph, using edge scores:
```
mike@gchq.com
23 Sep 20:05
paul@gchq.com
Time pattern: 86
23 Sep 19:59
dave@gchq.com
1-Aug 13:30
james4@gchq.com
13-July 12:20
offers@big-win.com
PSOUP Correlation: 80
15 Jul 03:34
james3@gchq.com
james1@gchq.com
james2@gchq.com
mike@gchq.com
86
paul@gchq.com
86
29
dave@gchq.com
144
james4@gchq.com
25
offers@big-win.com
80
10
10
10
james1@gchq.com
james2@gchq.com
james3@gchq.com
```
© Crown the Freedom of Inf
Slide 21
---
SECRET STRAP 1
## Contact-chaining, problem 3:
### Scoring nodes
* Using total edge-scores as weights, score nodes to answer specific questions:
```
Most significant 5 non-targets,
connected to your targets:
paul@gchq.com 95
dave@gchq.com 70
james1@gchq.com 50
mike@gchq.com 42
offers@big-win.com 13
mike@gchq.com
42
25
eve@cse.com
86
86
29
144
james4@gchq.com
25
offers@big-win.com
13
Spam score: 86
80 10
james1@gchq.com 50
```
---
SECRET STRAP 1
## Contact-chaining, problem 4:
### Interactive visualisation
* Multiple views – graph, lists of scores, textual information
* Descriptions of scores
* X sent 2 emails to Y, both had no other recipients
* Ability to drill down on detail
```
mike@gchq.com
86
james4@gchq.com
mike@gchq.com
23 Sep 20:05
Time pattern: 86
paul@gchq.com
james4@gchq.com
23 Sep 19:59
```
* Interactive recalculation
* E.g. non-target promoted to be a target
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]
Slide 23
---
SECRET STRAP 1
## Contact-chaining:
### CHART BREAKER
* Prototype software – based on Jung Java graph framework
* Flexible enough to quickly try different scoring techniques
* Uses CONTACT LENS to query Fleximart VLSP data
* Prototype visualisation will be through i2 / MAMBA
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ o[REDACTED]
Slide 24
---
SECRET STRAP 1
# APPLIED RESEARCH
## Contact chaining
Task leader: [REDACTED]
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]
Slide 15
---
SECRET STRAP 1
## Contact-chaining
* Start with known targets (comms identifiers)
* Expand data involving them to find new interesting comms identifiers
* B17 has worked on this for more than 8 years...
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]
Slide 16
Contact-chaining research is all about starting with a known set of targets (usually comms identifiers), and then looking at the data surrounding them to learn more or discover interesting new targets.
B17 have been doing research on this topic for at least 8 years!
---
SECRET STRAP 1
## Contact-chaining:
### Past prototypes
```
SNAPDRAGON
PRIME TIME
CARPAT
MoaG
SORTING FRIENDS
CHART BREAKER
```
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ o[REDACTED]
Slide 17
There have been a whole load of past prototypes
CARPAT
PRIME TIME – telephony analysis timing tool (which I will talk about later in the section of collaborative projects)
SNAPDRAGON – a C2C contact-chaining tool didn't work as intended
MOAG – our contact-chaining tool arising from the London bombings work, which is still being used regularly
SORTING FRIENDS – another telephony contact-chaining tool developed by an integree at NSA.
The contact-chaining task is taking all the lessons from these and will eventually create a new prototype, CHARTBREAKER.
---
SECRET STRAP 1
## Contact-chaining:
### Drivers of current work
* Scoring, not just filtering
* Combining different algorithms
* C2C-specific techniques
* Graph representation – loss of information
* Far more rubbish in C2C
* Growth of events volumes
* Question-based, interactive analysis
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ o[REDACTED]
Slide 18
The main themes of our contact-chaining work are:
-Scoring, not just filtering – CARPAT is a filtering tool, it takes bulk data and filters to get the n/w surrounding targets. However, these networks can still be pretty big, and will get bigger as data volumes continue to grow. We think it is going to be important to be able to score nodes in these networks, so that users can home in on the most important bits
-Another important factor is combining together different algorithms and data sources without confusing the user.
-We also think it is important to develop CC algorithms specifically for CC as our recent experience is that taking an algorithm designed for telephony analysis and just porting to C2C data does not work very well. The reasons for this are:...
-Finally, we think it is important to make contact-chaining tools question-based and interactive. For example, one of the problems with many tools is that they present you will a whole load of configurable options and parameters, which can overwhelm the user. We think we should move to a situation where the user can ask a specific analysis questions of the tool – for example “who are the most strongly connected email addresses to this set of targets” or “what are the links between these two groups of targets?”. The tool presents the best answers it can, allowing the user to interact with the results, drilling-down on detail, in a similar way to IMMINGLE.
---
SECRET STRAP 1
## Combining scores:
### SORTING FRIENDS
* Clustering telephone numbers that have a strong relationship – extrapolating social relationships from events
* Direct communication
* Co-location relationships
* Timing and sequence
* Call backs
* Similar number
* Equipment sharing
* Calling profile similarity
* Etc., etc.
* (currently 12 relationships in total)
* Research prototype – SD can run... operational prototype will be through CHART BREAKER
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ o[REDACTED]
Slide 19
I mentioned combining of scores.
Good example is a telephony prototype, SF, developed between GCHQ and R62 at NSA (by [REDACTED], GCHQ integree there)
SORTING FRIENDS takes a large number of tel call records (e.g we've been running it on the A24 datamart) and reduces it down to all sorts of different relationships between phone numbers. The aim is to try and use lots of different measures of strength of relationship to get closer to the real social relationship.
Describe patterns
Then there is a clustering and visualisation step.
Currently SF is very much a research prototype, and Lorraine in SD can run it for large projects in Ops. Much work needed on it, we don't think it will be a pulled-through on its own, but rather will be part of CHART BREAKER.
---
SECRET STRAP 1
## Contact-chaining, problem 1:
### Data-mart generation
* Can no longer feasibly build a TNNN data-mart for a set of targets
* Streaming data-mart creation solves the problem
* But... quite complex, loses information and requires stable target set
* Instead use scoring to reduce work...
```
T T
/ \ / \
1 30 1 90 1 50
/ | \ / \ \
2 2 2 2 2 2
```
CONTACT LENS
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ o[REDACTED]
Slide 20
---
SECRET STRAP 1
## Contact-chaining, problem 2:
### Scoring edges
* Represent complex events as a graph, using edge scores:
```
mike@gchq.com
23 Sep 20:05
paul@gchq.com
Time pattern: 86
23 Sep 19:59
dave@gchq.com
1-Aug 13:30
james4@gchq.com
13-July 12:20
offers@big-win.com
PSOUP Correlation: 80
15 Jul 03:34
james3@gchq.com
james1@gchq.com
james2@gchq.com
mike@gchq.com
86
paul@gchq.com
86
29
dave@gchq.com
144
james4@gchq.com
25
offers@big-win.com
80
10
10
10
james1@gchq.com
james2@gchq.com
james3@gchq.com
```
© Crown the Freedom of Inf
Slide 21
Mentioned problem of not being able to represent full complexity of C2C events as a graph... Here's an example of a full network... emails and other things such as PSOUP score and timing pattern.
We think the key to this is to collapse down to edges, but with weights on those edges that measure how strong a relationship there is.
So the full network reduces to this graph – blue timing pattern, black sender-to-receiver, red relationship between 2 recipients, and green PSOUP.
Trick is to get the edge weights right – at the moment we're working on algorithms that can do this for all the relationships in email (sender-to-receiver, receiver-to-receiver, event sender-to-sender).
Also looking at timing analysis for C2C.
Where you have multiple types of relationship between people, you can also calculate a total score.
---
SECRET STRAP 1
## Contact-chaining, problem 3:
### Scoring nodes
* Using total edge-scores as weights, score nodes to answer specific questions:
```
Most significant 5 non-targets,
connected to your targets:
paul@gchq.com 95
dave@gchq.com 70
james1@gchq.com 50
mike@gchq.com 42
offers@big-win.com 13
mike@gchq.com
42
25
eve@cse.com
86
86
29
144
james4@gchq.com
25
offers@big-win.com
13
Spam score: 86
80 10
james1@gchq.com 50
```
After you have edge scores, you can use these to score nodes, to answer specific questions.
For example, in CHART BREAKER we have developed a new scoring technique that scores non-targets according to how strong the relationship is to a small group of targets... as shown here. This allows you to rank non-targets by suspiciousness. The algorithm uses a technique that gives targets a weight and sort of iteratively distributes it down the links to connected nodes over a number of iterations, (not just to first circle nodes). We want to develop similar algorithms for other questions. Node scoring algorithms may be independent of whether telephony or C2C due to the edge scoring stage.
This is where you include pre-calculated scores that concern nodes themselves – for example, offers@ has a spam score, which lowers score in answer to the question. The scores allow you to rank nodes according to how well they answer the question, and only show the top 5 scores.
---
SECRET STRAP 1
## Contact-chaining, problem 4:
### Interactive visualisation
* Multiple views – graph, lists of scores, textual information
* Descriptions of scores
* X sent 2 emails to Y, both had no other recipients
* Ability to drill down on detail
```
mike@gchq.com
86
james4@gchq.com
mike@gchq.com
23 Sep 20:05
Time pattern: 86
paul@gchq.com
james4@gchq.com
23 Sep 19:59
```
* Interactive recalculation
* E.g. non-target promoted to be a target
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]
Slide 23
We think the visualisation will be key to using contact-chaining algorithm.
We think it is important for the prototype to be able to show multiple views of the data....
Also, descriptions of "why"
For this reason, we've started some work on using MAMBA to visualise the results of contact chaining algorithms
---
SECRET STRAP 1
## Contact-chaining:
### CHART BREAKER
* Prototype software – based on Jung Java graph framework
* Flexible enough to quickly try different scoring techniques
* Uses CONTACT LENS to query Fleximart VLSP data
* Prototype visualisation will be through i2 / MAMBA
© Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ o[REDACTED]
Slide 24
Mention software will probably need to be rewritten to be more efficient once we know what we the scores are.

View file

@ -0,0 +1,69 @@
{
"title": "Contact chaining",
"type": "REPORTED",
"description": "A secret GCHQ Applied Research presentation describing how the agency's analysts use automated tools to map out a surveillance target's social network — starting with a known email address or phone number, then scoring and ranking every person that target communicates with to identify the most \"suspicious\" or strongly connected contacts, using twelve different relationship types including direct calls, shared locations, timing patterns, equipment sharing and calling profile similarity.",
"long_description": "This is a SECRET STRAP1 GCHQ Applied Research presentation on contact chaining, appearing twice in the document — once as slides only and once with presenter notes providing additional operational context. The presentation describes over eight years of research by GCHQ's B17 unit into automated methods for starting with known surveillance targets and expanding outward through their communications data to discover new targets and map social networks.\n\nThe presentation lists five past prototype tools developed over this period and their planned successor: SNAPDRAGON (a computer-to-computer contact chaining tool described in the presenter notes as not having worked as intended), PRIME TIME (a telephony timing analysis tool), CARPAT (a filtering tool that extracts network data surrounding targets from bulk collection), MoaG (a contact chaining tool arising from the 7 July 2005 London bombings investigation, still in regular operational use at time of presentation), SORTING FRIENDS (a telephony contact chaining tool developed by a GCHQ integree at NSA's R62 unit) and CHART BREAKER (the new prototype intended to consolidate lessons from all previous tools).\n\nSORTING FRIENDS receives the most detailed description. It clusters telephone numbers by extrapolating social relationships from communications events across twelve relationship types: direct communication, co-location relationships, timing and sequence patterns, callbacks, similar numbers, equipment sharing, calling profile similarity and five additional unnamed relationship categories. The presenter notes state it was being run on the A24 datamart and that an analyst in SD could run it for large operational projects. The tool was considered a research prototype whose techniques would eventually be incorporated into CHART BREAKER rather than being operationalised independently.\n\nThe presentation identifies four technical problems driving current research. The first is data mart generation — it was no longer feasible to build traditional TNNN data marts for target sets due to growing data volumes, so CONTACT LENS was developed to use scoring to reduce the analytical workload, querying data from Fleximart VLSP repositories. The second is edge scoring — representing complex communications events as weighted graphs where edge scores measure relationship strength between identifiers. The worked example uses fictional @gchq.com email addresses to demonstrate how different relationship types (sender-to-receiver, timing patterns, PSOUP correlations between recipients) are collapsed into numerical edge weights. PSOUP appears as a named correlation technique producing scores between co-recipients of emails. The third is node scoring — using aggregated edge scores to rank non-target identifiers by suspiciousness relative to known targets. The algorithm described distributes weight from targets iteratively along links across multiple hops, not just to first-circle contacts, producing a ranked list of the most significant non-targets. Pre-calculated node-level scores such as spam scores can be incorporated to lower the ranking of known non-intelligence identifiers. The fourth is interactive visualisation — providing multiple views (graph, score lists, textual descriptions), natural language explanations of why scores were assigned, drill-down capability into underlying events, and interactive recalculation when an analyst promotes a non-target to target status.\n\nCHART BREAKER is described as the consolidation prototype built on the Jung Java graph framework, designed to be flexible enough to rapidly test different scoring techniques. It uses CONTACT LENS to query Fleximart VLSP data and visualises results through i2 (a commercial intelligence analysis tool) and MAMBA (an internal GCHQ visualisation platform). The presenter notes indicate the software would likely need to be rewritten for efficiency once the optimal scoring algorithms were determined.\n\nThe document is significant for revealing the mathematical and algorithmic sophistication of GCHQ's social network analysis research — moving beyond simple contact lists to multi-dimensional relationship scoring that combines direct communications, temporal patterns, geographic co-location, equipment sharing and behavioural similarity into unified suspiciousness rankings. The iterative weight distribution algorithm described for node scoring is conceptually similar to Google's PageRank algorithm, applied to identifying intelligence targets rather than ranking web pages. The London bombings origin of MoaG and the NSA collaboration on SORTING FRIENDS confirm these tools were developed in response to real operational requirements and through active Five Eyes research partnerships.",
"tags": [
"metadata analysis",
"communications network",
"graph analysis",
"node scoring",
"edge scoring",
"interactive visualisation",
"contact chaining",
"GCHQ",
"Applied Research",
"social network analysis",
"telephony analysis",
"co-location",
"timing analysis",
"equipment",
"calling profile similarity",
"London bombings",
"datamart",
"target discovery",
"SIGDEV"
],
"document": {
"creator": "GCHQ",
"date_created": "",
"classification": [
"SECRET",
"STRAP 1"
],
"dissemination": [],
"banner": "SECRET STRAP 1",
"genre": "slides",
"programs": [
"CARPAT",
"PRIME TIME",
"SNAPDRAGON",
"MoaG",
"SORTING FRIENDS",
"CHART BREAKER",
"MAMBA",
"CONTACTLENS",
"FLEXIMART",
"PSOUP",
"IMMINGLE"
],
"countries": [],
"people": []
},
"publication": [
{
"publisher": "The Intercept",
"date_published": "2018-03-01",
"title": "How London's 7/7 Bombings Led to \"Unprecedented\" Surveillance Tactics",
"link": "https://theintercept.com/2018/03/01/london-7-7-bombings-gchq-nsa-surveillance/",
"reporters": [
"Gallagher, Ryan"
],
"contributors": [],
"doc_source": "https://theintercept.com/document/2018/03/01/contact-chaining-gchq/",
"doc_url": "https://s3.documentcloud.org/documents/4390394/B17-TDB-Knowledge-Sharing.pdf"
}
],
"pdf_sha256": "c61c6e8686063529fe4eee5affc327a852d0fe6ba922cc324ce8e96433bf2d39",
"pages": 20
}