first commit

This commit is contained in:
libroot 2026-11-19 04:39:11 +02:00
commit 0d90363116
Signed by: libroot
GPG key ID: 2AC68C029EBE9ACE
6510 changed files with 679146 additions and 0 deletions

View file

@ -0,0 +1,73 @@
{
"title": "User-agents: Why and How and How to",
"type": "REPORTED",
"description": "A TOP SECRET NSA training presentation from 2010 teaching analysts how to use browser user-agent strings — the identifying information your web browser automatically sends to every website you visit — to track surveillance targets across the internet, link their known email activity to their anonymous web browsing, and identify opportunities for hacking their computers based on their operating system and software versions.",
"long_description": "This is a TOP SECRET//COMINT//REL TO USA, FVEY NSA training presentation from S2I61 dated July 2010. The presentation teaches analysts how to exploit HTTP user-agent strings for surveillance purposes. User-agent strings are automatically transmitted by web browsers to every website visited, containing the browser type and version, operating system, security settings, installed software frameworks and language preferences. The presentation explains that user-agents can link a target's \"selected\" activity (communications already being collected under tasking) to their \"unselected\" web browsing (untasked internet activity captured in bulk), create tentative links between targets sharing the same user-agent configuration, and identify CNE (hacking) opportunities by revealing a target's operating system version and software configuration. The presentation includes a lighthearted history of browser user-agent spoofing dating back to Internet Explorer pretending to be Mozilla/Netscape to receive frames-enabled web content.\n\nThe practical section demonstrates querying user-agents in XKeyscore and PINWALE. The key analytical judgment taught is assessing user-agent reliability — when an IP address shows only two user-agent strings with low traffic volume, the correlation between user-agent and target is \"probably reliable.\" When the same IP shows many different user-agent variations (the example shows 17 distinct strings including variations in .NET CLR versions, Google Toolbar versions and InfoPath installations), reliability \"plummets\" and cookie/TDI correlation becomes necessary. The presentation warns against assuming complex user-agents are unique — a standard Chrome 6 on Windows 7 string applies to every user with that configuration. Mobile user-agents are highlighted as particularly valuable because they typically include the phone model, creating \"IMEI correlation opportunities\" for linking web activity to specific physical handsets.\n\nThe presentation concludes with the operational caveat that user-agents can be manipulated or removed entirely — via Firefox plugins, Opera browser settings or the TOR Button — allowing targets to present a different user-agent for each browsing session. The final slides advertise S2I61's broader services including help with XKeyscore fingerprint development, target technology analysis and collateral description development, positioning the unit as an internal SIGDEV technical support team.",
"tags": [
"User-Agent",
"HTTP",
"browser fingerprinting",
"TDI",
"cookies",
"Computer Network Exploitation",
"operating system",
"Windows",
"Firefox",
"Chrome",
"Safari",
"Opera",
"Internet Explorer",
"mobile user-agent",
"iPhone",
"Android",
"BlackBerry",
"IMEI",
"Gecko",
"AppleWebKit",
"Presto",
"encryption",
"USSID-18",
"compliance",
"web browsing",
"SIGDEV",
"target tracking"
],
"document": {
"creator": "NSA",
"date_created": "2010-07-01",
"classification": [
"TOP SECRET",
"COMINT"
],
"dissemination": [
"REL TO USA, FVEY"
],
"banner": "TOP SECRET//COMINT//REL TO USA, FVEY",
"genre": "slides",
"programs": [
"XKEYSCORE",
"PINWALE",
"AGILITY"
],
"countries": [],
"people": []
},
"publication": [
{
"publisher": "The Intercept",
"date_published": "2015-07-01",
"title": "XKEYSCORE: NSA’s Google for the World’s Private Communications",
"link": "https://theintercept.com/2015/07/01/nsas-google-worlds-private-communications/",
"reporters": [
"Marquis-Boire, Morgan",
"Greenwald, Glenn",
"Lee, Micah"
],
"contributors": [],
"doc_source": "https://theintercept.com/document/2015/07/01/user-agents/",
"doc_url": "https://s3.documentcloud.org/documents/2116232/user-agents.pdf"
}
],
"pdf_sha256": "0177d72ec9aa3f502ed8c042c1c778fd558894c8757f18610bf84ca07a63790c",
"pages": 42
}

View file

@ -0,0 +1,676 @@
TOP SECRET//COMINT//REL TO USA, FVEY
# (S//SI//REL) User-agents: Why and How and How to
NATIONAL SECURITY AGENCY
UNITED STATES OF AMERICA
The overall classification of this briefing to TOP SECRET//COMINT//REL TO USA, FVEY
[REDACTED] – S2I61
July 2010
Derived From
NSA/CSSM 1-52
Dated 20070108
Declassify on: 20320108
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Agenda
* WHY: Learn why we care about user agents (UAs)
* HOW: Learn how to read a user-agent
* HOW TO: (get it?) Learn how to use user-agents in our tools
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## WHY?
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## What is a User-Agent?
A user-agent is a string which lets websites
know your:
* type of web-browser or application
* Operating System
* Security settings or permissions
* Versions of relevant programs (media, java, etc.)
* Etc. (Language settings, ad-ware)
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Why would I want to give a website that?
* Compatibility
* Specific Website Features
* Security permissions
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## User-Agents Can
* Link a target's “selected” activity to their unselected web-browsing
* Create a tentative link between targets that have the same user-agent
* Identify CNE opportunities
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## BUT...
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## User-Agents Also
* Can vary from very unique to extremely common
* Change with software updates
* Only identify the web-browser
* 2 web-browsers = 2 user-agents
* Can't be trusted...
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Who started this nonsense?
A History Lesson
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## The Great Browser Wars
Back...in the 20th Century!
Before Now but After What is Below
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Remember Frames?
* Netscape's new fancy web-browser support them!
* The original “web-browser” Mosaic did not
* And so began browser “sniffing”
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Worthless Trivia!
* Mosaic was the first web-browser to embed images with text
* It supported FTP, Usenet, and Gopher!
* Its web-browser competitors at the time were Erwise and ViolaWWW
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## The Great Internet Explorer Hoax
Ever wonder why so many user agents start with “Mozilla” but aren't Firefox?
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Internet Explorer fools us all
* When Internet Explorer was released it did frames too!
* But since its user-agent didn't say so, no websites would send their super cool frames version to the IE users
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## What to do?
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## LIE, of course!
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Internet Explorer starts to spoof
* Internet Explorer changed its user agent starting their user agent with Mozilla/1.22
* Mozilla/1.22 (compatible; MSIE 2.0; Windows 95)
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## And it continues to this very day...
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## The How...
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Three Basic Pieces of the UA
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
Mozilla/1.22 (compatible; MSIE 2.0; Windows 95)
Mozilla/1.22
Part 1: The Netscape Historical Token
* Appears in primarily Mozilla Firefox, Google Chrome, and MSIE browsers
* Modern Version: Mozilla 4.0 or Mozilla 5.0
* Does not indicate a target uses “Mozilla Firefox”
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
Mozilla/1.22 (compatible; MSIE 2.0; Windows 95)
MSIE 2.0
Part 2: The Web Browser Identifier
* Appears in generally all user-agents
* Not always in the same place, but usually self explanatory
* Opera X.X = Opera
* Firefox X.X = Firefox
* Safari X.X= Safari
* Chrome X.X Safari X.X = Google Chrome
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
Mozilla/1.22 (compatible; MSIE 2.0; Windows 95)
Windows 95
Part 3: The Operating System Token
* Appears in basically all HTTP user-agent strings
* Examples:
* Windows NT 6.1 = Windows Version 7
* Windows NT 6.0 = Windows Vista
* Windows NT 5.2 = Windows XP 64bit
* Windows NT 5.1 = Windows XP
* Windows NT 4.0 actually equals Windows NT 4.0
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Other operating systems UAs
* Mobile
* MAC OS X
* Linux
* Linux i686
* Free BSD
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Game Consoles
* Opera/9.30 (Nintendo Wii; U; 2047-7; en)
* Mozilla/5.0 (Playstation 3; 2.00)
* PSP (PlayStation Portable); 2.00
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Mobile User Agents
* Usually self-explanatory
* Iphone
* Ipad
* Blackberry
* Android
* Mobile user agents also usually give you the phone model (Read: IMEI correlation opportunities)
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Ever wonder what that was?
* Gecko: a rendering engine used by Firefox and others
* AppleWebkit: Apple's version of KHTML rendering engine used in Safari and Chrome most commonly
* Presto : the “core” of the Opera platform suite
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Ever Wonder Contd.
* .NET CLR is the .NET Framework version
* SV1 is an artifact created by MSIE 6.0 to make its security better
* Win64 can indicate that the system is running a 64 bit processor
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Ever Wonder... One more
Many web browsers will also have an “encryption strength” marker
U = USA (128 bit encryption)
I = International (40 bit encryption)
N = No encryption (Woo!)
Most Browsers nowadays come with a U
Since the USG no longer requires encryption changes for international usage.
TOP SECRET//COMINT//REL TO USA, FVEY
--
TOP SECRET//COMINT//REL TO USA, FVEY
## Your target's user-agents can shorten and lengthen!
* Each website may require different information
* Longer user-agents may have various rendering engine, java versions, and language settings
* If you see a shortened or longer version of a UA close to your targets logins. Check it out! Carefully...
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## The How To...
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Querying in Xkeyscore
```
Navigation Menu
* Classic A-M
* Alert
* BlackBerry
* Call Logs
* Category DNI
* Cellular DNI
* Cisco Passwords
* Clarent
* Crypt Scanner
* DNS
* Document Metadata
* Document Tagging
* Email Addresses
* Extracted Files
* Full Log DNI
* Gen Info
* HTTP Activity
* IKE Parser
```
```
Browser: Mozilla 5.0*
```
Remember:
Since Xkeyscore no longer supports leading wildcards you need to be specific with your User-Agent
A User-agent alone is not a strong query
Time Frame, Active IP,
Country, etc. all will help make
your query compliant
```
Browser
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.99 Safari/533.4
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Uniqueness – When to use a User-Agent
```
Reports
VIEW
FILTERS
Browser
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
Geo Info
Sort Ascending
Sort Descending
Filters
Color By
Group By
Histogram
Pivot Data
Histogram Grid
Show/Hide...
AutoFit Column Width
```
Results of an IP-based search for a target
```
Histogram Grid
Page 1 of 1
Clear Selection Export
Filter Browser
Count
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
6
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
6
```
Low Number of users and traffic volume as well as only 2 User-agents
Verdict: Probably reliable
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## More User = More Problems
| Browser | Count |
|-------- |-------|
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) | 6 |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; | 1 |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; | 1 |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB0.0; .NET CLR 2.0.50727; AskTB5.6) | 1 |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB6.5) | 1 |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB6.5; .NET CLR 2.0.50727) | 1 |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB6.5; InfoPath.2) | 2 |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB6.5; InfoPath.2; .NET CLR 2.0.50727; .NET | 1 |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.1; .NET CLR 2.0.50727; .NET CLR 3.0. | 1 |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322) | 1 |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) | 6 |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322) | 1 |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; FunWebProducts; AskTbPTV/5.8.0.12304) | 1 |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6; .NET CLR 1.1.4322) | 1 |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; IEMB3; IEMB3) | 1 |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727 | 1 |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; InfoPath.2; .NET CLF | 2 |
Reliability of correlating your target's browser to his web-activity plummets due to User-agent variations and the threat of multiple users.
At this point, cookie and TDI correlation are necessary. User-agents can not be used by themselves.
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Common Sense Helps
Never assume because a User-agent is complicated that it is unique
* Example:
Mozilla 5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.3(KHTML, like Gecko) Chrome/6.0.464.0 Safari/534.3
This is the standard user agent for EVERYONE with an updated Chrome browser using Windows 7.
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Xkeyscore Storage
Push to Pinwale or Archive Results
```
Download Session Mode: S
AGILITY-RealTime
Agility
DNI Presenter
PINWALE
PINWALE and Decrypt
Download SOTF Session
Download Session
Download D124 Session
finished
Archive Results
finished
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Pinwale
Fields to add to Metadata View
```
AppProc Active User
User Agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) ...
```
Generally speaking, the User Agent listed belongs to the selector in the Active User column (if populated)
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Add a User-Agent to a compliant query
Fielded Search Form
```
1. Put in the User Agent that is being search for into the CONTENT\_META field under the fielded search form.
KEYWORDS and
cs=ISO8859_1 ( ( ("Opera Mini") ) )
CONTENT_META
Opera Mini
between tags UserAgent and
```
Smart Form/Native Query
```
Native, cs=ISO8859_1 ( (content_meta=(((("Opera Mini")) between \<UserAgent and \/UserAgent))))
(content_meta=(((("Opera Mini")) between \<UserAgent and \/UserAgent )))
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Not sure the specific User-Agent?
```
1. Add the following syntax to your query:
\TERM \<yourfield here (E.G. \TERM \<useragent )
2. Apply Native to the field
3. Apply the content_meta document zone to the field.
Free Form Query Smart Form
cs=ISO8859_1(((["selectorl"]))
selector1.
or
cs=ISO8859_1( ("selector2")))
selector2
or
) and
CONTENT_META:Native, cs=ISO8859_1 [[TERM \<useragent ]]
TERM \<useragent
or or and
(TS//SI//REL) This query basically ensures that a certain field exists in each result thus removing all the content not relevant to your query.
```
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## User-Agent Manipulation
* The best for last
* User-agents can completely be changed or not included by the user!
* By Firefox Plugin
* By Browser settings (Opera)
* Outside programs (TOR Button)
* These programs allow users to have a different user agent for each session!
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Questions??
* Contact Info: [REDACTED]@nsa.ic.gov
* Website on the High-Side:
* [REDACTED]
* [REDACTED]
* [REDACTED]
* [REDACTED]
* Lots of great stuff in open source as well!
http://www-archive.mozilla.org/build/user-agents-strings.html
TOP SECRET//COMINT//REL TO USA, FVEY
---
TOP SECRET//COMINT//REL TO USA, FVEY
## Got a Tech Problem?
Have a random SIGDEV question?
Need help with a target using new tech to communicate?
Need help developing an accurate collateral description of a technology?
Want help developing Xkeyscore fingerprints for a weird target behavior?
Let us know : DL S2I61_all
TOP SECRET//COMINT//REL TO USA, FVEY