first commit

This commit is contained in:
libroot 2026-11-19 04:39:11 +02:00
commit 0d90363116
Signed by: libroot
GPG key ID: 2AC68C029EBE9ACE
6510 changed files with 679146 additions and 0 deletions

View file

@ -0,0 +1,89 @@
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
How to configure category throttling in XKEYSCORE
Category throttling is performed by the selected output plugin, in the xks.config you need to make sure that you are using this plugin to perform forwarding. You will also need to have throttle=yes in the config for the plugin.
Ex:
```
plugin[selected_output] = enable=yes, config_file="throttle=yes
directory=/export/data/xkeyscore/outputs/mailorder_working
add_d706=true pass_fragments=true max_age=900 max_size=5000000"
```
If this was not already set, in order for the changes to take effect you will need to then perform the following commands:
```
xks setup plugins
xks proc saferestart
```
At this point category throttling is now enabled with the default values of 1GB and 1000000000000 sessions. Once either of these limits is reached XKEYSCORE will continue to process data but will no longer forward session that hit on that category. By navigating to the category throttle page you will be able to see the current status of the categories that have been throttled. A category will not appear on this page until it has hit once that day. In the following example the site default was changed to 5MB and 500 sessions:
```
SECRET
XKEYSCORE Welcome oper! Log Out
Home Admin Users Workflow Central Search Results Statistics My Account Help
Navigation Menu
Admin
Processing
Computer Resources
Input Directories
Plugin Configurations
Category Throttle
Age Off Settings
Databases
Search DBs
DB Registration
Casenotation Blacklist
New Fist | Change Site Default | Site Defaults --> Category Quota:5 MB, Sessions Quota:500
SECRET
```
After session start to hit you will see the status displayed on this page:
Derived From: NSA/CSSM 1-52
Dated: 20070108
Declassify On: 20350401
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
```
SECRET
XKEYSCORE Welcome oper! Log Out
Home Admin Users Workflow Central Search Results Statistics My Account Help
Navigation Menu
Admin
Processing
Computer Resources
Input Directories
Plugin Configurations
Category Throttle
Age Off Settings
Databases
Search DBs
DB Registration
Casenotation Blacklist
New Fist | Change Site Default | Site Defaults --> Category Quota:5 MB, Sessions Quota:500
TEST
Add Category Throttle Change Fist Default Reset Fist Default
Category | Daily Category Quota | Quota Remaining | Quota Over | Daily Session Quota | Sessions Remaining | Sessions Over
| -- | --- | ------- | -------- | --- | --- | ---- |
| 4 | 5 MB | 0 bytes | 342.8 KB | 500 | 298 | 0
| 2 | 5 MB | 5 MB | 0 bytes | 500 | 500 | 0
| 1 | 5 MB | 0 bytes | 26.5 MB | 500 | 0 | 2025
SECRET
```
In the example above categories 1 and 4 and currently being throttled, category 1 has reached both the session count and MB limit, while category 4 has only reached the MB limit. Without category throttling category 4 would have forwarded 342.8KB.
You can also check the status of the throttle from the command line using:
mysqls quota_status_current : show quota_limits forwarding status for all categories
mysqls quota_status_yesterday : show quota_limits forwarding status for previous day
mysqls quota_throttled_current : shows top 20 categories for current day. (negative remaining indicates cat was throttled)
SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

View file

@ -0,0 +1,74 @@
{
"title": "How to configure category throttling in XKEYSCORE",
"type": "REPORTED",
"description": "An NSA technical guide explains how to configure \"category throttling\" in XKEYSCORE—a feature that automatically stops forwarding intercepted communications once analyst-defined collection targets hit predetermined data volume or session count limits, allowing the surveillance system to continue processing but prevent overflow from high-volume targets.",
"long_description": "This NSA document provides technical instructions for implementing category throttling within XKEYSCORE surveillance systems, a mechanism designed to manage the volume of intercepted communications forwarded for analysis. The throttling system sets default limits of 1GB of data and one trillion sessions per category, automatically halting forwarding once these thresholds are reached while allowing XKEYSCORE to continue processing intercepted traffic. Administrators can customize these limits—the document shows an example where defaults were reduced to just 5MB and 500 sessions—and monitor throttling status through a web interface that displays real-time quota consumption.\n\nThe guide demonstrates how categories can hit either data volume limits or session count limits independently, with the system tracking which targets have been throttled and by how much. In one example, category 1 exceeded both its session count (exceeding its 500 limit by 2,025 sessions) and data limit (exceeding its 5MB limit by 26.5MB), while category 4 only exceeded its data quota. The document reveals that without throttling, the system would have continued forwarding all matching intercepts regardless of volume, suggesting throttling serves as a resource management tool to prevent high-volume collection targets from overwhelming storage or analysis capacity. The system resets daily and provides command-line tools for monitoring which surveillance categories are being throttled.",
"tags": [
"category throttling",
"data volume",
"surveillance quotas",
"SIGINT management",
"collection limitations",
"session throttling",
"traffic redirection",
"resource management",
"automation",
"plugin configuration",
"quota management",
"intercept volume control",
"target categories",
"database",
"collection overflow",
"storage management",
"session counting",
"data forwarding",
"surveillance infrastructure",
"technical configuration",
"collection capabilities",
"automated filtering",
"real-time surveillance",
"command-line interface",
"web interface",
"daily quotas",
"session limits",
"intelligence processing",
"target management"
],
"document": {
"creator": "NSA",
"date_created": "",
"classification": [
"SECRET",
"COMINT"
],
"dissemination": [
"REL TO USA, AUS, CAN, GBR, NZL"
],
"banner": "SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL",
"genre": "manual",
"programs": [
"XKEYSCORE",
"MAILORDER"
],
"countries": [],
"people": []
},
"publication": [
{
"publisher": "The Intercept",
"date_published": "2015-07-01",
"title": "XKEYSCORE: NSA’s Google for the World’s Private Communications",
"link": "https://theintercept.com/2015/07/01/nsas-google-worlds-private-communications/",
"reporters": [
"Marquis-Boire, Morgan",
"Greenwald, Glenn",
"Lee, Micah"
],
"contributors": [],
"doc_source": "https://theintercept.com/document/2015/07/01/category-throttling/",
"doc_url": "https://s3.documentcloud.org/documents/2115993/category-throttling.pdf"
}
],
"pdf_sha256": "c2161a397e8edfc0d6d90ff29dd62feacc7d70ebd51c6cd4000f21394c1aef4b",
"pages": 2
}