96 KiB
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
XKEYSCORE System Administration
December 2012
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Introduction
2
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Lesson Objectives
- Introduction to XKEYSCORE
- Purpose and Capabilities
- Data Flow
- What is a Cluster?
- XKEYSCORE Databases
3
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Introduction
- XKEYSCORE performs filtering and selection to enable analysts to quickly find information they need based on what they already know.
- XKEYSCORE also performs SIGDEV functions such as target development to allow analysts to discover new sources of information.
4
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Introduction
- XKEYSCORE processes data at field sites, where it is collected, and allows analysts from all over the world to query it.
- At field sites, the XKEYSCORE software can run in clusters of few or many servers, giving it the ability to scale in both processing power and storage.
- All processing is plugin or fingerprint based, which allows new capabilities to be quickly deployed to support operational needs.
5
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Purpose and Capabilities
- XKEYSCORE is a Computer to Computer (C2C) exploitation system.
- It is a fully distributed processing and query system.
- XKEYSCORE can run on multiple servers.
- Plugin and fingerprint architecture allows new capabilities to be quickly deployed.
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
6
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Hardware
- XKEYSCORE is typically installed with Red Hat AS5u8 operating system. The suggested disk set up is:
- Set up separate partitions for
/ (root),/var,/tmp, and/export/data
- Set up separate partitions for
- XKEYSCORE clusters can be composed of three different functionalities, which are:
- One host acts as the web server/user interface, etc...
- Another host normally runs as the real-time processing unit
- Other host acts as the search or query system.
- Hybrid system can perform multiple roles on one server, which enables efficient registration.
process_data_parent1 query_proc
7
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Data Flow (High-level)
- The backend is where the raw data for XKEYSCORE is processed; that is, we receive information from our sources (e.g. WEALTHYCLUSTER2), process it, and store it into a database.
[sessions] -> [processing engine] -> (database) <--> (user queries)
Session
phone numbers
email addresses
log ins
user activity
Database
metadata tables
full log
QUERY
8
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Data Flow - XKS Cluster
- A cluster is comprised of one master server and one or more slaves.
- All slaves in a cluster have their own copy of configurations (
/opt/xkeyscore/config) files via thexks rsync push_configcronjob.
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
9
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Data Flow - Databases
- There are two types of databases on an XKEYSCORE system: insert (i0) and query (q0)
D124
sessions
file_input_proc
process_data0
process_data1
SOTF
sessions
sotf_input_proc
process_dataN
xks_meta_ingester
i0
register_metadata_tables
q0
NOTE: sotf_input_proc is now called, sotf_dist
process_dataN's are now called, process_data_parent
10
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Data Flow - Databases
file_input_procandsotf_disttake in sessions from the front-end and load balances them across multipleprocess_data_parent's.process_data_parentis responsible for processing sessions and extracting metadataxks_meta_ingestertakes the metadata from theprocess_data_parent's and writes it to the insert database,i0register_metadata_tablestakes completed insert tables, indexes them, and moves them to the query database,q0
11
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
XKEYSCORE Architecture
12
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Lesson Objectives
- Operating System Services
- HTTPD
- MYSQL
- NFS
- AUTOFS
- Mount Points
/xks_data
- Directory Structure
13
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Operating System Services
- XKEYSCORE is typically installed on servers running Red Hat 5u8 operating system.
- This section discusses common operating system services used during XKEYSCORE operation.
14
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
HTTPD
- The http daemon is needed for the web-based GUI, viewing content, and is required on all servers.
- The master server is the web server and the slaves retrieve content through HTTPS.
15
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
MySQL Daemon
- The mysql daemon is a SQL-based database server for processing, querying, and is needed for the XKEYSCORE GUI.
- It is required on all servers for administration, processing, and querying metadata in databases.
16
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
NFS
- Mounting a directory uses the NFS service.
- NFS allows file systems that physically reside on one computer to be shared by other computers on the network.
- The machine with the hardware containing the directory must allow the hardware to be made available to other machines.
- Required on all computers for clustering.
17
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
NFS
/etc/exports/export/data/xkeyscoremaster(rw) slave(rw)/opt/xkeyscore/config/loadserver\*(rw)
18
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
AUTOFS
- Computers requiring shared access to the
/export/data/xkeyscoredirectory must be told where to find the directory.- This is accomplished via automounting.
- The autofs daemon listens for computers trying to connect to the directories, or mounts, that it is responsible for.
- The mounts are dropped after a time out, but autofs remounts the drive when drives need to be accessed.
19
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Automounting
- For a clustered XKEYSCORE, automounts must be set up on all of the computers in the cluster.
auto.masterandauto.datafiles in the/etcdirectory must be edited or created.- When finished, the mounted directories on the remote machines can be accessed.
- The
operaccount should have full read/write permissions on all shared drives.
20
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Mount Points
auto.master– designates mount points on the local computer and the directory to mount on the remote server.- Example:
/xks_data/etc/auto.data--timeout=60
- Example:
auto.data– enables all servers to see the/export/data/xkeyscoredirectory on other machines and locate databases, archived, data, and MAILORDER directory.- Example:
xks1 -rw,soft,intr,tcp xks1:/export/data/xkeyscorexks2 -rw,soft,intr,tcp xks2:/export/data/xkeyscore
- Example:
21
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Directory Structure
/opt/xkeyscore/– contains all of the XKEYSCORE software. Software includes the GUI, processing, scripts, and configurations.bashrc– XKEYSCORE environment variables file.beacon/– contains the beacon perl script (shm_beacon.pl) and a link to the beacon configuration file (shm_beacon.config).bin.shells/andbin.shells/sysadmin– contains miscellaneous bash, python, and C shell scripts.build/– contains libraries and plug-ins.install/– contains installation scripts.
22
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Directory Structure
/opt/xkeyscore/config/– consists of sub-directories and each contain configuration files for building and running XKEYSCORE.crontab/– contains the master and slave crontab file.dictionaries/– contains the dictionary files for the filtering, selection, TRAFFICTHIEF, CADENCE, fist tables, and any other local dictionaries.misc/– contains miscellaneous per-plug-in configuration files, (i.e.sotf_input_proc.xml).plugins/– contains event handler configuration files for each of the plugins (default.xml).www/– contains web configuration files andxscore.cfg.SERVICE/– contains the config files for all the services needed by XKEYSCORE (httpd, php, mysqld, etc.)
23
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Directory Structure
/opt/xkeyscore/www/– contains the contents of the web front end.docs/– contains documents viewable through the XKS GUI.html/– contains web pages and scripts that are not on the secure server.secured/– contains web pages and scripts that are on the secure server including:crons/- location of cron job scripts
src/– contains source code for the XKS GUI.
24
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Directory Structure
/export/data/xkeyscore/– is used for both internal databases and metadata archive databases, input, output, and archiving of data.archives/– (optional) destination for processed contentinputs/– (optional) used for file based inputmysql/– location of the MySQL database consisting of admin, insert, and query databases.outputs/– (optional) contain the following sub-directories:mailorder/- pickup pointmailorder_working/- file creation point before being moved to mailorder/
25
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Directory Structure
/xks_data/- logical mount point for all other XKEYSCORE (including itself)/export/data/xkeyscore.<hostname>/- mount point for the hostname's local directory/export/data/xkeyscore(referenced by host name).- All servers must export their
/export/data/xkeyscoredirectory and mount this on the/<hostname>directory for each hostname of each machine, including itself.
- All servers must export their
26
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
XKEYSCORE GUI
27
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Lesson Objectives
- Accessing the GUI
- Exiting a Session
- Main Menu Bar
- MyXKS
- Admin
- Computer Resources Option
- Start and Stop Processing
- Run a Process Manually
- Users
- Search
- Workflow Central
- Results
- Fingerprints
28
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Accessing the GUI
- In the address field of a web browser, type
https://<master hostname or IP address>. - PKI's or a UserID and password are required. After successfully launching a new session, the XKEYSCORE WELCOME window appears.
- Note: Compatible web browsers for XKEYSCORE version 1.5 are:
- Internet Explorer is not supported
- Firefox/3.0.\* and above
- Note: Compatible web browsers for XKEYSCORE version 1.5 are:
29
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Accessing the GUI
The system is accredited for USSID 18 and Human Rights Act compliance
TOP SECRET//SI//REL TO USA, AUS, CAN, GBR, NZL//20320108
Home MyXKS Admin Users Search Workflow Central Results Fingerprints Statistics Map Help
Navigation Filter
Home
Last Search Types
Full Log
LinkSummaryMapReduc...
Category (DNI)
Alerts
Email Addresses
MyXKS
Full Log DNI
HTTP Activity
My Fingerprints
My Overflows
My Recent Results
Profile
Admin
Processing
Computer Resources
Input Directories
Category Throttle
Databases
Search DBs
DB Registration
Utilities
Casenotation
Reload Config Files
News
Ip Summary Table
Crashlogger
Startup
Query
Users
User Accounts
Clearences
Privileges
Send Email
Users Online
Search
Search Wizard
Anonymizer
Tor Onion Address Su...
Tor Routers
Tor Software Downloa
tor Users Survey
Web Anonymizers
Botnet
XKEYSCORE
Suggested achievements
Created Workflow
Learn How
Histogram
Learn How
Exporting Meta data
Learn How
HUMAN RIGHTS ACT,
USSID 18 AND USSID 9
All (SYSTEM) queries require a justification to
ensure Human Rights Act (HRA), USSID 10 and
USSID 9 compliance. Please enter information
as prompted by the query interface. An audit
trail has been established and will be searched
as part of Menwith Hill Station's response to any
complaint brought under HRA and as part of the
USSID 10 and USSID 9 process.
Please note that SENSITIVE TARGETING
APPROVAL (STA) is required for HRA before
submitting any query which includes terms
specific to a person or company (eg name,
address, identity details such as
communications address, passport/bank
account number) who EITHER (a) is defined as
a UK, British Dependent Territory (BDT) or
Second Party "person" or (b) is located in the
UK, or a BDT or Second Party country. STA is
also required for wildcard pulls that are inevitably
going to retrieve a substantial proportion of such
entries (e.g. wildcarding on a UK city code). Full
legal guidance is available from the HRA
Compliance Officer at Menwith Hill Station.
30
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Main Menu Options
- The main menu bar across the top of the window has menus that, when selected, each has additional options available in a drop down menu form.
Home MyXKS Admin Users Search Workflow Central Results Fingerprints Tagging Reports Statistics Tasking Map Help
31
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Main Menu Options
| OPTION | DESCRIPTION |
|---|---|
| Home | Returns to the main page. |
| MyXKS | Can edit user settings, disable/enable access to databases, edit a search form search setting, and restore default settings. |
| Admin | Computer resources, Input Directories, Category Throttle, Search DBs, and DB Registration settings. |
| Users | Contains User Accounts, Clearances, Privileges, Send Email, Users Online, My Auditees, My Audit Logs, and All Audit Logs. |
| Search | Provides different search query forms, such as email addresses, category, full log, and user activity. |
| Workflow Central | Request, modify, and view standing queries that will execute at a specified time or interval. |
| Results | Can search personal searches by date time, query type, query name, output table, and user. |
| Fingerprints | Fingerprint builder and viewer. |
| Map | Brings up Google Earth |
| Help | Help Documentation, XK Forum, Account Maintenance, and About XKEYSCORE |
32
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Admin Menu
| SUB-MENU OPTION | DESCRIPTION |
|---|---|
| Computer Resources | Allows for process configuration and management. |
| Input Directories | Contains the configuration for file-based input directories. |
| Category Throttle | Edit CADENCE quota limits by category and/or fist table. |
| Search DBs | Configuration for query databases which are queried when a search is submitted. |
| DB Registration | Contains the mapping from insert databases to query database. |
| News | Add, modify, delete mandatory and home page News. |
33
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Computer Resources
- The Processing->Computer Resources option from the ADMIN menu allows control of the entire daemon-styled, or continuously running, processes for XKEYSCORE.
- Processes appears in a table following the convention:
<PROC_HOST><PROGRAM_NAME><PROGRAM_ARGUMENTS>`
xkey01 process_data_parent
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
34
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Computer Resources
35
Computer Resource Window Process Table
TOP SECRET//SI//REL TO USA, A
Home MyXKS Admin Users Search Workflow Central Results Fingerprints Statistics Map Help
Navigation Filter
Processing
Computer Resources
Input Directories
Category Throttle
Databases
Search DBs
DB Registration
Utilities
Casenotation Blacklist
Reload Config Files
News
Ip Summary Table
Crashlogger
Startup
Query Profiler
Help
Color Key
STOPPED STOPPING STARTING RUNNING WON'T START UP
STOPPED? APP LAUNCHER STOPPED
STOPPING? APP LAUNCHER STOPPED
STARTING? APP LAUNCHER STOPPED
RUNNING? APP LAUNCHER STOPPED
WON'T START UP? APP LAUNCHER STOPPED
Computer Resources
Help Add Actions App Launcher is Running
| Actions | Proc Host | Program Name | Program Arguments | Program PID | Commanded Status | Status | Datetime Started | Datetime Stopped |
|---|---|---|---|---|---|---|---|---|
| STOP | tlxksvr01 | GUId | 7607 | RUN | RUN | 2012-12-03 16:44:49.0 | 2012-12-03 16:44:49.0 | |
| STOP | tlxksvr01 | query_proc | 30965 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:32:47.0 | |
| STOP | tlxksvr01 | check_mailorder_site.php | 31019 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:32:47.0 | |
| STOP | tlxksvr01 | xks_meta_ingester | 31051 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:32:47.0 | |
| STOP | tlxksvr01 | clickstreamService.sh | 31053 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:32:47.0 | |
| STOP | tlxksvr01 | query_dispatch | 1311 | RUN | RUN | 2012-12-03 21:08:09.0 | 2012-12-03 21:08:07.0 | |
| STOP | tlxksvr01 | file_input_proc | 31104 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:32:47.0 | |
| STOP | tlxksvr01 | xks_system_monitor | 13986 | RUN | RUN | 2012-11-27 21:01:13.0 | 2012-11-27 21:01:13.0 | |
| STOP | tlxksvr01 | sotftoo124server | 31108 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:32:47.0 | |
| STOP | tlxksvr01 | tomcat.sh | 31124 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:32:47.0 | |
| STOP | tlxksvr01 | cadence_tasking_proc | --myfdi XYD --pddg IE --digraph X5 | 31136 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:32:47.0 |
| STOP | tlxksvr01 | xks_server_stats | 31138 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:32:47.0 | |
| STOP | tlxksvr01 | mailorder_proc | --copydir /export/data/xkeyscore/out... | 31140 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:32:47.0 |
| STOP | tlxksvr01 | register_metadata_tables | --loglevel error | 31143 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:33:50.0 |
35
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Computer Resources
- The
xks_app_launcherprocess runs on all servers from theinittab. - It tells the computer which program to run by looking at its tasking host.
/opt/xkeyscore/config/www/xscore.cfg- The config file specifying the location of the tasking database.
- Processes can be stopped, started, edited, or deleted from the Computer Resources window.
36
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Computer Resources
- Add a new process – click Add
- Edit a process – click Stop in the ACTION column, then click Edit.
- Delete process – click Stop in the ACTION column, then click Delete.
- Stop the App Launcher – disables the
xks_app_launcheron every host.
37
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Resources – Color Conventions
- Visual cues in the form of colors are used to help identify activities performed by XKEYSCORE and serve as status indicators for monitoring purposes.
- Red – indicates processes have been stopped
- Green – indicates processes are running
- Yellow – indicates processes are starting
- Orange – indicates processes are being stopped
- White – indicates processes won’t start
- Visual cues are also available in the COMMANDED STATUS and STATUS columns of the table.
38
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Resources – Start/Stop Processing
- It may be necessary to stop or start processes for troubleshooting or for a graceful server restart.
- Individual processes and programs – Click Stop in the ACTION column. To start it, click Run.
- To stop all individual programs, select ACTIONS->Start/Stop Resources. Enter the program name in PROGRAMS field, then click OK.
- Can use
xks procactions and commands to do the same function
39
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Resources – Start/Stop Processing
- All Processing - select START/STOP Resources from the ACTIONS drop-down menu, leave the PROGRAMS and ON HOSTS fields to their defaults, click OK.
- Specifying programs or hosts – select STOP or START, enter a wildcard expression such as
*or!in the PROGRAMS or HOSTS field, and click OK.- Example:
process*
- Example:
- Alternatively, in a terminal window can run:
xks proc stop process*
40
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Run a Process Manually
- It may be necessary to run a process manually for troubleshooting purposes. To run a process manually:
- Launch the GUI and log on as
operoradmin. - Click ADMIN > Processing > Computer Resources
- Click Stop in the ACTION column for the process.
- Open a terminal window and
sshto the host running the process, as the useroper. - Type
ps -ef | grep <process name>to verify that the process is stopped. - Type
<program name><program argument>
- Launch the GUI and log on as
- Example:
query_proc <program arguments, if any> --loglevel debug
41
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Users Menu
Home MyXKS Admin
Navigation Filter
User Accounts
Clearances
Privileges
Send Email
Users Online
- This menu is only accessible to users with system administration privileges.
- An SA can add/modify user accounts, add groups, clearance levels, privileges, and email users from this menu.
42
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
MyXKS
- From the main menu bar, click
MyXKSto view your profile, accesses, privileges, auditors, settings, fingerprints, workflows, and recent results. - Right click on any search form name to add a shortcut for that search form.
Navigation Filter
Full Log DNI
HTTP Activity
My Fingerprints
My Workflows
My Recent Results
Profile
Full Log DNI
HTTP Activity
My Fingerprints
My Workflows
My Recent Results
Profile
43
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Search Menu
- From the main menu bar, click SEARCH. Menu options display in the vertical pane on the left.
Navigation Filter
Search Wizard
Anonymizer
Tor Onion Address Survey
Tor Routers
Tor Software Downloads
Tor User Survey
Web Anonymizers
Botnet
Quantumbot
QUANTUMBOT Table
ROGUESTITCH
ROGUESTITCH
Rs Stolen Creds
CNE
ANIMALHOUSE Access Logs
Active Passive Map
CNE Triage V2
Connection Information
Cottonchainsaw
EYou Mail Server Logs
Email Addresses
Endpoint Attached Media
Endpoint File Hashes
Endpoint Related Sessions
Endpoint Stats Endpoint Reenable
Endpoint Stats Endpoint Related Counts
Extracted Files
Eyou Mail Logs
Fourth Party Passive Map
Full Log DNI
HTTP Request Cookies
IE Cookies
Kaspersky PSP
Key Loggers
Keylogger
MP TRII Access Logs
Machine Information
Machine Information
Network Information
Network Information
Norton PSP
Odnoklassniki
Odnoklassniki
Passwords in RDP
Registry
Registry
UserAssist
Registry Information
Router Survey
SCARFACE Network Information
UtmaIDs
Web Servers
Windows User Security Identifier (SID)
vBulletin Sessions
CT
Archive Known Password
Atabmail logs
Ebel logs
Email
Ffu Mfnt
MS2 User Activity
Ms2 Keys and User Activity
Opera Mini Headers
Parcel Tracking
RAR Known Password
User Activity and persistent evercookies
Yemen mobile IPs
Zamensis
Zamensis and extracted files
Zamensis and full-text
vBulletin Sessions
44
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Search Menu
- When choosing a plugin type from the menu options, the only data searched is the data that was identified as a hit when the plugin was processed.
| SUB-MENU OPTION | DESCRIPTION |
|---|---|
| Category DNI | Searches dictionary category hits. |
| Full Log DNI | Searches all sessions received by XKEYSCORE. |
| User Activity | Enables a user to search by a user’s activity. Example: a user can find a hotmail user’s msnMailToken |
45
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Search Menu
- All searches are conducted on database tables where the results of the XKEYSCORE engine are stored.
- Each row of a database table contains values from an individual session that was identified as a hit by XKEYSCORE when that plugin or microplugin processed the session.
- Each search type query is related to a plugin or microplugin, which performs the metadata extraction.
46
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Search Menu - Details
- Search details can be accessed from the Search status window by clicking Details.
- CURRENT SEARCH DETAILS window displays and allows the user to watch a query run through the appropriate databases.
- RESULTS link in the main menu bar can be used to display a list of all previous search results.
- Queries operate in parallel on each host.
47
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Search Menu - Details Window
Query Status : oper_0489153001249071967_1
Pause Hide Finished Show Query
Status
tlxksvr01:qsummary/new
tlxksvr02:qo/new
tlxksvr02:qsummary/new
tlxksvr03:qo/news
tlxksvr03:qsummary/new
tlxksvr04:qo/sent
tlxksvr04:gosummary/now
tlxksvr05:qo/new
tlxksvr05:qsummary/now
tlxksvr06:qo/now
tlxksvr06:qsummary/new
tlxksvr07:qo/new
tlxksvr07:qsummary/sent
tlxksvr08:qo/new
tlxksvr00:qsummary/new
tlxksvr09:qo/new
tlxksvr09:qsummary/new
tlxksvr10:qo/new
tlxksvr10:qsummary/new
tlxksvr11:qo/new
tlxksvr11:qsummary/new
tlxksvr12:qo/new
tlxksvr12:qsummary/new
tlxksvr13:qo/now
tlxksvr13:qsummary/clone querying
tlxksvr14:qo/new
tlxksvr14:qsummary/now
tlxksvr15:qo/new
tlxksvr15:qsummary/finished
tlxksvr16:qo/new
tlxksvr17:qsummary/clone querying
tlxksvr17:qo/new
tlxksvr17: q0 | 0.373134% |
tlxksvr18:qsummary/new
tlxksvr19:qo/new
tlxksvr qsummary/done_querying
tlxksvr20:qo/new
tlxksvr20:qsummary/new
tlxksvr21:qo/new
tlxksvr21:qsummary/new
48
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Results
- From the main menu bar, click
RESULTSto retrieve the results of previous queries. - By changing the start and stop dates, queries performed between those dates can be viewed.
- If the query name is known, it can be entered in the
QUERY_NAMEfield. - If the USERID is known, it can be entered.
- When complete, a window displays with the matching queries.
49
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
XKEYSCORE PROCESSES
50
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Lesson Objectives
- XKEYSCORE Process Data Flow
- Processing Programs
- Query Processes
- Other Processes
- Cronjobs
- crontab
51
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Back-End Process Data Flow
WC2
SOTF
sotf_dist
sotfinput_handler
xscore_proc
(process_data_parent)
MAILORDER
Insert Host
File System and other formats
file_input_proc
Scan dirs for new files
xs_task_db
xks_meta_ingester
i0
InsertDBs
register_metadata_tables
db_input_file_handler
Web UI
Master Host
xs_web_db
query_dispatch
xs_web_db
xscore_proc
(query_proc)
Query Host
q0
QueryDBs
52
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Processing Programs
- Processing programs are the main processes that extract metadata from the traffic and then database the information in insert databases.
| PROGRAM | DESCRIPTION |
|---|---|
file_input_proc |
Scans for new input files. (before processing moves the file to the .tmp directory of the input directory specified) |
sotf_dist |
Listens for incoming SOTF sessions |
process_data_parent |
Processes all new files discovered by file_input_proc or sotf_dist; optionally archives content and databases metadata. Parent process loads all dictionaries and starts up, then forks child processes which do the actual processing. |
53
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Processing – process_data_parent
- This process replaces
process_data0throughprocess_dataX - The “parent” process starts up and loads all the dictionaries, and then “forks” child processes which actually do the processing
- Parent acts similar to the
xks_app_launcher, managing restarts for the children when they die - When dictionaries are modified, parent reloads them and restarts the children
xks procwill show an “X/Y” number next toprocess_data_parent- This is the number of children currently running, over the number that should be running (based on the
xks.config num_data_processorssetting) pdpwill show up yellow anytime X != Y and green when everything is running normally- This means when you first (re)start
pdp, it will show yellow while it is loading the dictionaries, because none of the actual childprocess_data’s are running yet
- This is the number of children currently running, over the number that should be running (based on the
xks procwill report extra or missingprocess_dataXwith a PID of 0- Can’t tell what PID missing
process_datais suppose to have, because its managed by the parent now
- Can’t tell what PID missing
54
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Query Processes
- Query processes are processes that search and submit all necessary tables for the analysts queries.
| PROGRAM | DESCRIPTION |
|---|---|
query_dispatch |
Submits search jobs to search databases and propagates the status of the search and results back to the web server |
query_proc |
Searches through all the necessary tables for the analysts queries. |
55
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Other Process
- Other process which is run from the Application Launcher.
mailorder_proc– polls the/export/data/xkeyscore/outputs/mailorder_workingdirectory by default. Then renames and moves mailorder files to/export/data/xkeyscore/outputs/mailorder.
56
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Other Process
xks_meta_ingester– streams metadata over socket. This process improves database performance. Instead of eachxscore_procwriting to the database independently, they stream their metadata over socket to themeta_ingester, which combines it by plugin and writes to the database.- Reduces the number of connections to MySQL and gives better control over table size.
57
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Other Processes
register_metadata_tables– moves tables from processing database of XKEYSCORE system to query database.- Works against the
uber_indextable uber_index->table_name,base_table_name,join_table- Base table – contains common information amongst tables (
full_log_xxx_xxxxxxtable) - Extension table – extends the base table
- Registration process takes place in two phases:
- Register all base tables
- Register all extension tables that have had its base table registered
- Works against the
58
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Other Processes
signal_acquisition_loopback– process that feeds modified packets back into the system.- Front-end for packet recursion or any other process that feeds modified packets back into the system
- Reinjects back to front-end –
xfip - Process is completel independent
- Reinjects back to front-end –
- Front-end for packet recursion or any other process that feeds modified packets back into the system
59
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Other Processes
mpmr_server– this is the map-reduce server for microplugins, which runs the “Reducer” portion of GENESIS v5 microplugins.- Runs outside the normal processing flow, and will not affect the rest of the system.
- It has a telnet port (5850) just like an
xscore_proc.
60
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Other Processes
correlation_server_0– in-memory map-reduce server for correlation engine.- Each machine has one
correlation_server, and everyprocess_data_parentconnects to everycorrelation_serverxscore_proc– 8GB by default- uses port 4321
61
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Other Processes
xks_comms_server– a more efficient way to communicate with hosts within and outside an XKS cluster (not currently implement)- Automatically handles configuration for talking between slaves, master and overlord at site
- Configuration is needed to connect to the “peer” on the path towards, other sites
- Comms configuration lives in
$XSCORE_DIR/config/comms/comms.config - Supports a “quality of service” which “fairly” distributes available bandwidth to the services that are using comms
62
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Other Processes
xks_comms_server- Allow and Peer rules have a “network” parameter which the comms systems uses to determine an “inside” and an “outside” in proxies.
- Comms system will only accept connections from address ranges it has been specifically configured to allow.
- Every between 2 comms servers connection should have:
- “bandwidth_rule” on each side, name doesn’t matter but both rules should usually have same bandwidth cap
- “allow” rule on one side with a reciprocal “peer” rules on the other side
63
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Other Processes
-
xks_comms_server- Example: If we have a site named “US-123” connecting to
xks-centralover a 1Mbps link,US-123’s config would be:
bandwidth[world] = 1Mbps peer[00] = address=xks-central.corp.nsa.ic.gov, port=2412, bandwidth=world, network=externalAnd
xks-centralwould have:bandwidth[us123] = 1Mbps allow[00] = address=xkey-master.us123, bandwidth=us123, network=internal - Example: If we have a site named “US-123” connecting to
64
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
GUI Processes
- Other process which is run from the Application Launcher.
GUId– rescans content against fingerprints when a user clicks to view the content of a session.tomcat.sh– web server used to host XKS GUIsotftod124server– downloads sessions- Gets called from the GUId process
- Works with any downloaded traffic that is SOTF
65
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Statistic Processes
- Other process which is run from the Application Launcher.
xks_server_stats– sends toxks_system_monitoron Master and generates stats about the server itself.- CPU usage, memory usage, disk space, disk I/O, network traffic, etc.
- Stats are fed to
xks_system_monitorand the system monitor does magic with them.
66
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Statistic Processes
xks_system_monitor– collects stats messages from all over the system (front-end and back-end and the server itself) and summarizes them for forwarding. Optionally it can database stats locally.
67
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Cronjobs
- XKEYSCORE uses a number of cron jobs to perform tasks.
| CRONJOB | DESCRIPTION |
|---|---|
age_off_new.php |
Ages off metadata and content when the disk is near capacity, or when thresholds have been met. |
xks update_dictionaries |
Pulls updates from various sources. |
xks rsync push_config |
Copies the /opt/xkeyscore/config directory to the slaves. |
rwc_post_to_pub.py |
Once an hour kicks off an update request |
68
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
CRONTAB
Crontabis the program used to install, uninstall or list the tables used to drive the cron daemon.- The
crontabconsists ofage_off_new.phpxks update_dictionariesxks rsync push_configrwc_post_to_pub.py
69
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
CRONTAB
age_off_new.php- Options:
-debug: extra debug statements in the output-info: extra info statements in the output-task_db: explicitly state that the machine is a task host-web_db: explicitly state that the machine is a web host-nosleep: use if you want to run now
- This process ages off tables and archived data based on the settings in the
xks.configfile and the percentage of disk space used.
- Options:
70
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
CRONTAB
xks update_dictionaries- This process pulls the necessarily files from various sources to update the dictionary.
- Configure
/opt/xkeyscore/config/xks.config#[dictionaries] dictionary[0] = type=royale, \\ src=sftp://tssi_fvey:tssi_fvey@\\ xks-control/home/tssi_fvey/xks_dict_update.tar.gz, \\ dest=update/xks_dict_update.tar.gz, \\ action[0]="cd \\ $XSCORE_DIR/config/dictionaries/update;$XSCORE_DIR/config/\\ dictionaries/update/dup_install.pl >/dev/null 2>&1" dictionary[1] = type=cadence
71
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
CRONTAB
xks rsync push_config- Transfers Master configurations to its slaves.
- Excludes dot files, “httpd/logs”, loadserver/packages”, “httpd/log”
force: option toxksto forcepush_configwhen not on the master
72
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
CRONTAB
rwc_post_to_pub.py- The automatic
starProcprocess is as follows:- Hour 1: master asks whoever (say
xks-control) for an update, gets the rpm, installs it, there is much rejoicing. The slaves asks the master for the rpm at the same time the master asksxks-control, but obviously the master doesn’t have it, so nothing happens. - Hour 2: everyone asks for an update again, this time the master has the rpm, the slaves download it and install and there is much rejoicing.
- The rpm is installed and
process_data_parent’s are restarted as soon as the rpm is downloaded on a given machine.
- Hour 1: master asks whoever (say
- The automatic
73
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
DeepDive
74
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
DeepDive
- What is a DeepDive?
- Why DeepDive?
- What does a DeepDive look like?
- Front-End Processes
- xFIP
- Promoter
75
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
DeepDive XKEYSCORE
- XKEYSCORE packet processing solution
- XKEYSCORE’s software handles all packet processing
- No upfront filtering prior to XKEYSCORE
- XKEYSCORE “promoter” tries to promote richest/most interesting traffic
- All Strong Selectors
- Full take ASDF (User Activity metadata)
- Subset of GENESIS signatures
- List managed by XKEYSCORE team in concert with collection managers and site engineers
- 20% - 30% of site traffic is fully processed and can be found via XKEYSCORE search
- Typically does not include unknown or uninteresting protocols
76
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Why DeepDive?
- Access to most relevant DNI data supporting SigDev and collection missions. Enables new mission capabilities (e.g., Correlation)
- Session promotion can be synchronized and managed based on Genesis signatures, traditional tasking selectors and available resources
- Provides better scaling
- Drop unwanted data. Keep the rest and make decisions later and more accurately
- Better control of the processing space
- Instantiate new mission capabilities and dataflows quickly
- Troubleshooting and monitoring made easier
- Need access to “raw” packets to support new mission (e.g., Cyber, Bulk Crypt)
- Sessions can be displayed as Packet Bundles like Wireshark
77
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
DeepDive
- What does a DeepDive look like?
- XKEYSCORE full-take session processor (Back End)
- High speed packet ingest: an end-to-end solution
- Intelligent filtering to vary the proportion of traffic retained
DEEPDIVE
Front End
Packet Splatter
Mettlesome
Promoter
Defrag
Back End
Dictionary Scanner
Plugins
Microplugins
Fingerprints
Metadata
Content
Packets
Partial Sessions
Full Sessions
78
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
XKEYSCORE Front-End Processes
| What it's called | What it does | What it means |
|---|---|---|
Packet Splatter |
Ingests packets (from files, from the network, from a capture card) in a variety of formats. | If it's a packet stream, it can probably be fed into a DEEPDIVE. |
xFip |
Fast reassembly of TCP/IPv4, UDP/IPv4 streams\, and TCP/IPv6 and UDP/IPv6 streams\. | DEEPDIVE sessionizes everything before making a keep/drop decision. |
METTLESOME |
Reassembly of streams from less common protocol stacks. | |
Promoter |
Rule-based filtering of reassembled sessions, based on keyword, country code or appid/fingerprint. | DEEPDIVE intelligently chooses the most useful traffic for retention. |
Defrag |
Fully rebuilds sessions\\ | Enough content available to do full decoding/document descent at the Back End |
79
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xFIP
- Packet bundles
- Preserves original packets and packet order
- Preserves information that is lost during sessionization
- Original pcap available in the XKS Viewer
- Packet API
- Microplugins can iterate over raw packets
- Microplugins can use information that is lost during sessionization
- E.g. timestamps, flags, checksums
- Packet fingerprints
- Fired based on observations
xFiphas made- E.g. large sequence gaps, TTL variation
- Fired based on observations
80
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Promoter
- Filters sessions prior to back end processing
- keywords, regex, country code, appids*
- SIGDEV: promotion rather than strong selection
- Set the focus of the back end
- traffic types of interest
- regions of interest
- legal/policy constraints
- Set the width of the access aperture
- promote 20% of 20 signals?
- promote 100% of 4 signals?
- Set the length of data retention
- promote 20% and keep for 3 days?
- promote 30% and keep for 2 days?
allow appid chat.*
allow country_code PK
block country_code US-US
81
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks Script
82
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks Script
- Usage
- options
- General Commands
- Services
- Actions
- Options
83
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks Options
- Usage:
xks [options] <command> - Try
xks help <name>to get help on a specific service or action - General commands:
services- list available servicesactions- list available actionsdependencies [invert]- show service dependencieshelp [items]- print help on services or actions
- Services (specify one or more service names or ‘all’):
start <services>- start the specified servicesstop <services>- stop the specified servicesrestart <services>- restart the specified servicesstatus <services>- print the status of the specified servicessetup <services>- setup/configure/fix the current xks install
84
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks Actions
- Actions:
accounts_report- sends an email containing accounts usage to the specified usersadd_admin- sets up a local Linux user to administer XKSchange_db_password- changes the XKS database user’s password and updates all references to itcluster- cluster actionscompile_genesis- compiles GENESIS signaturesdisk_check- get raid and disk statusext4_format- format $XSCORE_DATA_DIR partition and convert to ext4 filesystemext4_upgrade_contents- convert to ext4 filesystem while preserving of $XSCORE_DATA_DIR (no formatting)fetch- fetch a remote fileforce_register- force metadata table registrationinfo- show cluster informationinstall_slave- install a slave machine in this clusterlocal_tagging- checks and/or loads tagging file
85
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks Actions
- Actions:
monitor- view XKS monitoring messages via activemqmpmr_register- forcempmrtable registrationmysqls- run a mysql scriptonall- run a command on all machines in this clusterpowertower- configure or run a powertower commandproc- control XKS processes on this clusterquery- display query status or submit a queryquery_dispatch- command line interface to the XKS query_dispatcher(s).rac- access remote admin portsreload_dictionaries- force running processes to reload dictionariesrsync- push configs or files to slavessearch_fields- populates user settings with search fieldsshow_config- show values from xksconfig for specified keysswitch- query or rebalance data switchsync_accounts- synchronize user accounts (except for classifications)tail- view realtime xks logstasking_dump- print out the contents of thexkTaskingandxksTasking_voipdatabases.
86
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks Actions
- Actions:
top- display system performanceupdate_dictionaries- update all XKS dictionariesupdate_gui_help- update the 'help' pull downs in GUIusers- display the users currently logged into the GUIversion- show XKS version informationwatchdog- check and (re) start essential XKS processes.workflow- manually submit a workflow
87
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks Options
- Options:
-verbose: print extra information to the screen-debug: used for debugging script problems
88
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks – General Commands
- Type:
xks help services- This will list all available services:
first– initialization service that runs before all othersvirus_scanner– sets up virus scanner, assuming tarballs are present.ftpd– enables ftp on the master if mailorder is enableddistcc– sets up distributed compiler serviceslash_proc– setup optimal /proc parametersmyricom– handles installation and configuration 10GigE network cardshome– sets up the home directory for the xks user accountgcc– check there is a working compiler on the systemupgrade– updates configuration files when upgrading to a new version of xksbashrc– sets up bash environment variablesbeacon– sets up xks monitoring beacon based on xks.configtt– checks connectivity to TRAFFICTHIEF server
- This will list all available services:
89
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks – General Commands
- Type:
xks help services- This will list all available services:
sendmail– configures sendmail for use with xksrole_files– this service installs role-specific filesissue– sets up the DoD mandatory login warningsroyale_with_cheese– setups automatic updatesntpd– configure ntp based on xks.configlink_summary– sets up xks link summary GUInfsd– sets up xks-specific nfs mountsserver_certs– sets up server certificates for SSL applicationsopenoffice– installs and configures OpenOffice for use in the xks GUIinit_d– sets up the xks init.d servicesresolver– sets up resolver configphp– sets up PHP related stuff. Except php.inihttpd– sets up xks-specific httpd configuration
- This will list all available services:
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
90
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks – General Commands
- Type:
xks help services- This will list all available services:
www– sets up GUI configuration filesvoip– sets up voip processingcrond– ensures xks can use cron and sets up xks cron jobssshd– configures the secure shell service for use with xkslicense– checks for a valid license file and if one isn’t found prints a messagesyslog– configures the syslog service for use with xks- all xks processes log to
/var/log/xks.log
- all xks processes log to
dictionaries– checks status of any configured dictionariescluster_check– checks network connectivity across the clusterautofs– start, stop, restart automountsloadserver– start, stop, and setup loadserverdirectories– sets up directories used for xksauditd– no help available
- This will list all available services:
91
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks – General Commands
- Type:
xks help services- This will list all available services:
ldap– no help availablemysqld– sets up the mysql server for use with xksdisks– checks status of disk partition used by xksdatabases– maintains database scheme consistencylocal_tasking– reapplies local tasking if necessaryworkflows– sets up xks default workflowscategory_throttle– overrides default category throttle settings based on overrides specified in xks.configenrichment_tomcat– sets up enrichment tomcat java application serverplugin_setup– populate plugin database tables from xml files, apply default plugin config specified in xks.config, apply overrides from xks.config, regenerate plugin config files from databasecrdb– no help availabletomcat– sets up tomcat java application serverclickstream– sets up clickstream service
- This will list all available services:
92
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks – General Commands
- Type:
xks help services- This will list all available services:
file_input– sets up directories and database entries needed for file-based inputage_off_db– synchronizes the database (xs_task_db.age_off) withxks.config’s settings for content and metadata. The values in the database will be unconditionally overwritten with those found in xks.configdb_connectivity– verifies connectivity to critical databasespdf– sets up xpdf language packsul_age_off– sets the maximum data retention time to a little over an hour in UL mode.mDNSResponder– sets up mDNSResponder for use with SOTF inputapp_launcher– controls the xks app launcher, which is responsible for monitoring xks processes and starting/stopping them as commanded from the GUIprocesses_setup– configures xks processes based on specifications in xks.configcomms– sets up the XKS communications system configurationadvanced– performs advanced configuration
- This will list all available services:
93
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks – General Commands
- Type:
xks help services- This will list all available services:
endace– handles all the installation and configuration for Endace Dag packet capture cardslast– cleanup service that runs after all others
- This will list all available services:
94
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks - Services
startxks start mysql
stopxks stop httpd
restartxks restart nfs
statusxks status autofs
setupxks setup plugins
95
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks - Actions
-
xks onall ‘ps -ef | grep xscore | grep -v grep’ -
xks force_register[oper@tlxksvr01 run]$ xks force_register Forced update on xks_meta_ingester -
xks rsync push_config -force- Usage:
xks rsync <options> [push_config|push_compiled|push_slaves|push] <src> <dest>
- Usage:
-
xks update_dictionaries- Usage:
xks update_dictionaries [test|check|print|force|help]
- Usage:
-
xks version[oper@tlxksvr01 run]$ xks version 1.5.9-65 -
xks info[oper@tlxksvr01 run]$ xks info site : Timberline - SV SIGAD : USF-790 PDDG : IE XKS version: 1.5.9-65 Master : tlxksvr01 Num slaves : 13 Input : file, sotf Config : managed_srtd
96
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks - Actions
-
xks query servers[oper@tlxksvr01 run]$ xks query servers tlxksvr02:q0 3a 14s 98n 54w 2012-12-05 16:07:22 (top) tlxksvr03:q0 230a 0s 0n 0w 2012-12-05 17:55:02 (top) tlxksvr04:q0 230a 0s 0n 0w 2012-12-05 17:55:02 (top) tlxksvr05:q0 225a 0s 4n 0w 2012-12-05 17:55:02 (top) tlxksvr07:q0 230a 0s 0n 0w 2012-12-05 17:55:02 (top) tlxksvr08:q0 225a 0s 5n 0w 2012-12-05 17:55:02 (top) tlxksvr09:q0 0a 0s 173n 2w 2012-12-05 17:55:02 (top) tlxksvr10:q0 230a 0s 0n 0w 2012-12-05 17:55:02 (top) tlxksvr11:q0 230a 0s 0n 0w 2012-12-05 17:55:02 (top) tlxksvr12:q0 225a 0s 4n 0w 2012-12-05 17:55:02 (top) a=awaiting dispatch, s=sent, n=new, w=working timestamp shows earliest submitted but unfinished query current time: 2012-12-05 18:02:09
97
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks Example
-
xks proc[oper@tlxksvr01 run]$ xks proc GUI GUId qp query_proc cli clickstreamservice.sh rmt register_metadata_tables cms check_mailorder_site.php s2d sotftod124server cs00 correlation_server_0 sab signal_acquisition_base ctp cadence_tasking_proc sal signal_acquisition_loopback enr enrichment-tomcat.sh sd sotf_dist file file_input_proc sst strong_selector_targeting mp mailorder_proc tom tomcat.sh mpmr mpmr_server xcs xks_comms_server pd# process_data# xmi xks_meta_ingester pdp process_data_parent xsm xks_system_monitor qd query_dispatch xss xks_server_stats Run 'xks proc full' to show full listing tlxksvr01 GUI cli cms ctp enr file mp qd qp rmt s2d sab:4/4 sst tom xcs xmi xsm xss tlxksvr02 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr03 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr04 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr05 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr06 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr07 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr08 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr09 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr10 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr11 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr12 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr13 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr14 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss Process consistency check OK on all hosts
98
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks Example
-
xks proc full[oper@tlxksvr01 run]$ xks proc full app launcher status: RUN (pid 30705) id hostname program arguments commanded actual pid 14 tlxksvr01 cadence_tasking_proc --myfdi XYD --pddg IE --dig... RUN RUN 31136 4 tlxksvr01 check_mailorder_site.php RUN RUN 31019 723 tlxksvr01 clickstreamservice.sh RUN RUN 31053 654 tlxksvr01 enrichment-tomcat.sh RUN RUN 31200 9 tlxksvr01 file_input_proc RUN RUN 31104 1 tlxksvr01 GUId RUN RUN 9335 548 tlxksvr01 mailorder_proc --copydir /export/data/xkey... RUN RUN 31140 8 tlxksvr01 query_dispatch RUN RUN 17549 3 tlxksvr01 query_proc RUN RUN 30965 193 tlxksvr01 register_metadata_tables --loglevel error RUN RUN 31143 709 tlxksvr01 signal_acquisition_base -f generic_packet_to_bundle... RUN RUN: 4/4 31236 12 tlxksvr01 sotftod124server RUN RUN 31108 461 tlxksvr01 strong_selector_targeting RUN RUN 31145 13 tlxksvr01 tomcat.sh RUN RUN 31124 653 tlxksvr01 xks_comms_server RUN RUN 31148 5 tlxksvr01 xks_meta_ingester RUN RUN 31051 462 tlxksvr01 xks_server_stats RUN RUN 31138 11 tlxksvr02 register_metadata_tables RUN RUN 13906 724 tlxksvr02 correlation_server_0 --loglevel debug RUN RUN 13946 31 tlxksvr02 mpmr_server RUN RUN 7150 730 tlxksvr02 process_data_parent --max-mem 20 RUN RUN: 4/4 22661 30 tlxksvr02 query_proc RUN RUN 14754 187 tlxksvr02 register_metadata_tables --loglevel error RUN RUN 14994 710 tlxksvr02 signal_acquisition_base -f generic_packet_to_bundle... RUN RUN: 4/4 14996 41 tlxksvr02 signal_acquisition_loopback -f packet_aux.config -i loo... RUN RUN: 6/6 14990 259 tlxksvr02 sotf_dist RUN RUN 14845 679 tlxksvr02 xks_comms_server RUN RUN 14992 38 tlxksvr02 xks_meta_ingester RUN RUN 14970 473 tlxksvr02 xks_server_stats RUN RUN 14988
99
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks Example
-
xks query[oper@tlxksvr01 run]$ xks query id user type search start search stop duration status 66250201 http_parser 00:00 12/2/12 00:00 12/6/12 00:00:10 ongoing 66250183 full_log 00:00 12/3/12 00:00 12/6/12 00:00:17 ongoing 66250155 full_log 00:00 12/4/12 00:00 12/6/12 00:00:40 ongoing 66250127 geo_info 00:00 11/30/12 00:00 12/6/12 00:01:16 ongoing 66250052 email_addresses 00:00 11/21/12 00:00 12/6/12 00:03:36 ongoing 66249873 full_log 22:00 12/3/12 21:59 12/4/12 00:11:31 ongoing 66249660 full_log 00:00 12/2/12 00:00 12/6/12 00:18:57 ongoing 66244233 category 00:00 11/5/12 00:00 12/6/12 00:42:17 ongoing 66244135 full_log 00:00 11/28/12 00:00 12/6/12 00:44:30 ongoing 66244009 http_parser 00:00 11/5/12 00:00 12/6/12 00:48:49 ongoing 66243967 http_parser 00:00 11/5/12 00:00 12/6/12 00:49:34 ongoing 66243855 document_metadata 00:00 11/21/12 00:00 12/6/12 00:50:48 ongoing 66243785 correlation 00:00 11/5/12 00:00 12/6/12 00:52:46 ongoing 66243463 correlation 00:00 11/21/12 00:00 12/6/12 00:56:13 ongoing 66243071 email_addresses 00:00 11/1/12 00:00 12/6/12 01:08:24 ongoing 66242973 user_activity_exif 00:00 11/21/12 00:00 12/6/12 01:12:03 ongoing 66242413 http_parser 00:00 11/28/12 00:00 12/6/12 01:26:32 ongoing 66242315 full_log 00:00 12/4/12 00:00 12/6/12 01:30:52 ongoing There are 18 queries in progress
100
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks query detail
[oper@tlxksvr01 run]$ xks query detail id=66251065
Query Summary
Userid:
Type: email_addresses
Searching from 00:00 11/30/12 to 00:00 12/6/12
Duration: 00:01:29
Priority: 5
Cancel: N(o)
Max Results: 10000
Max Time: 6000
Query SQL
Name: smstev3_4
Classification: S,IS/SI,NSA NOFORN,IS,HCS,S/SI,MUSCULAR,REL_USA,NSA_NOFORN,SI,C,R
Where: WHERE datetime >= '2012-11-30 00:00:00' AND datetime <= '2012-12-06 00:00:00' AND email = '[REDACTED]' AND domain = 'hotmail.com'
Query Status
host database status
tlxksvr01 q0 finished
tlxksvr02 q0 ongoing
tlxksvr03 q0 finished
tlxksvr04 q0 finished
tlxksvr05 q0 finished
tlxksvr06 q0 finished
tlxksvr07 q0 finished
tlxksvr08 q0 finished
tlxksvr09 q0 finished
tlxksvr10 q0 finished
tlxksvr11 q0 finished
tlxksvr12 q0 finished
tlxksvr13 q0 finished
tlxksvr14 q0 finished
101
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Monitoring
102
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Lesson Objectives
- Executables
- mysqls
- onall
- xks onall
- xks monitor
- sotf_stat
- xks top
- Web Status
- Additional Monitoring
103
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Executables
- System monitoring can be performed from the command line using the following executable commands:
mysqlsonallxks onallsotf_statxks top
104
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
mysqls
- The
mysqlsbash shell script can be used to execute MySQL statements from in the/opt/xkeyscore/bin.shells/sysadmin/mysqlsdirectory. The most commonly used options inmysqlsare:status– displays file-based input statistics.speed– displays the total file based input processing rate (Mbps)speed1– displays file-based input processing rate (Mbps) per input source.speed2– displays file-based input processing rate (Mbps) per xkeyscore processing server.count– displays the count of input files in the new, working, error, and done states.
105
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
mysqls
-
mysqls status[oper@tlxksvr01 run]$ mysqls status status count(*) sum(filesize) priority bitrate_Mbps NULL
106
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks onall
-
xks onall ‘xks mysql status’[oper@tlxksvr01 run]$ xks onall 'xks mysql status' Do you want to execute "xks mysql status" onall? [y|n] y ---- tlxksvr01 ---- status mysqld mysqls is running ---- tlxksvr02 ---- status mysqld mysqls is running ---- tlxksvr03 ---- status mysqld mysqls is running ---- tlxksvr04 ---- status mysqld mysqls is running ---- tlxksvr05 ---- status mysqld mysqls is running
107
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks monitor
- This script will monitor your front-end processes.
- Type:
xks montororxks monitor hto receive the help menu
---XKEYSCORE-Menu---
Command Name Description
c config Configure this utility
d dataflow_all FrontEnd Dataflow Menu
b dataflow_be BackEnd Dataflow Menu
m,h menu View this menu
a packet_splatter Packet Acquisition [Front End]
p process_data Process Data [Back End]
q quit Quit/Exit
s servers Server Stats (CPU,IO,etc.)
t sotf_input SOTF Input [Back End] 'xks top' replacement
f xfip Sessionization [Front End]
108
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks monitor
-
Type:
xks montor fto receivexfipstats--XKEYSCORE-Sessionization [Server View Press '4']-- Casenotation Rate(Mbps) Loading% TCPQuality% Rate(Pkts) Count(Pkts) Pkts% Fragments% 7DH115181980000 0.00 0.00 0.00 0.00 0 0.00 0.00 7DH115209040000 0.00 0.00 0.00 0.00 0 0.00 0.00 Total: 0.00 [KEY: STM1,STM4,STM16,STM64]
109
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
sotf stat
- The
sotf_statcommand is used to display the SOTF (streaming object transfer format) input statistics for an entire cluster. - The statistics include total number of
process_data’s running on the cluster, session input rate (sessions/sec), total bytes input (Mbps), and total bytes output toprocess_data(s)(Mbps).
110
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
sotf stat
- To execute the
sotf_statscript:- Log on to the server and open a terminal window.
- Type
sotf_statbecause the command is in the path - Type
sto toggle the summary statistics view from total statistics to individual host statistics. - Type
qto quit the program
111
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
sotf stat
- The
sotf_statscript lists the hostname, number ofprocess_data’s currently running, Mbps, number of sessions, and number of bytes.
---XKEYSCORE SOTF Statistics---
Hostname #In #OA/#OC Mbps In Sess In Bytes In Sess Q MaxBlk
mhxkssvr02 7 4/4 18.66 410920147 4818112974976 0 0
mhxkssvr03 3 4/4 16.15 410121822 4783549865004 0 0
mhxkssvr04 3 4/4 16.65 410444622 4781320276992 0 0
mhxkssvr05 3 4/4 15.79 409831857 4759939303920 0 0
---PRC: 15/ 15 Rate: 64.52 Mbps Sessions: 1641358767 Bytes: 19143289212772---
112
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
xks top
- The
xks topscript lists the hostname, Mbps sotf rate, number ofprocess_data’s running, the % of CPU, and % of IO wait.
hostname sotf #procs cpu% iowait%
mhxkssvr01 -0.00 0 0.53 0.02
mhxkssvr02 21.08 4 12.88 7.94
mhxkssvr03 13.55 4 13.35 7.28
mhxkssvr04 14.97 4 14.50 8.63
mhxkssvr05 14.13 4 17.14 8.01
TOTAL 63.74 16 11.68 6.38
113
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Additional Monitoring
-
xks tail[oper@mhxkssvr02 ~]$ xks tail Dec 5 13:27:29 mhxkssvr02 register_metadata_tables[13877] : <register st (automatic?) repair failed Dec 5 13:27:29 mhxkssvr02 register_metadata_tables[13877] : <register st (automatic?) repair failed Dec 5 13:27:29 mhxkssvr02 register_metadata_tables[13877] : <register st (automatic?) repair failed Dec 5 13:27:59 mhxkssvr02 register_metadata_tables[13877] : <register st (automatic?) repair failed Dec 5 13:27:59 mhxkssvr02 register_metadata_tables[13877] : <register st (automatic?) repair failed Dec 5 18:27:59 mhxkssvr02 register_metadata_tables[13877] : <register st (automatic?) repair failed Dec 5 18:28:08 mhxkssvr02 sotf_dist[13986] : <sotf_dist_t> NOTICE: cu Dec 5 18:28:29 mhxkssvr02 register_metadata_tables[13877] : <register st (automatic?) repair failed Dec 5 18:28:29 mhxkssvr02 register_metadata_tables[13877] : <register st (automatic?) repair failed Dec 5 18:28:29 mhxkssvr02 register_metadata_tables[13877] : <register st (automatic?) repair failed
114
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Troubleshooting
115
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Lesson Objectives
- Common Troubleshooting techniques
- Full Disk
- Sotf Problems
- MySQL
- Processing Problems
- Outputs
- Query Problems
- Directory Permissions
116
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Disk Full
/var/log/xks.log(xks tail) – Relevant error messages can be viewed in this file. This directory may fill the disk, some known reasons are:process_datahas lost its connection with thesotf_distand is continuously trying to reconnect tosotf_dist.nfserror may have occurred and a detailed message can be found in the file/var/log/messages.- Corrupt tables in the insert database.
- Check to make sure the
age_off_new.phpcronjob aged off old metadata and content.
117
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Disk Full continued…
/export/data/xkeyscore/inputs- If there are too many files in the directory:
file_input_procmay be running improperly or not at all. Verify thatfile_input_procis running from the command line type:ps -ef | grep file_ | grep -v grepxks proc
- The
file_input_procmay need to be restarted.
- If there are too many files in the directory:
- No new files in the directory:
- The directory may not be cross-mounted properly, if automounting is used.
118
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Disk Full continued…
/export/data/xkeyscore/mysql/i0ori1- If
/export/data/xkeyscore/mysql/i0ori1are filling andq0and/orq1maintains its size,register_metadata_tablesmay not be working properly.- Restart process and watch the databases to see if it is transferring files or run the process by hand to troubleshoot further.
- If
/export/data/xkeyscore/mysql/q0orq1is filling, theage_off_new.phpscript may be running improperly or not at all.- First run the command:
ps -ef | grep age_- If script isn’t running, try running it by hand.
- If script is running, then stop script and try running it by hand to see if there are any errors.
- First run the command:
- If
119
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
SOTF Problems
- Can an
sotf_input_procrun with a file_basedfile_input_proc?- Yes. Both input types can run on XKEYSCORE given that each are independently configured correctly.
- Can file-based input be disabled so that only
sotf_inputis processed?- If moving from file-based input to
sotf_input, and no additional file-based input is expected, the plug-in for file-based input,db_input_file_handler, should be disabled. - From the TERMINAL WINDOW:
- Stop all the processes :
xks stop all - Change
/opt/xkeyscore/config/xks.configto setfile_inputto ‘no’. - Setup the config :
xks setup plugins,xks setup processes - Rsync change to slaves :
xks rsync push_config - Restart
process_data’s :xks proc restart pdp
- Stop all the processes :
- If moving from file-based input to
120
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
SOTF Problems continued…
- Is XKEYSCORE receiving input?
- To verify whether XKEYSCORE is receiving input, run the
sotf_statcommand to get the current input statistics. - If no connection is visible, from the command line:
- Type
telnet localhost 5042 - Output statistics for the specified
sotf_dist - If running, type
ps -ef | grep sotf_dist - Determine if
sotf_dist’s are listening on the specified port: Typetelnet localhost 5040If command is refused, thesotf_distis not listening on the port. Continue with step 5. - Type
netstat -a | grep 5040If a connection is established for this port then most likely thesotf_distis listening on this port.
- Type
- To verify whether XKEYSCORE is receiving input, run the
121
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
SOTF Problems continued…
netstatwill tell if...sotf_distis listening for connections- If connections have been made to the
sotf_dist - If we are “backing up”- i.e., if
sotf_distis running but has noprocess_data’s connected to it, it won’t be able to send data anywhere, so eventually its network receive queue will get large.- Ideally, the receive queue should always be 0.
122
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
SOTF Problems continued…
- Is the
process_data_parentrunning?- At least one
process_datamust be running and synchronized with thesotf_distfor it to receive input.- If problems continue, run the
sotf_distin a terminal to further troubleshoot and identify error messages.
- If problems continue, run the
- At least one
123
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
MySQL – D124 file troubleshooting
- Symptom: A lot of errors or too many errors display when performing the command
‘mysqls status’:- First try,
mysqls cleanup, in a terminal window. - Type
mysqls status - Type
mysql xs_task_db;to log into MySQL database and use thexs_task_dbdatabase. - Execute the following command:
delete from tar_files where status="error"; - Exit out of the MySQL database
- Type
mysqls statusThere should no longer be any error files.
- First try,
124
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Processing Problems
- The heart of the XKEYSCORE processing engine is the
xscore_procwith related plugins. - Input to the
xscore_procis either file-based and from anfile_input_proc, or streaming from ansotf_input_proc. - After processing, the written metadata to the insert databases can be sent to a follow on system for additional processing.
125
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Processing Problems continued…
- How many
process_data’s should be running on a host?- From the XKEYSCORE GUI:
- Click ADMIN > Processing > Computer Resources
- Determine how many
process_data’s are configured to be running on the specified host.
- From the XKEYSCORE GUI:
126
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Processing Problems continued…
- How many
xscore_proc’s are actually running on a host?- Log onto the XKEYSCORE server and open a terminal window.
- Type
ps -ef | grep xscore | grep -v managed_
[oper@tlxksvr02 ~]$ ps -ef | grep xscore | grep -v managed_
oper 22661 11569 15 Dec03 ? 07:36:24 xscore_proc --parent --base_port 5550 --external_appid compile --db i0 --renice_level -5 --command db_force_next --max-mem 20 --loglevel error --restart_childre
n --childpid 31396 --childpid 29482 --childpid 29619 --childpid 29710
oper 29482 22661 26 19:08 ? 00:06:48 xscore_proc child-1 --port 5551 --external_appid compile --db i0 --renice_level -5 --command db_force_next --max-mem 20 --loglevel error
oper 29619 22661 26 19:08 ? 00:06:31 xscore_proc child-2 --port 5552 --external_appid compile --db i0 --renice_level -5 --command db_force_next --max-mem 20 --loglevel error
oper 29710 22661 26 19:09 ? 00:06:24 xscore_proc child-3 --port 5553 --external_appid compile --db i0 --renice_level -5 --command db_force_next --max-mem 20 --loglevel error
oper 31396 22661 41 19:28 ? 00:02:13 xscore_proc child-0 --port 5550 --external_appid compile --db i0 --renice_level -5 --command db_force_next --max-mem 20 --loglevel error
127
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Processing Problems continued…
xks_app_launcheris running, but not starting processes specified in the Computer Resources window?- This may indicate that the
xks_app_launcheris defunct. Use thekillcommand to kill theapp_launcherand its related sub-processes:- Type
pkill -f app_- If a PID is not being specified, use the
pkillcommand. The-foption kills all of the sub-processes.
- If a PID is not being specified, use the
- Type
psto look for the newxks_app_launcherprocess.
- Type
- This may indicate that the
128
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Processing Problems continued…
- If, after performing the procedures, the
xks_app_launcheris still not starting applications:- In a terminal window, manually run the problem process to see if there are any error messages.
- The
xks_app_launcheron any host is dependent on the access of thexs_task_db.proc_resourcesdatabase table on the master. Verify that the specified host can access the master’s database and/optdirectory. - On the slave system type
mysql xs_task_db-h <masterhostname>- performs a remote MySQL server login
129
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Processing Problems continued…
- To test the
xscore_proc, type:telnet <process host> <port number>Optional commands to assist trouble shooting are:sbr– prints the processing rate for the single xscore-proc.sh– displays dictionary hit statistics.ss– displays statistics on the internal plug-in processing rates.help– there are many commands and can be described in the help menu.
130
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Processing Problems continued…
- If the
process_data_parentcontinues to deny access through the command port, and input still has not started processing, check the input source. - Run the process in a terminal window with the argument
--loglevel debug, to view debug messages. - The command port also provides processing rates and statistics for troubleshooting performance issues, outages, and general administration issues.
131
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Outputs - Mailorder
/export/data/xkeyscore/outputs/mailorder- If there are no new files in the MAILORDER directory, MAILORDER may not be working properly. Possible causes are that:
- Files are being written to the wrong directory or it is not configured properly
- Permissions on the MAILORDER directory will not allow MAILORDER to move files
132
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Query - Dispatch
- Query dispatch is the process that submits search jobs to search databases and propagates the status of the search and the results of the search back to the web server.
- After submitting a new query, Search Status window displays a summary listing query name, date and time submitted, number of databases complete, and number of results.
133
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Query Problems
- The query never moves to the finished state.
- If a database outage or a comms outage occurs, results will not be reported from the single system. However, results from all other databases will return properly with the query results, but they will not appear in this state.
134
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Query Problems
- Query job status is stuck in
awaiting_disbatch.- If a status appears stuck in this state, the
query_dispatchmay not be running on the web server. To determine whether it is running:- Type
ps -ef | grep query_
- Type
- If the process is not running, restart it from the XKEYSCORE GUI or troubleshoot the
xks_app_launcher.
- If a status appears stuck in this state, the
135
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Query Problems
- Another cause of this scenario is that a query database may have hung up the query dispatch process. Check the progress of queries on the query database hosts by viewing the table
sdb_query_jobsin the query database, which tracks the status of queries:- Type
mysql q0 - Type
select status,count(*) from sdb_query_jobs where group by status;
- Type
- The select statement displays the current state of the queries on the query host. If many more queries appear in the new state when compared to other query databases, begin troubleshooting the problem
query_procon the specified query database.
136
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Query Processing
- The query is in the sent state, but never appears in new.
- After the
query_dispatchprocess disbatches the query, the status is moved tosent. A query moves to thenewstate when the query has been placed in the query processing queue on thequery_host. - If a query does not move to the
newstate in a reasonable amount of time, the connectivity of the database should be tested.
- After the
137
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Query Processing
- To check the progress of queries on the query database hosts, view the table
sdb_query_jobsin the query database, which tracks the status of queries:- Type
mysql q0 - Type
select status,count(*) from sdb_query_jobs where group by status;
- Type
- The select statement displays the current state of the queries on the query host. If many more queries appear in the
newstate when compared to other query databases, begin troubleshooting the problemquery_procon the specified query database.
138
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Query Processing
- The query appears in the new state, but never finishes.
- query is in the
newstate, has been received by the query host and placed in a queue waiting to be processed. - Queries can become backlogged with a large number of queries waiting in the
newstate, though thequery_procis processing the queries properly. It is hard to predict the time to work off a query backlog, but using the followingselectstatement the status of queries for the current day can be checked for processing trends.
- query is in the
139
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Query Processing
- To display the number queries in each state for the current day:
-
Type
select status, count(*),datetime_submitted, (UNIX_TIMESTAMP(now())- UNIX_TIMESTAMP(datetime_submitted))/3600 from sdb_query_jobs where (datetime_submitted>(now() - INTERVAL '1' DAY)) group by status;
-
- To display the number of queries processed per hour for the current day:
-
Type
select status,count(*)/24 AS queries_per_hour from sdb_query_jobs where cancel!="C" and (datetime_submitted>(now() - INTERVAL '1' DAY)) AS Backlog group by cancel;
-
- If processing properly, queries can take hours, if not days, to complete based on the backlog and the processing trends.
140
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Retrieving Metadata and Content
- Queries complete but there are no results.
- If queries complete, but no results are visible, verify that the date range of the query coincides with the collection date of the data. If using test data, test the query system by putting the start date range at a year or two older to assure it is not old test data.
- Verify that query metadata is in the query database by checking the contents of the
/export/data/xkeyscore/mysql/{query_db}/directory.
141
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Retrieving Metadata and Content
- Queries complete and metadata returns, but there is no content.
- The metadata in the XKEYSCORE viewer displays the host and directory path of the content file. Verify the content file exists using the
ls -lcommand. Trace a dataflow issue if the file does not exists. If the content file exists, confirm thehttpddaemon is started on all slave systems. To confirm thehttpddaemon:-
- Type
su - oper
- Type
-
- Type
xks status httpd
- Type
-
- If the daemon is not on, type
xks start httpd
- If the daemon is not on, type
-
- The metadata in the XKEYSCORE viewer displays the host and directory path of the content file. Verify the content file exists using the
142
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Query Results
- To troubleshoot problems with metadata or content from a query, it will be necessary to retrieve the actual content, since recreating the problem is very difficult. This can be accomplished from the XKEYSCORE GUI. Click RESULTS and begin a search of the questionable queries.
143
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL
Questions?
144
TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL