TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## XKEYSCORE System Administration December 2012 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Introduction 2 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Lesson Objectives * Introduction to XKEYSCORE * Purpose and Capabilities * Data Flow * What is a Cluster? * XKEYSCORE Databases 3 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Introduction * XKEYSCORE performs filtering and selection to enable analysts to quickly find information they need based on what they already know. * XKEYSCORE also performs SIGDEV functions such as target development to allow analysts to discover new sources of information. 4 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Introduction * XKEYSCORE processes data at field sites, where it is collected, and allows analysts from all over the world to query it. * At field sites, the XKEYSCORE software can run in clusters of few or many servers, giving it the ability to scale in both processing power and storage. * All processing is plugin or fingerprint based, which allows new capabilities to be quickly deployed to support operational needs. 5 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Purpose and Capabilities * XKEYSCORE is a Computer to Computer (C2C) exploitation system. * It is a fully distributed processing and query system. * XKEYSCORE can run on multiple servers. * Plugin and fingerprint architecture allows new capabilities to be quickly deployed. TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL 6 --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Hardware * XKEYSCORE is typically installed with Red Hat AS5u8 operating system. The suggested disk set up is: * Set up separate partitions for `/ (root)`, `/var`, `/tmp`, and `/export/data` * XKEYSCORE clusters can be composed of three different functionalities, which are: * One host acts as the web server/user interface, etc... * Another host normally runs as the real-time processing unit * Other host acts as the search or query system. * Hybrid system can perform multiple roles on one server, which enables efficient registration. * `process_data_parent` * `1 query_proc` 7 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Data Flow (High-level) * The backend is where the raw data for XKEYSCORE is processed; that is, we receive information from our sources (e.g. WEALTHYCLUSTER2), process it, and store it into a database. ``` [sessions] -> [processing engine] -> (database) <--> (user queries) Session phone numbers email addresses log ins user activity Database metadata tables full log QUERY ``` 8 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Data Flow - XKS Cluster * A cluster is comprised of one master server and one or more slaves. * All slaves in a cluster have their own copy of configurations (`/opt/xkeyscore/config`) files via the `xks rsync push_config` cronjob. TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL 9 --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Data Flow - Databases * There are two types of databases on an XKEYSCORE system: insert (i0) and query (q0) ``` D124 sessions file_input_proc process_data0 process_data1 SOTF sessions sotf_input_proc process_dataN xks_meta_ingester i0 register_metadata_tables q0 NOTE: sotf_input_proc is now called, sotf_dist process_dataN's are now called, process_data_parent ``` 10 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Data Flow - Databases * `file_input_proc` and `sotf_dist` take in sessions from the front-end and load balances them across multiple `process_data_parent`'s. * `process_data_parent` is responsible for processing sessions and extracting metadata * `xks_meta_ingester` takes the metadata from the `process_data_parent`'s and writes it to the insert database, `i0` * `register_metadata_tables` takes completed insert tables, indexes them, and moves them to the query database, `q0` 11 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## XKEYSCORE Architecture 12 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Lesson Objectives * Operating System Services * HTTPD * MYSQL * NFS * AUTOFS * Mount Points * `/xks_data` * Directory Structure 13 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Operating System Services * XKEYSCORE is typically installed on servers running Red Hat 5u8 operating system. * This section discusses common operating system services used during XKEYSCORE operation. 14 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## HTTPD * The http daemon is needed for the web-based GUI, viewing content, and is required on all servers. * The master server is the web server and the slaves retrieve content through HTTPS. 15 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## MySQL Daemon * The mysql daemon is a SQL-based database server for processing, querying, and is needed for the XKEYSCORE GUI. * It is required on all servers for administration, processing, and querying metadata in databases. 16 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## NFS * Mounting a directory uses the NFS service. * NFS allows file systems that physically reside on one computer to be shared by other computers on the network. * The machine with the hardware containing the directory must allow the hardware to be made available to other machines. * Required on all computers for clustering. 17 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## NFS * `/etc/exports` * `/export/data/xkeyscore` master(rw) slave(rw) * `/opt/xkeyscore/config/loadserver` \\*(rw) 18 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## AUTOFS * Computers requiring shared access to the `/export/data/xkeyscore` directory must be told where to find the directory. * This is accomplished via automounting. * The autofs daemon listens for computers trying to connect to the directories, or mounts, that it is responsible for. * The mounts are dropped after a time out, but autofs remounts the drive when drives need to be accessed. 19 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Automounting * For a clustered XKEYSCORE, automounts must be set up on all of the computers in the cluster. * `auto.master` and `auto.data` files in the `/etc` directory must be edited or created. * When finished, the mounted directories on the remote machines can be accessed. * The `oper` account should have full read/write permissions on all shared drives. 20 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Mount Points * `auto.master` – designates mount points on the local computer and the directory to mount on the remote server. * Example: * `/xks_data` `/etc/auto.data` `--timeout=60` * `auto.data` – enables all servers to see the `/export/data/xkeyscore` directory on other machines and locate databases, archived, data, and MAILORDER directory. * Example: * `xks1 -rw,soft,intr,tcp xks1:/export/data/xkeyscore` * `xks2 -rw,soft,intr,tcp xks2:/export/data/xkeyscore` 21 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Directory Structure * `/opt/xkeyscore/` – contains all of the XKEYSCORE software. Software includes the GUI, processing, scripts, and configurations. * `bashrc` – XKEYSCORE environment variables file. * `beacon/` – contains the beacon perl script (`shm_beacon.pl`) and a link to the beacon configuration file (`shm_beacon.config`). * `bin.shells/` and `bin.shells/sysadmin` – contains miscellaneous bash, python, and C shell scripts. * `build/` – contains libraries and plug-ins. * `install/` – contains installation scripts. 22 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Directory Structure * `/opt/xkeyscore/config/` – consists of sub-directories and each contain configuration files for building and running XKEYSCORE. * `crontab/` – contains the master and slave crontab file. * `dictionaries/` – contains the dictionary files for the filtering, selection, TRAFFICTHIEF, CADENCE, fist tables, and any other local dictionaries. * `misc/` – contains miscellaneous per-plug-in configuration files, (i.e. `sotf_input_proc.xml`). * `plugins/` – contains event handler configuration files for each of the plugins (`default.xml`). * `www/` – contains web configuration files and `xscore.cfg`. * `SERVICE/` – contains the config files for all the services needed by XKEYSCORE (httpd, php, mysqld, etc.) 23 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Directory Structure * `/opt/xkeyscore/www/` – contains the contents of the web front end. * `docs/` – contains documents viewable through the XKS GUI. * `html/` – contains web pages and scripts that are not on the secure server. * `secured/` – contains web pages and scripts that are on the secure server including: * `crons/` - location of cron job scripts * `src/` – contains source code for the XKS GUI. 24 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Directory Structure * `/export/data/xkeyscore/` – is used for both internal databases and metadata archive databases, input, output, and archiving of data. * `archives/` – (optional) destination for processed content * `inputs/` – (optional) used for file based input * `mysql/` – location of the MySQL database consisting of admin, insert, and query databases. * `outputs/` – (optional) contain the following sub-directories: * `mailorder/` - pickup point * `mailorder_working/` - file creation point before being moved to mailorder/ 25 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Directory Structure * `/xks_data/` - logical mount point for all other XKEYSCORE (including itself) `/export/data/xkeyscore`. * `/` - mount point for the hostname's local directory `/export/data/xkeyscore` (referenced by host name). * All servers must export their `/export/data/xkeyscore` directory and mount this on the `/` directory for each hostname of each machine, including itself. 26 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## XKEYSCORE GUI 27 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Lesson Objectives * Accessing the GUI * Exiting a Session * Main Menu Bar * MyXKS * Admin * Computer Resources Option * Start and Stop Processing * Run a Process Manually * Users * Search * Workflow Central * Results * Fingerprints 28 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Accessing the GUI * In the address field of a web browser, type `https://`. * PKI's or a UserID and password are required. After successfully launching a new session, the XKEYSCORE WELCOME window appears. * Note: Compatible web browsers for XKEYSCORE version 1.5 are: * Internet Explorer is not supported * Firefox/3.0.\\* and above 29 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Accessing the GUI ``` The system is accredited for USSID 18 and Human Rights Act compliance TOP SECRET//SI//REL TO USA, AUS, CAN, GBR, NZL//20320108 Home MyXKS Admin Users Search Workflow Central Results Fingerprints Statistics Map Help Navigation Filter Home Last Search Types Full Log LinkSummaryMapReduc... Category (DNI) Alerts Email Addresses MyXKS Full Log DNI HTTP Activity My Fingerprints My Overflows My Recent Results Profile Admin Processing Computer Resources Input Directories Category Throttle Databases Search DBs DB Registration Utilities Casenotation Reload Config Files News Ip Summary Table Crashlogger Startup Query Users User Accounts Clearences Privileges Send Email Users Online Search Search Wizard Anonymizer Tor Onion Address Su... Tor Routers Tor Software Downloa tor Users Survey Web Anonymizers Botnet XKEYSCORE Suggested achievements Created Workflow Learn How Histogram Learn How Exporting Meta data Learn How HUMAN RIGHTS ACT, USSID 18 AND USSID 9 All (SYSTEM) queries require a justification to ensure Human Rights Act (HRA), USSID 10 and USSID 9 compliance. Please enter information as prompted by the query interface. An audit trail has been established and will be searched as part of Menwith Hill Station's response to any complaint brought under HRA and as part of the USSID 10 and USSID 9 process. Please note that SENSITIVE TARGETING APPROVAL (STA) is required for HRA before submitting any query which includes terms specific to a person or company (eg name, address, identity details such as communications address, passport/bank account number) who EITHER (a) is defined as a UK, British Dependent Territory (BDT) or Second Party "person" or (b) is located in the UK, or a BDT or Second Party country. STA is also required for wildcard pulls that are inevitably going to retrieve a substantial proportion of such entries (e.g. wildcarding on a UK city code). Full legal guidance is available from the HRA Compliance Officer at Menwith Hill Station. ``` 30 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Main Menu Options * The main menu bar across the top of the window has menus that, when selected, each has additional options available in a drop down menu form. ``` Home MyXKS Admin Users Search Workflow Central Results Fingerprints Tagging Reports Statistics Tasking Map Help ``` 31 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Main Menu Options | OPTION | DESCRIPTION | | :------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Home | Returns to the main page. | | MyXKS | Can edit user settings, disable/enable access to databases, edit a search form search setting, and restore default settings. | | Admin | Computer resources, Input Directories, Category Throttle, Search DBs, and DB Registration settings. | | Users | Contains User Accounts, Clearances, Privileges, Send Email, Users Online, My Auditees, My Audit Logs, and All Audit Logs. | | Search | Provides different search query forms, such as email addresses, category, full log, and user activity. | | Workflow Central | Request, modify, and view standing queries that will execute at a specified time or interval. | | Results | Can search personal searches by date time, query type, query name, output table, and user. | | Fingerprints | Fingerprint builder and viewer. | | Map | Brings up Google Earth | | Help | Help Documentation, XK Forum, Account Maintenance, and About XKEYSCORE | 32 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Admin Menu | SUB-MENU OPTION | DESCRIPTION | | :------------------ | :------------------------------------------------------------------------------------------------------- | | Computer Resources | Allows for process configuration and management. | | Input Directories | Contains the configuration for file-based input directories. | | Category Throttle | Edit CADENCE quota limits by category and/or fist table. | | Search DBs | Configuration for query databases which are queried when a search is submitted. | | DB Registration | Contains the mapping from insert databases to query database. | | News | Add, modify, delete mandatory and home page News. | 33 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Computer Resources * The Processing->Computer Resources option from the ADMIN menu allows control of the entire daemon-styled, or continuously running, processes for XKEYSCORE. * Processes appears in a table following the convention: ``` ` xkey01 process_data_parent ``` TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL 34 --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Computer Resources 35 ``` Computer Resource Window Process Table TOP SECRET//SI//REL TO USA, A Home MyXKS Admin Users Search Workflow Central Results Fingerprints Statistics Map Help Navigation Filter Processing Computer Resources Input Directories Category Throttle Databases Search DBs DB Registration Utilities Casenotation Blacklist Reload Config Files News Ip Summary Table Crashlogger Startup Query Profiler Help Color Key STOPPED STOPPING STARTING RUNNING WON'T START UP STOPPED? APP LAUNCHER STOPPED STOPPING? APP LAUNCHER STOPPED STARTING? APP LAUNCHER STOPPED RUNNING? APP LAUNCHER STOPPED WON'T START UP? APP LAUNCHER STOPPED Computer Resources Help Add Actions App Launcher is Running ``` | Actions | Proc Host | Program Name | Program Arguments | Program PID | Commanded Status | Status | Datetime Started | Datetime Stopped | |------|---|---|---|---|---|---|---|---| | STOP | tlxksvr01 | GUId | | 7607 | RUN | RUN | 2012-12-03 16:44:49.0 | 2012-12-03 16:44:49.0 | | STOP | tlxksvr01 | query_proc | | 30965 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:32:47.0 | | STOP | tlxksvr01 | check_mailorder_site.php | | 31019 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:32:47.0 | | STOP | tlxksvr01 | xks_meta_ingester | | 31051 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:32:47.0 | | STOP | tlxksvr01 | clickstreamService.sh | | 31053 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:32:47.0 | | STOP | tlxksvr01 | query_dispatch | | 1311 | RUN | RUN | 2012-12-03 21:08:09.0 | 2012-12-03 21:08:07.0 | | STOP | tlxksvr01 | file_input_proc | | 31104 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:32:47.0 | | STOP | tlxksvr01 | xks_system_monitor | | 13986 | RUN | RUN | 2012-11-27 21:01:13.0 | 2012-11-27 21:01:13.0 | | STOP | tlxksvr01 | sotftoo124server | | 31108 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:32:47.0 | | STOP | tlxksvr01 | tomcat.sh | | 31124 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:32:47.0 | | STOP | tlxksvr01 | cadence_tasking_proc | --myfdi XYD --pddg IE --digraph X5 | 31136 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:32:47.0 | | STOP | tlxksvr01 | xks_server_stats | | 31138 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:32:47.0 | | STOP | tlxksvr01 | mailorder_proc | --copydir /export/data/xkeyscore/out... | 31140 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:32:47.0 | | STOP | tlxksvr01 | register_metadata_tables | --loglevel error | 31143 | RUN | RUN | 2012-11-27 17:33:57.0 | 2012-11-27 17:33:50.0 | 35 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Computer Resources * The `xks_app_launcher` process runs on all servers from the `inittab`. * It tells the computer which program to run by looking at its tasking host. * `/opt/xkeyscore/config/www/xscore.cfg` * The config file specifying the location of the tasking database. * Processes can be stopped, started, edited, or deleted from the Computer Resources window. 36 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Computer Resources * Add a new process – click **Add** * Edit a process – click **Stop** in the ACTION column, then click **Edit**. * Delete process – click **Stop** in the ACTION column, then click **Delete**. * Stop the App Launcher – disables the `xks_app_launcher` on every host. 37 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Resources – Color Conventions * Visual cues in the form of colors are used to help identify activities performed by XKEYSCORE and serve as status indicators for monitoring purposes. * **Red** – indicates processes have been stopped * **Green** – indicates processes are running * **Yellow** – indicates processes are starting * **Orange** – indicates processes are being stopped * **White** – indicates processes won’t start * Visual cues are also available in the COMMANDED STATUS and STATUS columns of the table. 38 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Resources – Start/Stop Processing * It may be necessary to stop or start processes for troubleshooting or for a graceful server restart. * Individual processes and programs – Click **Stop** in the ACTION column. To start it, click **Run**. * To stop all individual programs, select **ACTIONS->Start/Stop Resources**. Enter the program name in PROGRAMS field, then click OK. * Can use `xks proc` actions and commands to do the same function 39 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Resources – Start/Stop Processing * All Processing - select START/STOP Resources from the ACTIONS drop-down menu, leave the PROGRAMS and ON HOSTS fields to their defaults, click OK. * Specifying programs or hosts – select STOP or START, enter a wildcard expression such as `*` or `!` in the PROGRAMS or HOSTS field, and click OK. * Example: `process*` * Alternatively, in a terminal window can run: * `xks proc stop process*` 40 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Run a Process Manually * It may be necessary to run a process manually for troubleshooting purposes. To run a process manually: 1. Launch the GUI and log on as `oper` or `admin`. 2. Click **ADMIN > Processing > Computer Resources** 3. Click **Stop** in the ACTION column for the process. 4. Open a terminal window and `ssh` to the host running the process, as the user `oper`. 5. Type `ps -ef | grep ` to verify that the process is stopped. 6. Type `` * Example: ``` query_proc --loglevel debug ``` 41 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Users Menu ``` Home MyXKS Admin Navigation Filter User Accounts Clearances Privileges Send Email Users Online ``` * This menu is only accessible to users with system administration privileges. * An SA can add/modify user accounts, add groups, clearance levels, privileges, and email users from this menu. 42 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## MyXKS * From the main menu bar, click `MyXKS` to view your profile, accesses, privileges, auditors, settings, fingerprints, workflows, and recent results. * Right click on any search form name to add a shortcut for that search form. ``` Navigation Filter Full Log DNI HTTP Activity My Fingerprints My Workflows My Recent Results Profile Full Log DNI HTTP Activity My Fingerprints My Workflows My Recent Results Profile ``` 43 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Search Menu * From the main menu bar, click SEARCH. Menu options display in the vertical pane on the left. ``` Navigation Filter Search Wizard Anonymizer Tor Onion Address Survey Tor Routers Tor Software Downloads Tor User Survey Web Anonymizers Botnet Quantumbot QUANTUMBOT Table ROGUESTITCH ROGUESTITCH Rs Stolen Creds CNE ANIMALHOUSE Access Logs Active Passive Map CNE Triage V2 Connection Information Cottonchainsaw EYou Mail Server Logs Email Addresses Endpoint Attached Media Endpoint File Hashes Endpoint Related Sessions Endpoint Stats Endpoint Reenable Endpoint Stats Endpoint Related Counts Extracted Files Eyou Mail Logs Fourth Party Passive Map Full Log DNI HTTP Request Cookies IE Cookies Kaspersky PSP Key Loggers Keylogger MP TRII Access Logs Machine Information Machine Information Network Information Network Information Norton PSP Odnoklassniki Odnoklassniki Passwords in RDP Registry Registry UserAssist Registry Information Router Survey SCARFACE Network Information UtmaIDs Web Servers Windows User Security Identifier (SID) vBulletin Sessions CT Archive Known Password Atabmail logs Ebel logs Email Ffu Mfnt MS2 User Activity Ms2 Keys and User Activity Opera Mini Headers Parcel Tracking RAR Known Password User Activity and persistent evercookies Yemen mobile IPs Zamensis Zamensis and extracted files Zamensis and full-text vBulletin Sessions ``` 44 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Search Menu * When choosing a plugin type from the menu options, the only data searched is the data that was identified as a hit when the plugin was processed. | SUB-MENU OPTION | DESCRIPTION | | :-------------- | :----------------------------------------------------------------- | | Category DNI | Searches dictionary category hits. | | Full Log DNI | Searches all sessions received by XKEYSCORE. | | User Activity | Enables a user to search by a user’s activity. Example: a user can find a hotmail user’s msnMailToken | 45 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Search Menu * All searches are conducted on database tables where the results of the XKEYSCORE engine are stored. * Each row of a database table contains values from an individual session that was identified as a hit by XKEYSCORE when that plugin or microplugin processed the session. * Each search type query is related to a plugin or microplugin, which performs the metadata extraction. 46 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Search Menu - Details * Search details can be accessed from the Search status window by clicking **Details**. * **CURRENT SEARCH DETAILS** window displays and allows the user to watch a query run through the appropriate databases. * **RESULTS** link in the main menu bar can be used to display a list of all previous search results. * Queries operate in parallel on each host. 47 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Search Menu - Details Window ``` Query Status : oper_0489153001249071967_1 Pause Hide Finished Show Query Status tlxksvr01:qsummary/new tlxksvr02:qo/new tlxksvr02:qsummary/new tlxksvr03:qo/news tlxksvr03:qsummary/new tlxksvr04:qo/sent tlxksvr04:gosummary/now tlxksvr05:qo/new tlxksvr05:qsummary/now tlxksvr06:qo/now tlxksvr06:qsummary/new tlxksvr07:qo/new tlxksvr07:qsummary/sent tlxksvr08:qo/new tlxksvr00:qsummary/new tlxksvr09:qo/new tlxksvr09:qsummary/new tlxksvr10:qo/new tlxksvr10:qsummary/new tlxksvr11:qo/new tlxksvr11:qsummary/new tlxksvr12:qo/new tlxksvr12:qsummary/new tlxksvr13:qo/now tlxksvr13:qsummary/clone querying tlxksvr14:qo/new tlxksvr14:qsummary/now tlxksvr15:qo/new tlxksvr15:qsummary/finished tlxksvr16:qo/new tlxksvr17:qsummary/clone querying tlxksvr17:qo/new tlxksvr17: q0 | 0.373134% | tlxksvr18:qsummary/new tlxksvr19:qo/new tlxksvr qsummary/done_querying tlxksvr20:qo/new tlxksvr20:qsummary/new tlxksvr21:qo/new tlxksvr21:qsummary/new ``` 48 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Results * From the main menu bar, click `RESULTS` to retrieve the results of previous queries. * By changing the start and stop dates, queries performed between those dates can be viewed. * If the query name is known, it can be entered in the `QUERY_NAME` field. * If the USERID is known, it can be entered. * When complete, a window displays with the matching queries. 49 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## XKEYSCORE PROCESSES 50 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Lesson Objectives * XKEYSCORE Process Data Flow * Processing Programs * Query Processes * Other Processes * Cronjobs * crontab 51 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Back-End Process Data Flow ``` WC2 SOTF sotf_dist sotfinput_handler xscore_proc (process_data_parent) MAILORDER Insert Host File System and other formats file_input_proc Scan dirs for new files xs_task_db xks_meta_ingester i0 InsertDBs register_metadata_tables db_input_file_handler Web UI Master Host xs_web_db query_dispatch xs_web_db xscore_proc (query_proc) Query Host q0 QueryDBs ``` 52 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Processing Programs * Processing programs are the main processes that extract metadata from the traffic and then database the information in insert databases. | PROGRAM | DESCRIPTION | | :------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `file_input_proc` | Scans for new input files. (before processing moves the file to the .tmp directory of the input directory specified) | | `sotf_dist` | Listens for incoming SOTF sessions | | `process_data_parent` | Processes all new files discovered by `file_input_proc` or `sotf_dist`; optionally archives content and databases metadata. Parent process loads all dictionaries and starts up, then forks child processes which do the actual processing. | 53 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Processing – `process_data_parent` * This process replaces `process_data0` through `process_dataX` * The “parent” process starts up and loads all the dictionaries, and then “forks” child processes which actually do the processing * Parent acts similar to the `xks_app_launcher`, managing restarts for the children when they die * When dictionaries are modified, parent reloads them and restarts the children * `xks proc` will show an “X/Y” number next to `process_data_parent` * This is the number of children currently running, over the number that should be running (based on the `xks.config num_data_processors` setting) * `pdp` will show up yellow anytime X != Y and green when everything is running normally * This means when you first (re)start `pdp`, it will show yellow while it is loading the dictionaries, because none of the actual child `process_data`’s are running yet * `xks proc` will report extra or missing `process_dataX` with a PID of 0 * Can’t tell what PID missing `process_data` is suppose to have, because its managed by the parent now 54 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Query Processes * Query processes are processes that search and submit all necessary tables for the analysts queries. | PROGRAM | DESCRIPTION | | :--------------- | :------------------------------------------------------------------------------------------------------------ | | `query_dispatch` | Submits search jobs to search databases and propagates the status of the search and results back to the web server | | `query_proc` | Searches through all the necessary tables for the analysts queries. | 55 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Other Process * Other process which is run from the Application Launcher. * `mailorder_proc` – polls the `/export/data/xkeyscore/outputs/mailorder_working` directory by default. Then renames and moves mailorder files to `/export/data/xkeyscore/outputs/mailorder`. 56 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Other Process * `xks_meta_ingester` – streams metadata over socket. This process improves database performance. Instead of each `xscore_proc` writing to the database independently, they stream their metadata over socket to the `meta_ingester`, which combines it by plugin and writes to the database. * Reduces the number of connections to MySQL and gives better control over table size. 57 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Other Processes * `register_metadata_tables` – moves tables from processing database of XKEYSCORE system to query database. * Works against the `uber_index` table * `uber_index->table_name`, `base_table_name`, `join_table` * Base table – contains common information amongst tables (`full_log_xxx_xxxxxx` table) * Extension table – extends the base table * Registration process takes place in two phases: * Register all base tables * Register all extension tables that have had its base table registered 58 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Other Processes * `signal_acquisition_loopback` – process that feeds modified packets back into the system. * Front-end for packet recursion or any other process that feeds modified packets back into the system * Reinjects back to front-end – `xfip` * Process is completel independent 59 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Other Processes * `mpmr_server` – this is the map-reduce server for microplugins, which runs the “Reducer” portion of GENESIS v5 microplugins. * Runs outside the normal processing flow, and will not affect the rest of the system. * It has a telnet port (5850) just like an `xscore_proc`. 60 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Other Processes * `correlation_server_0` – in-memory map-reduce server for correlation engine. * Each machine has one `correlation_server`, and every `process_data_parent` connects to every `correlation_server` * `xscore_proc` – 8GB by default * uses port 4321 61 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Other Processes * `xks_comms_server` – a more efficient way to communicate with hosts within and outside an XKS cluster (not currently implement) * Automatically handles configuration for talking between slaves, master and overlord at site * Configuration is needed to connect to the “peer” on the path towards, other sites * Comms configuration lives in `$XSCORE_DIR/config/comms/comms.config` * Supports a “quality of service” which “fairly” distributes available bandwidth to the services that are using comms 62 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Other Processes * `xks_comms_server` * Allow and Peer rules have a “network” parameter which the comms systems uses to determine an “inside” and an “outside” in proxies. * Comms system will only accept connections from address ranges it has been specifically configured to allow. * Every between 2 comms servers connection should have: * “bandwidth_rule” on each side, name doesn’t matter but both rules should usually have same bandwidth cap * “allow” rule on one side with a reciprocal “peer” rules on the other side 63 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Other Processes * `xks_comms_server` * Example: If we have a site named “US-123” connecting to `xks-central` over a 1Mbps link, `US-123`’s config would be: ``` bandwidth[world] = 1Mbps peer[00] = address=xks-central.corp.nsa.ic.gov, port=2412, bandwidth=world, network=external ``` And `xks-central` would have: ``` bandwidth[us123] = 1Mbps allow[00] = address=xkey-master.us123, bandwidth=us123, network=internal ``` 64 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## GUI Processes * Other process which is run from the Application Launcher. * `GUId` – rescans content against fingerprints when a user clicks to view the content of a session. * `tomcat.sh` – web server used to host XKS GUI * `sotftod124server` – downloads sessions * Gets called from the GUId process * Works with any downloaded traffic that is SOTF 65 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Statistic Processes * Other process which is run from the Application Launcher. * `xks_server_stats` – sends to `xks_system_monitor` on Master and generates stats about the server itself. * CPU usage, memory usage, disk space, disk I/O, network traffic, etc. * Stats are fed to `xks_system_monitor` and the system monitor does magic with them. 66 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Statistic Processes * `xks_system_monitor` – collects stats messages from all over the system (front-end and back-end and the server itself) and summarizes them for forwarding. Optionally it can database stats locally. 67 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Cronjobs * XKEYSCORE uses a number of cron jobs to perform tasks. | CRONJOB | DESCRIPTION | | :------------------ | :--------------------------------------------------------------------------------------------------------------------------------------- | | `age_off_new.php` | Ages off metadata and content when the disk is near capacity, or when thresholds have been met. | | `xks update_dictionaries` | Pulls updates from various sources. | | `xks rsync push_config` | Copies the `/opt/xkeyscore/config` directory to the slaves. | | `rwc_post_to_pub.py` | Once an hour kicks off an update request | 68 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## CRONTAB * `Crontab` is the program used to install, uninstall or list the tables used to drive the cron daemon. * The `crontab` consists of * `age_off_new.php` * `xks update_dictionaries` * `xks rsync push_config` * `rwc_post_to_pub.py` 69 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## CRONTAB * `age_off_new.php` * Options: * `-debug` : extra debug statements in the output * `-info` : extra info statements in the output * `-task_db` : explicitly state that the machine is a task host * `-web_db` : explicitly state that the machine is a web host * `-nosleep` : use if you want to run now * This process ages off tables and archived data based on the settings in the `xks.config` file and the percentage of disk space used. 70 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## CRONTAB * `xks update_dictionaries` * This process pulls the necessarily files from various sources to update the dictionary. * Configure `/opt/xkeyscore/config/xks.config` ``` #[dictionaries] dictionary[0] = type=royale, \\ src=sftp://tssi_fvey:tssi_fvey@\\ xks-control/home/tssi_fvey/xks_dict_update.tar.gz, \\ dest=update/xks_dict_update.tar.gz, \\ action[0]="cd \\ $XSCORE_DIR/config/dictionaries/update;$XSCORE_DIR/config/\\ dictionaries/update/dup_install.pl >/dev/null 2>&1" dictionary[1] = type=cadence ``` 71 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## CRONTAB * `xks rsync push_config` * Transfers Master configurations to its slaves. * Excludes dot files, “httpd/logs”, loadserver/packages”, “httpd/log” * `force`: option to `xks` to force `push_config` when not on the master 72 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## CRONTAB * `rwc_post_to_pub.py` * The automatic `starProc` process is as follows: * Hour 1: master asks whoever (say `xks-control`) for an update, gets the rpm, installs it, there is much rejoicing. The slaves asks the master for the rpm at the same time the master asks `xks-control`, but obviously the master doesn’t have it, so nothing happens. * Hour 2: everyone asks for an update again, this time the master has the rpm, the slaves download it and install and there is much rejoicing. * The rpm is installed and `process_data_parent`’s are restarted as soon as the rpm is downloaded on a given machine. 73 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## DeepDive 74 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## DeepDive * What is a DeepDive? * Why DeepDive? * What does a DeepDive look like? * Front-End Processes * xFIP * Promoter 75 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## DeepDive XKEYSCORE * XKEYSCORE packet processing solution * XKEYSCORE’s software handles all packet processing * No upfront filtering prior to XKEYSCORE * XKEYSCORE “promoter” tries to promote richest/most interesting traffic * All Strong Selectors * Full take ASDF (User Activity metadata) * Subset of GENESIS signatures * List managed by XKEYSCORE team in concert with collection managers and site engineers * 20% - 30% of site traffic is fully processed and can be found via XKEYSCORE search * Typically does not include unknown or uninteresting protocols 76 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Why DeepDive? * Access to most relevant DNI data supporting SigDev and collection missions. Enables new mission capabilities (e.g., Correlation) * Session promotion can be synchronized and managed based on Genesis signatures, traditional tasking selectors and available resources * Provides better scaling * Drop unwanted data. Keep the rest and make decisions later and more accurately * Better control of the processing space * Instantiate new mission capabilities and dataflows quickly * Troubleshooting and monitoring made easier * Need access to “raw” packets to support new mission (e.g., Cyber, Bulk Crypt) * Sessions can be displayed as Packet Bundles like Wireshark 77 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## DeepDive * What does a DeepDive look like? * XKEYSCORE full-take session processor (Back End) * High speed packet ingest: an end-to-end solution * Intelligent filtering to vary the proportion of traffic retained ``` DEEPDIVE Front End Packet Splatter Mettlesome Promoter Defrag Back End Dictionary Scanner Plugins Microplugins Fingerprints Metadata Content Packets Partial Sessions Full Sessions ``` 78 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## XKEYSCORE Front-End Processes | What it's called | What it does | What it means | | :--------------- | :-------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------- | | `Packet Splatter` | Ingests packets (from files, from the network, from a capture card) in a variety of formats. | If it's a packet stream, it can probably be fed into a DEEPDIVE. | | `xFip` | Fast reassembly of TCP/IPv4, UDP/IPv4 streams\\*, and TCP/IPv6 and UDP/IPv6 streams\\*. | DEEPDIVE sessionizes everything before making a keep/drop decision. | | `METTLESOME` | Reassembly of streams from less common protocol stacks. | | | `Promoter` | Rule-based filtering of reassembled sessions, based on keyword, country code or appid/fingerprint. | DEEPDIVE intelligently chooses the most useful traffic for retention. | | `Defrag` | Fully rebuilds sessions\\*\\* | Enough content available to do full decoding/document descent at the Back End | 79 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xFIP * Packet bundles * Preserves original packets and packet order * Preserves information that is lost during sessionization * Original pcap available in the XKS Viewer * Packet API * Microplugins can iterate over raw packets * Microplugins can use information that is lost during sessionization * E.g. timestamps, flags, checksums * Packet fingerprints * Fired based on observations `xFip` has made * E.g. large sequence gaps, TTL variation 80 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Promoter * Filters sessions prior to back end processing * keywords, regex, country code, appids\* * SIGDEV: promotion rather than strong selection * Set the focus of the back end * traffic types of interest * regions of interest * legal/policy constraints * Set the width of the access aperture * promote 20% of 20 signals? * promote 100% of 4 signals? * Set the length of data retention * promote 20% and keep for 3 days? * promote 30% and keep for 2 days? ``` allow appid chat.* allow country_code PK block country_code US-US ``` 81 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks Script 82 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks Script * Usage * options * General Commands * Services * Actions * Options 83 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks Options * Usage: `xks [options] ` * Try `xks help ` to get help on a specific service or action * General commands: * `services` - list available services * `actions` - list available actions * `dependencies [invert]` - show service dependencies * `help [items]` - print help on services or actions * Services (specify one or more service names or ‘all’): * `start ` - start the specified services * `stop ` - stop the specified services * `restart ` - restart the specified services * `status ` - print the status of the specified services * `setup ` - setup/configure/fix the current xks install 84 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks Actions * Actions: * `accounts_report` - sends an email containing accounts usage to the specified users * `add_admin` - sets up a local Linux user to administer XKS * `change_db_password` - changes the XKS database user’s password and updates all references to it * `cluster` - cluster actions * `compile_genesis` - compiles GENESIS signatures * `disk_check` - get raid and disk status * `ext4_format` - format $XSCORE_DATA_DIR partition and convert to ext4 filesystem * `ext4_upgrade_contents` - convert to ext4 filesystem while preserving of $XSCORE_DATA_DIR (no formatting) * `fetch` - fetch a remote file * `force_register` - force metadata table registration * `info` - show cluster information * `install_slave` - install a slave machine in this cluster * `local_tagging` - checks and/or loads tagging file 85 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks Actions * Actions: * `monitor` - view XKS monitoring messages via activemq * `mpmr_register` - force `mpmr` table registration * `mysqls` - run a mysql script * `onall` - run a command on all machines in this cluster * `powertower` - configure or run a powertower command * `proc` - control XKS processes on this cluster * `query` - display query status or submit a query * `query_dispatch` - command line interface to the XKS query_dispatcher(s). * `rac` - access remote admin ports * `reload_dictionaries` - force running processes to reload dictionaries * `rsync` - push configs or files to slaves * `search_fields` - populates user settings with search fields * `show_config` - show values from xksconfig for specified keys * `switch` - query or rebalance data switch * `sync_accounts` - synchronize user accounts (except for classifications) * `tail` - view realtime xks logs * `tasking_dump` - print out the contents of the `xkTasking` and `xksTasking_voip` databases. 86 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks Actions * Actions: * `top` - display system performance * `update_dictionaries` - update all XKS dictionaries * `update_gui_help` - update the 'help' pull downs in GUI * `users` - display the users currently logged into the GUI * `version` - show XKS version information * `watchdog` - check and (re) start essential XKS processes. * `workflow` - manually submit a workflow 87 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks Options * Options: * `-verbose` : print extra information to the screen * `-debug` : used for debugging script problems 88 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL # xks – General Commands * Type: `xks help services` * This will list all available services: * `first` – initialization service that runs before all others * `virus_scanner` – sets up virus scanner, assuming tarballs are present. * `ftpd` – enables ftp on the master if mailorder is enabled * `distcc` – sets up distributed compiler service * `slash_proc` – setup optimal /proc parameters * `myricom` – handles installation and configuration 10GigE network cards * `home` – sets up the home directory for the xks user account * `gcc` – check there is a working compiler on the system * `upgrade` – updates configuration files when upgrading to a new version of xks * `bashrc` – sets up bash environment variables * `beacon` – sets up xks monitoring beacon based on xks.config * `tt` – checks connectivity to TRAFFICTHIEF server 89 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks – General Commands * Type: `xks help services` * This will list all available services: * `sendmail` – configures sendmail for use with xks * `role_files` – this service installs role-specific files * `issue` – sets up the DoD mandatory login warnings * `royale_with_cheese` – setups automatic updates * `ntpd` – configure ntp based on xks.config * `link_summary` – sets up xks link summary GUI * `nfsd` – sets up xks-specific nfs mounts * `server_certs` – sets up server certificates for SSL applications * `openoffice` – installs and configures OpenOffice for use in the xks GUI * `init_d` – sets up the xks init.d services * `resolver` – sets up resolver config * `php` – sets up PHP related stuff. Except php.ini * `httpd` – sets up xks-specific httpd configuration TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL 90 --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks – General Commands * Type: `xks help services` * This will list all available services: * `www` – sets up GUI configuration files * `voip` – sets up voip processing * `crond` – ensures xks can use cron and sets up xks cron jobs * `sshd` – configures the secure shell service for use with xks * `license` – checks for a valid license file and if one isn’t found prints a message * `syslog` – configures the syslog service for use with xks * all xks processes log to `/var/log/xks.log` * `dictionaries` – checks status of any configured dictionaries * `cluster_check` – checks network connectivity across the cluster * `autofs` – start, stop, restart automounts * `loadserver` – start, stop, and setup loadserver * `directories` – sets up directories used for xks * `auditd` – no help available 91 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks – General Commands * Type: `xks help services` * This will list all available services: * `ldap` – no help available * `mysqld` – sets up the mysql server for use with xks * `disks` – checks status of disk partition used by xks * `databases` – maintains database scheme consistency * `local_tasking` – reapplies local tasking if necessary * `workflows` – sets up xks default workflows * `category_throttle` – overrides default category throttle settings based on overrides specified in xks.config * `enrichment_tomcat` – sets up enrichment tomcat java application server * `plugin_setup` – populate plugin database tables from xml files, apply default plugin config specified in xks.config, apply overrides from xks.config, regenerate plugin config files from database * `crdb` – no help available * `tomcat` – sets up tomcat java application server * `clickstream` – sets up clickstream service 92 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks – General Commands * Type: `xks help services` * This will list all available services: * `file_input` – sets up directories and database entries needed for file-based input * `age_off_db` – synchronizes the database (`xs_task_db.age_off`) with `xks.config`’s settings for content and metadata. The values in the database will be unconditionally overwritten with those found in xks.config * `db_connectivity` – verifies connectivity to critical databases * `pdf` – sets up xpdf language packs * `ul_age_off` – sets the maximum data retention time to a little over an hour in UL mode. * `mDNSResponder` – sets up mDNSResponder for use with SOTF input * `app_launcher` – controls the xks app launcher, which is responsible for monitoring xks processes and starting/stopping them as commanded from the GUI * `processes_setup` – configures xks processes based on specifications in xks.config * `comms` – sets up the XKS communications system configuration * `advanced` – performs advanced configuration 93 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks – General Commands * Type: `xks help services` * This will list all available services: * `endace` – handles all the installation and configuration for Endace Dag packet capture cards * `last` – cleanup service that runs after all others 94 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks - Services * `start` * `xks start mysql` * `stop` * `xks stop httpd` * `restart` * `xks restart nfs` * `status` * `xks status autofs` * `setup` * `xks setup plugins` 95 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks - Actions * `xks onall ‘ps -ef | grep xscore | grep -v grep’` * `xks force_register` ``` [oper@tlxksvr01 run]$ xks force_register Forced update on xks_meta_ingester ``` * `xks rsync push_config -force` * Usage: `xks rsync [push_config|push_compiled|push_slaves|push] ` * `xks update_dictionaries` * Usage: `xks update_dictionaries [test|check|print|force|help]` * `xks version` ``` [oper@tlxksvr01 run]$ xks version 1.5.9-65 ``` * `xks info` ``` [oper@tlxksvr01 run]$ xks info site : Timberline - SV SIGAD : USF-790 PDDG : IE XKS version: 1.5.9-65 Master : tlxksvr01 Num slaves : 13 Input : file, sotf Config : managed_srtd ``` 96 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks - Actions * `xks query servers` ``` [oper@tlxksvr01 run]$ xks query servers tlxksvr02:q0 3a 14s 98n 54w 2012-12-05 16:07:22 (top) tlxksvr03:q0 230a 0s 0n 0w 2012-12-05 17:55:02 (top) tlxksvr04:q0 230a 0s 0n 0w 2012-12-05 17:55:02 (top) tlxksvr05:q0 225a 0s 4n 0w 2012-12-05 17:55:02 (top) tlxksvr07:q0 230a 0s 0n 0w 2012-12-05 17:55:02 (top) tlxksvr08:q0 225a 0s 5n 0w 2012-12-05 17:55:02 (top) tlxksvr09:q0 0a 0s 173n 2w 2012-12-05 17:55:02 (top) tlxksvr10:q0 230a 0s 0n 0w 2012-12-05 17:55:02 (top) tlxksvr11:q0 230a 0s 0n 0w 2012-12-05 17:55:02 (top) tlxksvr12:q0 225a 0s 4n 0w 2012-12-05 17:55:02 (top) a=awaiting dispatch, s=sent, n=new, w=working timestamp shows earliest submitted but unfinished query current time: 2012-12-05 18:02:09 ``` 97 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks Example * `xks proc` ``` [oper@tlxksvr01 run]$ xks proc GUI GUId qp query_proc cli clickstreamservice.sh rmt register_metadata_tables cms check_mailorder_site.php s2d sotftod124server cs00 correlation_server_0 sab signal_acquisition_base ctp cadence_tasking_proc sal signal_acquisition_loopback enr enrichment-tomcat.sh sd sotf_dist file file_input_proc sst strong_selector_targeting mp mailorder_proc tom tomcat.sh mpmr mpmr_server xcs xks_comms_server pd# process_data# xmi xks_meta_ingester pdp process_data_parent xsm xks_system_monitor qd query_dispatch xss xks_server_stats Run 'xks proc full' to show full listing tlxksvr01 GUI cli cms ctp enr file mp qd qp rmt s2d sab:4/4 sst tom xcs xmi xsm xss tlxksvr02 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr03 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr04 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr05 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr06 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr07 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr08 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr09 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr10 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr11 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr12 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr13 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss tlxksvr14 cs00 mpmr pdp:4/4 qp rmt sab:4/4 sal:6/6 sd xcs xmi xss Process consistency check OK on all hosts ``` 98 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks Example * `xks proc full` ``` [oper@tlxksvr01 run]$ xks proc full app launcher status: RUN (pid 30705) id hostname program arguments commanded actual pid 14 tlxksvr01 cadence_tasking_proc --myfdi XYD --pddg IE --dig... RUN RUN 31136 4 tlxksvr01 check_mailorder_site.php RUN RUN 31019 723 tlxksvr01 clickstreamservice.sh RUN RUN 31053 654 tlxksvr01 enrichment-tomcat.sh RUN RUN 31200 9 tlxksvr01 file_input_proc RUN RUN 31104 1 tlxksvr01 GUId RUN RUN 9335 548 tlxksvr01 mailorder_proc --copydir /export/data/xkey... RUN RUN 31140 8 tlxksvr01 query_dispatch RUN RUN 17549 3 tlxksvr01 query_proc RUN RUN 30965 193 tlxksvr01 register_metadata_tables --loglevel error RUN RUN 31143 709 tlxksvr01 signal_acquisition_base -f generic_packet_to_bundle... RUN RUN: 4/4 31236 12 tlxksvr01 sotftod124server RUN RUN 31108 461 tlxksvr01 strong_selector_targeting RUN RUN 31145 13 tlxksvr01 tomcat.sh RUN RUN 31124 653 tlxksvr01 xks_comms_server RUN RUN 31148 5 tlxksvr01 xks_meta_ingester RUN RUN 31051 462 tlxksvr01 xks_server_stats RUN RUN 31138 11 tlxksvr02 register_metadata_tables RUN RUN 13906 724 tlxksvr02 correlation_server_0 --loglevel debug RUN RUN 13946 31 tlxksvr02 mpmr_server RUN RUN 7150 730 tlxksvr02 process_data_parent --max-mem 20 RUN RUN: 4/4 22661 30 tlxksvr02 query_proc RUN RUN 14754 187 tlxksvr02 register_metadata_tables --loglevel error RUN RUN 14994 710 tlxksvr02 signal_acquisition_base -f generic_packet_to_bundle... RUN RUN: 4/4 14996 41 tlxksvr02 signal_acquisition_loopback -f packet_aux.config -i loo... RUN RUN: 6/6 14990 259 tlxksvr02 sotf_dist RUN RUN 14845 679 tlxksvr02 xks_comms_server RUN RUN 14992 38 tlxksvr02 xks_meta_ingester RUN RUN 14970 473 tlxksvr02 xks_server_stats RUN RUN 14988 ``` 99 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks Example * `xks query` ``` [oper@tlxksvr01 run]$ xks query id user type search start search stop duration status 66250201 http_parser 00:00 12/2/12 00:00 12/6/12 00:00:10 ongoing 66250183 full_log 00:00 12/3/12 00:00 12/6/12 00:00:17 ongoing 66250155 full_log 00:00 12/4/12 00:00 12/6/12 00:00:40 ongoing 66250127 geo_info 00:00 11/30/12 00:00 12/6/12 00:01:16 ongoing 66250052 email_addresses 00:00 11/21/12 00:00 12/6/12 00:03:36 ongoing 66249873 full_log 22:00 12/3/12 21:59 12/4/12 00:11:31 ongoing 66249660 full_log 00:00 12/2/12 00:00 12/6/12 00:18:57 ongoing 66244233 category 00:00 11/5/12 00:00 12/6/12 00:42:17 ongoing 66244135 full_log 00:00 11/28/12 00:00 12/6/12 00:44:30 ongoing 66244009 http_parser 00:00 11/5/12 00:00 12/6/12 00:48:49 ongoing 66243967 http_parser 00:00 11/5/12 00:00 12/6/12 00:49:34 ongoing 66243855 document_metadata 00:00 11/21/12 00:00 12/6/12 00:50:48 ongoing 66243785 correlation 00:00 11/5/12 00:00 12/6/12 00:52:46 ongoing 66243463 correlation 00:00 11/21/12 00:00 12/6/12 00:56:13 ongoing 66243071 email_addresses 00:00 11/1/12 00:00 12/6/12 01:08:24 ongoing 66242973 user_activity_exif 00:00 11/21/12 00:00 12/6/12 01:12:03 ongoing 66242413 http_parser 00:00 11/28/12 00:00 12/6/12 01:26:32 ongoing 66242315 full_log 00:00 12/4/12 00:00 12/6/12 01:30:52 ongoing There are 18 queries in progress ``` 100 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks query detail ``` [oper@tlxksvr01 run]$ xks query detail id=66251065 Query Summary Userid: Type: email_addresses Searching from 00:00 11/30/12 to 00:00 12/6/12 Duration: 00:01:29 Priority: 5 Cancel: N(o) Max Results: 10000 Max Time: 6000 Query SQL Name: smstev3_4 Classification: S,IS/SI,NSA NOFORN,IS,HCS,S/SI,MUSCULAR,REL_USA,NSA_NOFORN,SI,C,R Where: WHERE datetime >= '2012-11-30 00:00:00' AND datetime <= '2012-12-06 00:00:00' AND email = '[REDACTED]' AND domain = 'hotmail.com' Query Status host database status tlxksvr01 q0 finished tlxksvr02 q0 ongoing tlxksvr03 q0 finished tlxksvr04 q0 finished tlxksvr05 q0 finished tlxksvr06 q0 finished tlxksvr07 q0 finished tlxksvr08 q0 finished tlxksvr09 q0 finished tlxksvr10 q0 finished tlxksvr11 q0 finished tlxksvr12 q0 finished tlxksvr13 q0 finished tlxksvr14 q0 finished ``` 101 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Monitoring 102 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Lesson Objectives * Executables * mysqls * onall * xks onall * xks monitor * sotf_stat * xks top * Web Status * Additional Monitoring 103 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Executables * System monitoring can be performed from the command line using the following executable commands: * `mysqls` * `onall` * `xks onall` * `sotf_stat` * `xks top` 104 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## mysqls * The `mysqls` bash shell script can be used to execute MySQL statements from in the `/opt/xkeyscore/bin.shells/sysadmin/mysqls` directory. The most commonly used options in `mysqls` are: * `status` – displays file-based input statistics. * `speed` – displays the total file based input processing rate (Mbps) * `speed1` – displays file-based input processing rate (Mbps) per input source. * `speed2` – displays file-based input processing rate (Mbps) per xkeyscore processing server. * `count` – displays the count of input files in the new, working, error, and done states. 105 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## mysqls * `mysqls status` ``` [oper@tlxksvr01 run]$ mysqls status status count(*) sum(filesize) priority bitrate_Mbps NULL ``` 106 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks onall * `xks onall ‘xks mysql status’` ``` [oper@tlxksvr01 run]$ xks onall 'xks mysql status' Do you want to execute "xks mysql status" onall? [y|n] y ---- tlxksvr01 ---- status mysqld mysqls is running ---- tlxksvr02 ---- status mysqld mysqls is running ---- tlxksvr03 ---- status mysqld mysqls is running ---- tlxksvr04 ---- status mysqld mysqls is running ---- tlxksvr05 ---- status mysqld mysqls is running ``` 107 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks monitor * This script will monitor your front-end processes. * Type: `xks montor` or `xks monitor h` to receive the help menu ``` ---XKEYSCORE-Menu--- Command Name Description c config Configure this utility d dataflow_all FrontEnd Dataflow Menu b dataflow_be BackEnd Dataflow Menu m,h menu View this menu a packet_splatter Packet Acquisition [Front End] p process_data Process Data [Back End] q quit Quit/Exit s servers Server Stats (CPU,IO,etc.) t sotf_input SOTF Input [Back End] 'xks top' replacement f xfip Sessionization [Front End] ``` 108 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks monitor * Type: `xks montor f` to receive `xfip` stats ``` --XKEYSCORE-Sessionization [Server View Press '4']-- Casenotation Rate(Mbps) Loading% TCPQuality% Rate(Pkts) Count(Pkts) Pkts% Fragments% 7DH115181980000 0.00 0.00 0.00 0.00 0 0.00 0.00 7DH115209040000 0.00 0.00 0.00 0.00 0 0.00 0.00 Total: 0.00 [KEY: STM1,STM4,STM16,STM64] ``` 109 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## sotf stat * The `sotf_stat` command is used to display the SOTF (streaming object transfer format) input statistics for an entire cluster. * The statistics include total number of `process_data`’s running on the cluster, session input rate (sessions/sec), total bytes input (Mbps), and total bytes output to `process_data(s)` (Mbps). 110 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## sotf stat * To execute the `sotf_stat` script: * Log on to the server and open a terminal window. * Type `sotf_stat` because the command is in the path * Type `s` to toggle the summary statistics view from total statistics to individual host statistics. * Type `q` to quit the program 111 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## sotf stat * The `sotf_stat` script lists the hostname, number of `process_data`’s currently running, Mbps, number of sessions, and number of bytes. ``` ---XKEYSCORE SOTF Statistics--- Hostname #In #OA/#OC Mbps In Sess In Bytes In Sess Q MaxBlk mhxkssvr02 7 4/4 18.66 410920147 4818112974976 0 0 mhxkssvr03 3 4/4 16.15 410121822 4783549865004 0 0 mhxkssvr04 3 4/4 16.65 410444622 4781320276992 0 0 mhxkssvr05 3 4/4 15.79 409831857 4759939303920 0 0 ---PRC: 15/ 15 Rate: 64.52 Mbps Sessions: 1641358767 Bytes: 19143289212772--- ``` 112 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## xks top * The `xks top` script lists the hostname, Mbps sotf rate, number of `process_data`’s running, the % of CPU, and % of IO wait. ``` hostname sotf #procs cpu% iowait% mhxkssvr01 -0.00 0 0.53 0.02 mhxkssvr02 21.08 4 12.88 7.94 mhxkssvr03 13.55 4 13.35 7.28 mhxkssvr04 14.97 4 14.50 8.63 mhxkssvr05 14.13 4 17.14 8.01 TOTAL 63.74 16 11.68 6.38 ``` 113 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Additional Monitoring * `xks tail` ``` [oper@mhxkssvr02 ~]$ xks tail Dec 5 13:27:29 mhxkssvr02 register_metadata_tables[13877] : NOTICE: cu Dec 5 18:28:29 mhxkssvr02 register_metadata_tables[13877] : Processing > Computer Resources** * Determine how many `process_data`’s are configured to be running on the specified host. 126 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Processing Problems continued… * How many `xscore_proc`’s are actually running on a host? * Log onto the XKEYSCORE server and open a terminal window. * Type `ps -ef | grep xscore | grep -v managed_` ``` [oper@tlxksvr02 ~]$ ps -ef | grep xscore | grep -v managed_ oper 22661 11569 15 Dec03 ? 07:36:24 xscore_proc --parent --base_port 5550 --external_appid compile --db i0 --renice_level -5 --command db_force_next --max-mem 20 --loglevel error --restart_childre n --childpid 31396 --childpid 29482 --childpid 29619 --childpid 29710 oper 29482 22661 26 19:08 ? 00:06:48 xscore_proc child-1 --port 5551 --external_appid compile --db i0 --renice_level -5 --command db_force_next --max-mem 20 --loglevel error oper 29619 22661 26 19:08 ? 00:06:31 xscore_proc child-2 --port 5552 --external_appid compile --db i0 --renice_level -5 --command db_force_next --max-mem 20 --loglevel error oper 29710 22661 26 19:09 ? 00:06:24 xscore_proc child-3 --port 5553 --external_appid compile --db i0 --renice_level -5 --command db_force_next --max-mem 20 --loglevel error oper 31396 22661 41 19:28 ? 00:02:13 xscore_proc child-0 --port 5550 --external_appid compile --db i0 --renice_level -5 --command db_force_next --max-mem 20 --loglevel error ``` 127 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Processing Problems continued… * `xks_app_launcher` is running, but not starting processes specified in the Computer Resources window? * This may indicate that the `xks_app_launcher` is defunct. Use the `kill` command to kill the `app_launcher` and its related sub-processes: * Type `pkill -f app_` * If a PID is not being specified, use the `pkill` command. The `-f` option kills all of the sub-processes. * Type `ps` to look for the new `xks_app_launcher` process. 128 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Processing Problems continued… * If, after performing the procedures, the `xks_app_launcher` is still not starting applications: * In a terminal window, manually run the problem process to see if there are any error messages. * The `xks_app_launcher` on any host is dependent on the access of the `xs_task_db.proc_resources` database table on the master. Verify that the specified host can access the master’s database and `/opt` directory. * On the slave system type `mysql xs_task_db` * `-h ` * performs a remote MySQL server login 129 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Processing Problems continued… * To test the `xscore_proc`, type: `telnet ` Optional commands to assist trouble shooting are: * `sbr` – prints the processing rate for the single xscore-proc. * `sh` – displays dictionary hit statistics. * `ss` – displays statistics on the internal plug-in processing rates. * `help` – there are many commands and can be described in the help menu. 130 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Processing Problems continued… * If the `process_data_parent` continues to deny access through the command port, and input still has not started processing, check the input source. * Run the process in a terminal window with the argument `--loglevel debug`, to view debug messages. * The command port also provides processing rates and statistics for troubleshooting performance issues, outages, and general administration issues. 131 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Outputs - Mailorder * `/export/data/xkeyscore/outputs/mailorder` * If there are no new files in the MAILORDER directory, MAILORDER may not be working properly. Possible causes are that: * Files are being written to the wrong directory or it is not configured properly * Permissions on the MAILORDER directory will not allow MAILORDER to move files 132 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Query - Dispatch * Query dispatch is the process that submits search jobs to search databases and propagates the status of the search and the results of the search back to the web server. * After submitting a new query, Search Status window displays a summary listing query name, date and time submitted, number of databases complete, and number of results. 133 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Query Problems * The query never moves to the finished state. * If a database outage or a comms outage occurs, results will not be reported from the single system. However, results from all other databases will return properly with the query results, but they will not appear in this state. 134 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Query Problems * Query job status is stuck in `awaiting_disbatch`. * If a status appears stuck in this state, the `query_dispatch` may not be running on the web server. To determine whether it is running: * Type `ps -ef | grep query_` * If the process is not running, restart it from the XKEYSCORE GUI or troubleshoot the `xks_app_launcher`. 135 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Query Problems * Another cause of this scenario is that a query database may have hung up the query dispatch process. Check the progress of queries on the query database hosts by viewing the table `sdb_query_jobs` in the query database, which tracks the status of queries: * Type `mysql q0` * Type `select status,count(*) from sdb_query_jobs where group by status;` * The select statement displays the current state of the queries on the query host. If many more queries appear in the new state when compared to other query databases, begin troubleshooting the problem `query_proc` on the specified query database. 136 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Query Processing * The query is in the sent state, but never appears in new. * After the `query_dispatch` process disbatches the query, the status is moved to `sent`. A query moves to the `new` state when the query has been placed in the query processing queue on the `query_host`. * If a query does not move to the `new` state in a reasonable amount of time, the connectivity of the database should be tested. 137 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Query Processing * To check the progress of queries on the query database hosts, view the table `sdb_query_jobs` in the query database, which tracks the status of queries: * Type `mysql q0` * Type `select status,count(*) from sdb_query_jobs where group by status;` * The select statement displays the current state of the queries on the query host. If many more queries appear in the `new` state when compared to other query databases, begin troubleshooting the problem `query_proc` on the specified query database. 138 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Query Processing * The query appears in the new state, but never finishes. * query is in the `new` state, has been received by the query host and placed in a queue waiting to be processed. * Queries can become backlogged with a large number of queries waiting in the `new` state, though the `query_proc` is processing the queries properly. It is hard to predict the time to work off a query backlog, but using the following `select` statement the status of queries for the current day can be checked for processing trends. 139 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Query Processing * To display the number queries in each state for the current day: * Type ```sql select status, count(*),datetime_submitted, (UNIX_TIMESTAMP(now())- UNIX_TIMESTAMP(datetime_submitted))/3600 from sdb_query_jobs where (datetime_submitted>(now() - INTERVAL '1' DAY)) group by status; ``` * To display the number of queries processed per hour for the current day: * Type ```sql select status,count(*)/24 AS queries_per_hour from sdb_query_jobs where cancel!="C" and (datetime_submitted>(now() - INTERVAL '1' DAY)) AS Backlog group by cancel; ``` * If processing properly, queries can take hours, if not days, to complete based on the backlog and the processing trends. 140 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Retrieving Metadata and Content * Queries complete but there are no results. * If queries complete, but no results are visible, verify that the date range of the query coincides with the collection date of the data. If using test data, test the query system by putting the start date range at a year or two older to assure it is not old test data. * Verify that query metadata is in the query database by checking the contents of the `/export/data/xkeyscore/mysql/{query_db}/` directory. 141 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Retrieving Metadata and Content * Queries complete and metadata returns, but there is no content. * The metadata in the XKEYSCORE viewer displays the host and directory path of the content file. Verify the content file exists using the `ls -l` command. Trace a dataflow issue if the file does not exists. If the content file exists, confirm the `httpd` daemon is started on all slave systems. To confirm the `httpd` daemon: * 1. Type `su - oper` * 2. Type `xks status httpd` * 3. If the daemon is not on, type `xks start httpd` 142 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Query Results * To troubleshoot problems with metadata or content from a query, it will be necessary to retrieve the actual content, since recreating the problem is very difficult. This can be accomplished from the XKEYSCORE GUI. Click RESULTS and begin a search of the questionable queries. 143 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL ## Questions? 144 TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL