SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL # XKEYSCORE Workflows 05 March 2009 SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL DERIVED FROM: NSA/CSSM 1-52 DATED: 20070108 DECLASSIFY ON: 20320108 --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## What is a workflow? * Workflows automate queries. * One-time * Standing * Every search type can be a workflow. * Same functionality and capability * Follow on actions * Email alert * Download actions * Metadata summary SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Who can submit a workflow? * Anyone! * One owner per workflow * Multiple-users can be notified * If ownership needs to be changed, a ticket can be submitted to the team. * Future: sharing workflows * Right now, only the owner has the results in their "My Results" view. SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## What can I do with a workflow? * Workflows can be configured to run once * Workflows can be configured to run daily * Every 1, 2, 3, 4, 6, 8, 12 or 24 hours * You can set an offset to start running at a certain hour * Download results * Email results and email alerts * MAILORDER results * MySQL report SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Why do I want a workflow? * XKEYSCORE has a rolling buffer of data * Repetitive queries * Sigdev purpose * Fingerprint and appid testing * Queries take a long time during high times * Follow on actions * Google Earth data * Statistics * Customizable – write a script! SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## How do I setup a workflow? ``` This system is audited for USSID 18 and Human Rights Act compliance TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, and NZL//20320108 XKEYSCORE Welcome: switch users Navigation Menu Explorer Home Workflow Central Request My Workflows Search Classic MultiSearch Classic A-M Classic N-Z Common Category DNI Document Metadata Email Addresses UserActivity VoIP Wireless Results My Recent Results My Previous Results My Ongoing Results My Downloads Statistics Link Summarization Tagging Local Tagging Tech Extractor Tagging Preferences Help Welcome to the New XKEYSCORE Home Page! If you have questions or bug reports go to XKEYSCORE New GUI Forum To use the old GUI, click here HUMAN RIGHTS ACT, USSID 18 AND USSID 9 I (SYSTEM) queries require a justification to ensure Human Rights Act (HRA), USSID 18 and SSID 9 compliance. Please enter information as prompted by the query interface. An audit all has been established and will be searched as part of Menwith Hill Station's response to y complaint brought under HRA and as part of the USSID 18 and USSID 9 process. ease note that SENSITIVE TARGETING APPROVAL (STA) is required for HRA before submitting y query which includes terms specific to a person or company (eg name, address, identity etails such as communications address, passport/bank account number) who EITHER (a) is efined as a UK, British Dependent Territory (BDT) or Second Party "person" or (b) is located in e UK, or a BDT or Second Party country. STA is also required for wildcard pulls that are evitably going to retrive a substantial proportion of such enties (e.g. wildcarding on a UK city de). Full legal guidance is available from the HRA Compliance Officer at Menwith Hill Station. This system is audited for USSID 18 and Human Rights Act compliance TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, and NZL//20320108 ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## How do I setup a workflow? First, s a workflo ``` Workflow Central Request Wizard Please select a Search Type. Full Log Every session collected, indexed by "standard" DNI meta-data (to/from IP, port, casenotation, application id, sigad, etc). Search Type Help Cancel Prev Next Submit ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## How do I setup a workflow? ``` Workflow Central Request Wizard Basic Information Query Name: Find_my_appid Query Justification: Testing appid signature Additional Justification: Miranda Number: Datetime: 1 Day Start: 2009-03-04 00:00 Stop: 2009-03-05 23:59 Reccurring Search | One Time Search Basic Features Help Runs once over a set datetime range Cancel Prev Next Submit ``` ring or one- st be unique per user must have a justification justifications SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## How do I setup a workflow? Selec nt to searc Select a field to search or every field, ou must select e PLUS key ``` Workflow Central Request Wizard Add Search Fields Search Values are ANDed by default. To OR Search Fields: * Use the Multiple Field Search tab (below the input fields). * Select all the fields you wish to search. To OR Search Values: * Type 'OR' between each value (no quotes). See Search Value Help below for more details or for a description of boolean logic go to here. Search Field Search Value Remove From IP Address OR To IP Address 1.2.3.4 Attribute Info From IP Address To IP Address From Port To Port Single Field Search | Multiple Field Search Search Value Help Cancel Prev Next Submit ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Group by option * Group b ta results. * Red * Retu ``` Workflow Central Request Wizard Group Search Fields Would you like to group any fields? No Yes Group By Type Table Unique Values: Global Unique Values: Group By Type Help Columns to Group By Datetime: Client IP (X-Fowarded-For): Username: Attribute Info: From IP Address: To IP Address: From Port: To Port: From Country (IP): To Country (IP): From City (IP): To City (IP): From Latitude (IP): This option groups each and concatenated. Select the fields you want to group by. Cancel Prev Next Submit ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Select databases ``` Workflow Central Request Wizard Select the Database(s) to query xks- :q0(xks :qo) xks- :qsummary (xks- qsummary) Content must exist Check All Uncheck All Basic Features Help If this is selected, results are only returned if the content still exists at site. Cancel Prev Next Submit ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Follow on Actions * All * All ntent) to another loca * All ``` Workflow Central Request Wizard Follow-on Actions Would you like to add any follow on actions No Yes Script Script Arguments Add Email Alert Email Alert SQL Report Download Sessions Script Arguments Email To: ROWR: Return Only With Results mpleted. ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Email alert Comma delimited email addresses. This option only sends an email if you workflow has results. ``` Workflow Central Request Wizard Follow-on Actions Would you like to add any follow on actions No Yes Script Email Alert Script Arguments Email To: ROWR: Return Only With Results Add ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## SQL report ``` Workflow Central Request Wizard Follow-on Actions Would you like to add any follow on actions No Yes Script SQL Report Script Arguments Type: Email To: Email Subject: Email Content: Email Attachment: Email Attachment ROWR: Return Only With Results Filename: Mail Order Trigraph: SQL: SELECT FROM%{OUTPUT_TABLE} WHERE GROUP BY GZIP: Compress Contents Add Cancel Prev Next Submit CSV or HTML This must be a VALID SQL statement. Email metadata that a user can set. Example. SELECT casenotation, sigad FROM %{OUTPUT_TABLE} WHERE sigad!='' GROUP BY casenotation ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Download Results ``` Workflow Central Request Wizard Follow-on Actions Would you like to add any follow on actions No Yes Script Download Sessions Script Arguments User ID: Email To: Email Subject: Email Content: ROWR: Return Only With Results Filename: Mail Order Trigraph: GZIP: Compress Contents Send To Agility: Send To Agility Add Cancel Prev Next Submit ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## You're almost done! ``` Workflow Central Request Wizard Workflow Review This query (Find_my_appid) will search the Full Log table in database(s): xks-jychan:q0 The query will run CONTINUOUSLY executing every 6 hours beginning at 5:00 EST The query will execute the following search criteria: From IP Address 1.2.3.4 To Port 80 AppID (+Fingerprints)* search/google* Workflow Values Workflow XML Cancel Prev Next Submit ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Workflow Pending ``` This system is audited for USSID 19 and Human Rights Act compliance TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL//20320108 XKEYSCORE Welcome: jychan switch users Home Workflow Central Seach Results Statistics Tagging Preferences Help Navigation Menu Explorer Home Workflow Central Request My Workflows Search Classic MultiSearch Classic A-M Classic N-Z Common Category DNI Document Metadata Email Addresses Extracted Files Full Log DNI HTTP Activity Phone Number Extractor User Activity Dictionary Hits File Transfer MultiSearch IP Addresses Mac Addresses Username Network Management Search Wizard User Activity VoIP Wireless Results My Recent Results My Previous Results My Ongoing Results My Downloads Statistics Link Summarization Tagging Local Tagging Tech Extractor Tagging My Workflows Help Actions Query Type Query Name Last Modified State Actions full_log Find_my_appid 2009-03-05 14:14: 5 Pending State Actions pending ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- ## Workflow Approved ``` This system is audited for USSID 18 and Human Rights Act compliance TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, and NZL//20320108 XKEYSCORE Welcome: jychan switch users Home Workflow Central Seach Results Statistics Tagging Preferences Help Navigation Menu Explorer Home Workflow Central Request My Workflows Search Classic MultiSearch Classic A-M Classic N-Z Common Category DNI Document Metadata Email Addresses Extracted Files Full Log DNI HTTP Activity Phone Number Extractor User Activity Dictionary Hits File Transfer MultiSearch IP Addresses Mac Address Username Network Management Search Wizard User Activity VoIP Wireless Results My Recent Results My Previous Results My Ongoing Results My Downloads Statistics Link Summarization Tagging Local Tagging Tech Extractor Tagging My Workflows Help Actions Query Type full_log Workflow: Find_my_appid 1 1236264295 [REDACTED] [REDACTED] 18837b706121a0ca full_log Find_my_appid Testing appid signature 6 5 fm_ip 1.2.3.4 to_ap 80 fingerprint search/google* to_ip unique key(to_ip) true xks-jychan:q0 Cancel Save/Submit Page 1 of 1 Page Size: 30 Displaying 1 - 1 of 1 This system is audited for USSID 18 and Human Rights Act compliance TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, and NZL//20320108 ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Common mistakes * From IP and To IP with the same value. * In this view, terms are ANDed together. * Use Multiple Field Search Tab. ``` Workflow Central Request Wizard Add Search Fields Search Values are ANDed by default. To OR Search Fields: * Use the Multiple Field Search tab (below the input fields). * Select all the fields you wish to search. To OR Search Values: * Type 'OR' between each value (no quotes). See Search Value Help below for more details or for a description of boolean logic to go here. Search Field Search Value Remove From IP Address OR To IP Address 1.2.3.4 Attribute Info From IP Address To IP Address From Port To Port Single Field Search | Multiple Field Search Search Value Help Cancel Prev Next Submit ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- ## Common mistakes * Using the multiple field search does not break this up into 3 search<->value pairs. * Enter each term separately in the singe fieldsearch. ``` Workflow Central Request Wizard Add Search Fields Search Values are ANDed by default. To OR Search Fields: * Use the Multiple Field Search tab (below the input fields). * Select all the fields you wish to search. To OR Search Values: * Type 'OR' between each value (no quotes). See Search Value Help below for more details or for a description of boolean logic to go here. Search Field Search Value Remove From IP Address 1.2.3.4 To IP Address 5.6.7.8 From Port 80 Single Field Search | Multiple Field Search Search Value Help Cancel Prev Next Submit ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Common mistakes * This will return ALL casenotations. * "a" will be defeated by "!a" but "a" does equal "!b" * All the defeated values must be ANDed together. ``` Workflow Central Request Wizard Add Search Fields Search Values are ANDed by default. To OR Search Fields: * Use the Multiple Field Search tab (below the input fields). * Select all the fields you wish to search. To OR Search Values: * Type 'OR' between each value (no quotes). See Search Value Help below for more details or for a description of boolean logic to go here. Search Field Search Value Remove Casenotation !a Casenotation !b Casenotation !c Casenotation !d Single Field Search | Multiple Field Search Search Value Help ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Common mistakes * If you are selecting specific SIGADs, only select the sites that have data from that SIGAD. * Queries will return faster. * All sites selected * Less work for the system. ``` Workflow Central Request Wizard Add Search Fields Search Values are ANDed by default. To OR Search Fields: * Use the Multiple Field Search tab (below the input fields). * Select all the fields you wish to search. To OR Search Values: * Type 'OR' between each value (no quotes). See Search Value Help below for more details or for a description of boolean logic to go here. Search Field Search Value Remove Casenotation !c Casenotation !d SIGAD AUC-993 Select the Database(s) to query =AUS sites =F6 sites =NZ sites Content must exist Check All Uncheck All Basic Features Help ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- ## Common mistakes * If you select the SQL Report option, make sure you put a valid SQL statement! SQL statement filled in: SELECT casenotation, count(\*) FROM %{OUTPUT_TABLE} WHERE casenotation!='' GROUP BY casenotation ``` Workflow Central Request Wizard Follow-on Actions Would you like to add any follow on actions No Yes Script SQL Report Script Arguments Type: CSV Email To: analyst@work.com Email Subject: My Workflow Results Email Content: Bad SQL - empty Email Attachment: Email Attachment ROWR: Return Only With Results Filename: Mail Order Trigraph: SQL: SELECT casenotation, count(*) FROM %{OUTPUT_TABLE} WHERE casenotation!='' GROUP BY casenotation GZIP: Compress Contents Add Cancel Prev Next Submit ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Questions? xks_workflow@r1.r.nsa SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL