SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL # XKEYSCORE Workflows 19 September 2011 DERIVED FROM: NSA/CSSM 1-52 DATED: 20070108 DECLASSIFY ON: 20320108 SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## What is a workflow? * Workflows automate queries. * One-time * Standing * Every search type can be a workflow. * Same functionality and capability * Follow on actions * Email alert * Download actions * Metadata summary SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Who can submit a workflow? * Anyone! * One owner per workflow, but using follow-on actions: * Multiple-users can be notified of results and/or sent summary information * Result table can be automatically shared * If ownership needs to be changed, a ticket can be submitted to the team. SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## What can I do with a workflow? * Workflows can be configured to run once * Workflows can be configured to run daily * Every 1, 2, 3, 4, 6, 8, 12 or 24 hours * You can set an offset to start running at a certain hour * Download results * Email results and email alerts * MAILORDER results * MySQL report SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Why do I want a workflow? * XKEYSCORE has a rolling buffer of data * Repetitive queries * Sigdev purpose * Fingerprint and appid testing * Queries take a long time during high times * Follow on actions * Google Earth data * Statistics * Customizable – write a script! SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## How do I setup a workflow? * Two main ways * Based on the results of a recent query * Simplifies the process & more likely to produce the desired result! * This is done by right-clicking on the result set from the desired query and selecting **Create Workflow from this Search**. This populates the Workflow Wizard with the same criteria that was used by the selected query. * From scratch using the Workflow Wizard * Not recommended – but we'll show you anyway SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## How do I setup a workflow? * The next ten slides demonstrate how to step through the workflow wizard from scratch * But if you create the workflow from an existing query result many of the steps will already be correctly populated! ``` Right click to get the menu and choose this option My Recent Results Help | Actions View ``` | Query Name | Num Results | Num DBs | Datetime Submitted | Query ID | |-----|-----|-----|----|---| | byz rapt rdp india both dir | 3006 | 51 of 51 | 2011-09-19 17:01:44 | jb_e00b | | byz rapt rdp class c | 132 | 49 of 51 | 2011-09-19 16:38:07 | jb_e00b9 | | byz rapt rdp | 98 | 49 of 51 | 2011-09-19 16:35:36 | jb_e00b9 | | ww bb pin imsi no five eyes (WORKFLOW) | 9055 | 54 of 57 | 2011-09-19 08:55:57 | xml_job_2 | | afghan pin imsi correlation (WORKFLOW) | 7124 | 12 of 12 | 2011-09-18 23:55:19 | xml_job_2 | ``` Result Grid Row Actions View Metadata View Metadata (New Window) Delete Row Rename Query Share Results... Repopulate this Search into Form Create Workflow from this Search Split Both Sides of Traffic Archive Results Result Grid Cell Actions Filter: Query Name Equal 'by z rapt rdp india b...' Filter: Query Name Not Equal to 'byz rapt rdp indie b...' Show Full Cell Value ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## How do I setup a workflow? ``` This system is audited for USSID 18 and Human Rights Act compliance TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL//20320108 XKEYSCORE Welcome: switch users Navigation Menu Explorer Home Workflow Central Request My Workflows Search Classic MultiSearch Classic A-M Classic N-Z Common Category DNI Document Metadata Email Addresses UserActivity VoIP Wireless Results My Recent Results My Previous Results My Ongoing Results My Downloads Statistics Link Summarization Tagging Local Tagging Tech Extractor Tagging Preferences Help Welcome to the New XKEYSCORE Home Page! If you have questions or bug reports go to XKEYSCORE New GUI Forum To use the old GUI, click here HUMAN RIGHTS ACT, USSID 18 AND USSID 9 I (SYSTEM) queries require a justification to ensure Human Rights Act (HRA), USSID 18 and SSID 9 compliance. Please enter information as prompted by the query interface. An audit all has been established and will be searched as part of Menwith Hill Station's response to y complaint brought under HRA and as part of the USSID 18 and USSID 9 process. ease note that SENSITIVE TARGETING APPROVAL (STA) is required for HRA before submitting y query which includes terms specific to a person or company (eg name, address, identity etails such as communications address, passport/bank account number) who EITHER (a) is efined as a UK, British Dependent Territory (BDT) or Second Party "person" or (b) is located in e UK, or a BDT or Second Party country. STA is also required for wildcard pulls that are evitably going to retrive a substantial proportion of such enties (e.g. wildcarding on a UK city de). Full legal guidance is available from the HRA Compliance Officer at Menwith Hill Station. This system is audited for USSID 18 and Human Rights Act compliance TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL//20320108 ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## How do I setup a workflow? First, s workflo ``` Workflow Central Request Wizard Please select a Search Type. Full Log Every session collected, indexed by "standard" DNI meta-data (to/from IP, port, casenotation, application id, sigad, etc). Search Type Help Cancel Prev Next Submit ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## How do I setup a workflow? ``` Workflow Central Request Wizard Basic Information Query Name: Find_my_appid Query Justification: Testing appid signature Additional Justification: Miranda Number: Datetime: 1 Day Start: 2009-03-04 00:00 Stop: 2009-03-05 23:59 Reccurring Search | One Time Search Basic Features Help Runs once over a set datetime range Cancel Prev Next Submit ``` ring or one- st be unique per user must have a justification justifications SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## How do I setup a workflow? Selec nt to searc Select a field to search or every field, ou must be select e PLUS key ``` Workflow Central Request Wizard Add Search Fields Search Values are ANDed by default. To OR Search Fields: * Use the Multiple Field Search tab (below the input fields). * Select all the fields you wish to search. To OR Search Values: * Type 'OR' between each value (no quotes). See Search Value Help below for more details or for a description of boolean logic go to here. Search Field Search Value Remove From IP Address OR To IP Address 1.2.3.4 Attribute Info From IP Address To IP Address From Port To Port Single Field Search | Multiple Field Search Search Value Help Cancel Prev Next Submit ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Group by option * Group b ta results. * Red * Retu ``` Workflow Central Request Wizard Group Search Fields Would you like to group any fields? No Yes Group By Type Table Unique Values: Global Unique Values: Group By Type Help Columns to Group By Datetime: Client IP (X-Fowarded-For): Username: Attribute Info: From IP Address: To IP Address: From Port: To Port: From Country (IP): To Country (IP): From City (IP): To City (IP): From Latitude (IP): This option groups each and concatenated. Select the fields you want to group by. Cancel Prev Next Submit ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL # Select databases * Choose the search databases you would like to use * Can use an alias for multiple databases * Prepopulated if created from an existing search ``` Workflow Central Edit Request Wizard TAO STAT Team (tao-stat:xs_web_db) TEC (tc1xks1.tec.ces.nsa:xs_web_db TEC DEEPDIVE (xksdd1.tec.ces.nsa:xs_web_db) TEC SSO DEEPDIVE NOFORN (ssoxksdd1:xs_web_db) TEC TURTLERACE (turtlerace:xs_web_db) Timberline SV (timberline-sv:xs_web_db) TURBULENCE at the TEC (turbotec:xs_web_db) TURBULENCE MHS live (TURBOPOUND) (turbopound:xs_web_db) Please only enable if necessary TURTLEALE MHS live system (turboale:xs_web_db) XKSVOIP1 NOFORN (xksvoip-nf:qo) XKSVOIP2 REL (xksvoip-rel:qo) Yakima Deep Dive (jacknife-dd:xs_web_db) Yakima mission system (jacknife:xs_web_db) Content must exist Basic Features Help Cancel Prev Next Submit ``` If this is selected, results are only returned if the content still exists at site. SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- ## Follow on Actions * Allows you alter your results * Allows u content) to another location. * Allows y * Allows y ``` Workflow Central Edit Request Wizard Follow-on Actions Would you like to add any follow on actions No Yes Script Email Alert SQL Report Download Sessions Find and Foward Voip Script Arguments Email To: ROWR: Return Only With Results Share Results: Share Results with users above An email is sent out once your workflow is completed. Setup a MySQL statement to alter your results Download your results to another location. Used to forward VoIP to NUCLEON ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Email alert Comma delimited email addresses. This option only sends an email if you workflow has results. This will make the results appear for all of the listed users ``` Workflow Central Edit Request Wizard Follow-on Actions Would you like to add any follow on actions No Yes Script Email Alert SQL Report Download Sessions Find and Foward Voip Script Arguments Email To: ROWR: Return Only With Results Share Results with users above Add ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## SQL report ``` Workflow Central Request Wizard Follow-on Actions Would you like to add any follow on actions No Yes Script SQL Report Script Arguments Type: Email To: Email Subject: Email Content: Email Attachment: Email Attachment ROWR: Return Only With Results Filename: Mail Order Trigraph: SQL: SELECT FROM%{OUTPUT_TABLE} WHERE GROUP BY GZIP: Compress Contents Add Cancel Prev Next Submit CSV or HTML This must be a VALID SQL statement. Email metadata that a user can set. Example. SELECT casenotation, sigad FROM %{OUTPUT_TABLE} WHERE sigad!='' GROUP BY casenotation ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Download Results ``` Workflow Central Request Wizard Follow-on Actions Would you like to add any follow on actions No Yes Script Download Sessions Script Arguments User ID: Email To: Email Subject: Email Content: ROWR: Return Only With Results Filename: Mail Order Trigraph: GZIP: Compress Contents Send To Agility: Send To Agility Add Cancel Prev Next Submit ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## You're almost done! ``` Workflow Central Request Wizard Workflow Review This query (Find_my_appid) will search the Full Log table in database(s): xks-jychan:q0 The query will run CONTINUOUSLY executing every 6 hours beginning at 5:00 EST The query will execute the following search criteria: From IP Address 1.2.3.4 To Port 80 AppID (+Fingerprints)* search/google* Workflow Values Workflow XML Cancel Prev Next Submit ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Workflow Pending ``` This system is audited for USSID 18 and Human Rights Act compliance TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL//20320108 XKEYSCORE Welcome: jychan switch users Home Workflow Central Search Results Statistics Tagging Preferences Help Navigation Menu Explorer Home Workflow Central Request My Workflows Search Classic MultiSearch Classic A-M Classic N-Z Common Category DNI Document Metadata Email Addresses Extracted Files Full Log DNI HTTP Activity Phone Number Extractor User Activity Dictionary Hits File Transfer MultiSearch IP Addresses Mac Addresses Username Network Management Search Wizard User Activity VoIP Wireless Results My Recent Results My Previous Results My Ongoing Results My Downloads Statistics Link Summarization Tagging Local Tagging Tech Extractor Tagging My Workflows Help Actions Query Type Query Name Last Modified State Actions full_log Find_my_appid 2009-03-05 14:14: 5 Pending State Actions pending ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Workflow Approved ``` This system is audited for USSID 18 and Human Rights Act compliance TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL//20320108 XKEYSCORE Welcome: jychan switch users Home Workflow Central Search Results Statistics Tagging Preferences Help Navigation Menu Explorer Home Workflow Central Request My Workflows Search Classic MultiSearch Classic A-M Classic N-Z Common Category DNI Document Metadata Email Addresses Extracted Files Full Log DNI HTTP Activity Phone Number Extractor User Activity Dictionary Hits File Transfer MultiSearch IP Addresses Mac Addresses Username Network Management Search Wizard User Activity VoIP Wireless Results My Recent Results My Previous Results My Ongoing Results My Downloads Statistics Link Summarization Tagging Local Tagging Tech Extractor Tagging My Workflows Help Actions Query Type full_log Workflow: Find_my_appid 1 1236264295 [REDACTED] [REDACTED] 18837b706121a0ca full_log Find_my_appid Testing appid signature 6 5 fm_ip 1.2.3.4 to_ap 80 fingerprint search/google* to_ip unique key(to_ip) true xks-jychan:q0 Cancel Save/Submit Page 1 of 1 Page Size: 30 Displaying 1 - 1 of 1 This system is audited for USSID 18 and Human Rights Act compliance TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL//20320108 ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- ## Common mistakes * From IP and To IP with the same value. * In this view, terms are ANDed together. * Use Multiple Field Search Tab. ``` Workflow Central Request Wizard Add Search Fields Search Values are ANDed by default. To OR Search Fields: * Use the Multiple Field Search tab (below the input fields). * Select all the fields you wish to search. To OR Search Values: * Type 'OR' between each value (no quotes). See Search Value Help below for more details or for a description of boolean logic go to here. Search Field Search Value Remove From IP Address OR To IP Address 1.2.3.4 Attribute Info From IP Address To IP Address From Port To Port Single Field Search | Multiple Field Search Search Value Help Cancel Prev Next Submit ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- ## Common mistakes * Using the multiple field search does not break this up into 3 search<->value pairs. * Enter each term separately in the singe fieldsearch. ``` Workflow Central Request Wizard Add Search Fields Search Values are ANDed by default. To OR Search Fields: * Use the Multiple Field Search tab (below the input fields). * Select all the fields you wish to search. To OR Search Values: * Type 'OR' between each value (no quotes). See Search Value Help below for more details or for a description of boolean logic go to here. Search Field Search Value Remove From IP Address 1.2.3.4 To IP Address 5.6.7.8 From Port 80 Single Field Search | Multiple Field Search Search Value Help Cancel Prev Next Submit ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Common mistakes * This will return ALL casenotations. * a will be deafeted by "!a" but a does equal "!b" * All the defeated values must be ANDed together. ``` Workflow Central Request Wizard Add Search Fields Search Values are ANDed by default. To OR Search Fields: * Use the Multiple Field Search tab (below the input fields). * Select all the fields you wish to search. To OR Search Values: * Type 'OR' between each value (no quotes). See Search Value Help below for more details or for a description of boolean logic go to here. Search Field Search Value Remove Casenotation !a Casenotation !b Casenotation !c Casenotation !d Single Field Search | Multiple Field Search Search Value Help ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Common mistakes * If you are selecting specific SIGADs, only select the sites that have data from that SIGAD. * Queries will return faster. * All sites selected. * Less work for the system. ``` Workflow Central Request Wizard Add Search Fields Search Values are ANDed by default. To OR Search Fields: * Use the Multiple Field Search tab (below the input fields). * Select all the fields you wish to search. To OR Search Values: * Type 'OR' between each value (no quotes). See Search Value Help below for more details or for a description of boolean logic go to here. Search Field Search Value Remove Casenotation !c Casenotation !d SIGAD AUC-993 Select the Database(s) to query =AUS sites =F6 sites =NZ sites Content must exist Check All Uncheck All Basic Features Help ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL Common mistakes * If you select the SQL Report option, make sure you put a valid SQL statement! SQL statement filled in: SELECT casenotation, count(\*) FROM %{OUTPUT_TABLE} WHERE casenotation!='' GROUP BY casenotation ``` Workflow Central Request Wizard Follow-on Actions Would you like to add any follow on actions No Yes Script SQL Report Script Arguments Type: CSV Email To: analyst@work.com Email Subject: My Workflow Results Email Content: Bad SQL - empty Email Attachment: Email Attachment ROWR: Return Only With Results Filename: Mail Order Trigraph: SQL: SELECT casenotation, count(*) FROM %{OUTPUT_TABLE} WHERE casenotation!='' GROUP BY casenotation GZIP: Compress Contents Add Cancel Prev Next Submit ``` SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Questions? xks_workflow@r1.r.nsa SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL