TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL # Advanced HTTP Activity Analysis 2009 TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Goal The goal of this training is to get you familiar with basic HTTP traffic and understand how to target and expliot it using X-KEYSCORE TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Agenda * ? TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## What is HTTP? HTTP stands for Hypertext Transfer Protocol and it’s the primary protocol for transferring data on the World Wide Web TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Why are we interested in HTTP? * facebook * Yahoo! * twitter * myspace.com a place for friends * CNN.com * Google Earth * mail.ru * Wikipedia The Free Encyclopedia * Gmail by Google BETA Because nearly everything a typical user does on the Internet uses HTTP TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Why are we interested in HTTP? * Almost all web-browsing uses HTTP: * Internet surfing * Webmail (Yahoo/Hotmail/Gmail/etc.) * OSN (Facebook/MySpace/etc.) * Internet Searching (Google/Bing/etc.) * Online Mapping (Google Maps/Mapquest/etc.) TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## How does HTTP work? * HTTP is comprised of requests from clients to servers and their corresponding responses * Many analysts are already familiar with the terms “client-to-server” or “server-to-client” collection (also referred to as “client side” or “server side” collection). TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## How does HTTP work? * A “Client” is usually referring to a Browser (like Firefox or IE) which is also referred to as the “User Agent” * The “Server” can also be referred to as the “web-server” or “origin-server” which is the machine that is storing the data that is being accessed (like a web-page, a map, an inbox, etc) TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Activity * HTTP activity comes in two types: Website.com Client-to-Server “requests” Server Server-to-Client “responses” Client User TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Activity * HTTP activity comes in two types: Website.com Server Internet Client User While there may be a variety of Proxies, Gateways or Tunnels in between the client and the server, traffic is always going in one direction or the other. TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Client vs. Server Side Traffic * How do you know which side you’re looking at? * Client-to-Server requests are generally small in size and are computers talking to other computers * They contain standard HTTP header fields like “Host:” “Accept:” “Connection” etc. TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Activity Examples Client-to-Server request: ``` TOP SECRET//COMINT//20320108 ID: sess_orig_proc Type: HTTP-GET Printer Friendly Version DNI Display | Raw Data | DNI Format Services GET /Hezbollah-Terrorism-Judith-Palmer-Harik/dp/1860648932 HTTP/1.1 User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/1.0.154.48 Safari/525.19 Referer: http://www.google.com.pk/search?hl=en&q=written books on hizbollah&btnG=Google Search&meta= Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 Accept-Encoding: gzip, deflate, bzip2, sdch Cookie: ubid-main=185-5525816-8765531 apn-user-id=P1YXY7QF1PUYQ5 Accept-Language: en-US,en Accept-Charset: ISO-8859-1,*,utf-8 Host: www.amazon.com Connection: Keep-Alive ``` TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Client vs. Server Side Traffic * Server-to-Client responses are generally larger in size and are what web-pages look like at the internet. * When you’re at a computer accessing the Internet, you’re only seeing Server-to-Client traffic. TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Activity Examples Server-to-Client Response: ``` ID: sess_orig_proc + Document information Type: HTTP Printer Friendly Version DNI Display Raw Data DNI Forma + HTTP Header Information Services Barca reinstates 6-point lead over Real Isfahan to exhibit expressionist art Home Page Iran Middle East Iraq Palestine Lebanon Turkey Persian Gulf Others US Asia/Pacific Africa Europe Americas Sci/Tech Health Kuwait government 'resigns' over economy Mon, 16 Mar 2009 19:07:16 GMT The Kuwaiti government has submitted its resignation to the county's emir amid a row over the premier's handling of the economic crisis. "The resignation has been submitted formally and it's up to the emir (ruler) to decide," Reuters quoted Nasser al-Duwailah, a parliamentarian, as saying on Monday. The resignation would further delay the approval of 5 billion dinars (USD 5.11 billion) rescue package which is to be injected to the Persian Gulf nation's economy to ease the impact of global financial crisis. The government has not commented on the report. Go Latest News Kuwait govern economy Childhood diet risk 'US-Russian pa shield row' Judges want M confiscated Leader pardon Ancient book Lieberman eyes ally Intelligent peop ``` Bonus question: Why are the images in this web-page missing? TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Activity * XKS HTTP Activity Meta-data differs greatly depending on which side of traffic we’re collecting * In nearly all cases it’s better to have client-to-server traffic TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Activity Client-to-Server ``` GET /search?tab=urdu&order=sortboth&q=musharraf&start=3&scope=urdu&link=next HTTP/1.1 Accept: */* Referer: http://search.bbc.co.uk/search?tab=urdu&order=sortboth&q=musharraf&start=2&scope=urdu Accept-Language: en-us Accept-Encoding: gzip, deflate User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Host: search.bbc.co.uk Cookie: BBC-UID=b479a5f4ad230a53063d513630203acb22584634a0e0b164c45f96efc054cf950Mozilla%2F4%2E0%20%28comp Cache-Control: max-stale=0 Connection: Keep-Alive X-BlueCoat-Via: 66808702E9A98546 Host search.bbc.co.uk URL Path /search URL Args tab=urdu&order=sortboth&q=musharraf&start=3&scope=urdu&link=next Search Terms musharraf Language en Browser Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Via 66808702E9A98546 Referer http://search.bbc.co.uk/search?tab=urdu&order=sortboth&q=musharraf&start=2&scope=urdu Cookie BBC-UID=b479a5f4ad230a5306d351630023cb2260463a40e0b164c45f9e6cf054cf950Mozilla%2F4%2E0%20%28comp ``` TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Activity Server-to-Client ``` Application Info Press TV - Kuwait government 'resigns' over economy HTTP Type response ID: sess_orig_proc + Document information Type: HTTP Printer Friendly Version DNI Display Raw Data DNI Forma + HTTP Header Information Services Barca reinstates 6-point lead over Real Isfahan to exhibit expressionist art Home Page Iran Middle East Iraq Palestine Lebanon Turkey Persian Gulf Others US Asia/Pacific Africa Europe Americas Sci/Tech Health Kuwait government 'resigns' over economy Mon, 16 Mar 2009 19:07:16 GMT The Kuwaiti government has submitted its resignation to the country's emir amid a row over the premier's handling of the economic crisis. "The resignation has been submitted formally and it's up to the amir (ruler) to decide," Reuters quoted Nasser al-Duwailah, a parliamentarian, as saying on Monday. The resignation would further delay the approval of 5 billion dinars (USD 5.11 billion) rescue package which is to be injected to the Persian Gulf nation's economy to ease the impact of the global financial crisis. The government has not commented on the report. Go Latest News Kuwait govern economy Childhood diet risk 'US-Russian pa shield row' Judges want M confiscated Leader pardon Ancient book Lieberman eyes ally Intelligent peop ``` TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Activity – HTTP Types * Meta-data will also tell you which side of traffic you’re looking at * Client-to-server has two main types: HTTP Type get HTTP Type post * Server-to-client has only one: HTTP Type response TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Activity – Get vs Post * A ‘GET’ is you requesting data from the server (most web surfing) * A ‘POST’ is you sending data to the server (i.e. signing in, filling out a form, composing an E-mail, uploading a file etc.) TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL ## Let’s break down the important parts of a client-to-server request TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Client-to-Server ``` GET /home.html Host: sample.website.com User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 (USG-25) Firefox/3.0.10 Accept: image/png,image/*;q=0.8,*/*;q=0.5 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive ``` First thing to note is the Host: line which tells you the name of the server that the client is requesting data from TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Host Field It’s important to note, that in many cases users think they’re at websites like www.yahoo.com, but behind the scenes data is coming from a number of different servers without the user knowing it: ``` GET /mc/modules/mr/abContacts?mcnumb=RIIDb59jm & jsrand=98037807 & rand=2127033459 HTTP/1.0 Accept: */* Accept-Language: fa Referer: http://us.mc575.mail.yahoo.com/mc/showFolder?.ylc=X3oDMTBucmNobGR0BFSTA2M5ODMwMTAyNwRhYWNkZxNx2dz?mid=1_21857_AERkxEIAANvjSi6wUQ7fiLz4fY&fid=Inbox&sort=date&order=up&startMid=36&filterBy= x-requested-with: XMLHttpRequest Accept-Encoding: gzip, deflate User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727) Host: us.mc575.mail.yahoo.com Cookie: MG d=IvAXlFvaYnFGmmIfzw3zBCVVRe2jUKZLwwyoKSrjxxG0XVYaJhF95dLsZ5C0x1eDlcTcaHS_vpi Y ``` Bonus question: What would the impact of this be in how you formulate your X-KEYSCORE queries using the Host field? TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Client-to-Server ``` GET /home.html Host: sample.website.com User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 (USG-25) Firefox/3.0.10 Accept: image/png,image/*;q=0.8,*/*;q=0.5 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive ``` Second the GET line tells you which files the user is requesting from the server. If you simply take that line and append it to the Host line you have the live public URL that the user is requesting: http://sample.website.com/home.html TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Client-to-Server ``` GET /example.php?region=iraq Host: sample.website.com User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 (USG-25) Firefox/3.0.10 Accept: image/png,image/*;q=0.8,*/*;q=0.5 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive ``` When the GET line has a ? mark in it, then the GET request is also passing information to the server. So in this case the client is requesting the file example.php but it’s also passing along a value that could have been entered by the user. TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## URL Lines When there is a ? mark in the URL line, then X-KEYSCORE is breaking it up into two parts. The first part is called the URL Path and the second part is called the URL Argument. | URL Path | URL Args | |----------|----------| | /search | tab=urdu&order=sortboth&q=musharraf&start=3&scope=urdu&link=next | Notice all of the “arguments” (each separated by &’s) in this URL: ``` GET /search?tab=urdu&order=sortboth&q=musharraf&start=3&scope=urdu&link=next HTTP/1.1 Accept: */* Referer: http://search.bbc.co.uk/search?tab=urdu&order=sortboth&q=musharraf&start=2&scope=urdu Accept-Language: en-us Accept-Encoding: gzip, deflate User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Host: search.bbc.co.uk Cookie: BBC-UID=b479a5f4ad230a53063d513630203acb22b84634a0e0b164c45f96efc054c950Mozilla%2f4%2e0%20%2bcom Cache-Control: max-stale=0 Connection: Keep-Alive X-BlueCoat-Via: 66808702E9A98546 ``` Bonus question: Any idea what the information that is being passed in the URL Argument in this example are for? TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Client-to-Server ``` GET /home.html Host: sample.website.com User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 (USG-25) Firefox/3.0.10 Accept: image/png,image/*;q=0.8,*/*;q=0.5 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive ``` The User-Agent line gives you information on what type of client is requesting the data. In this case, we can see that it was a Firefox 3.0 browser from a Windows NT 5.1 (XP) machine. TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL ## User Agents TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## User Agents The User Agent (also known as the “browser”) can be very valuable. While it can not be trusted to be absolutely unique, in many cases you can use it to unwind a proxy or multi-user environment. It can also help provide hints if the origins of the request came from a mobile device: ``` User-Agent: Mozilla/5.0 (SymbianOS/9.2; U; Series60/3.1 NokiaE63-1/100.21.110; Profile/MIDP-2.0 Configurat like Gecko) Safari/413 User-Agent: NokiaN72/5.0706.4.0.1 Series60/2.8 Profile/MIDP-2.0 Configuration/CLDC-1.1 User-Agent: iPhone Mail (5H11) ``` TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Client-to-Server ``` GET /home.html Host: sample.website.com User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 (USG-25) Firefox/3.0.10 Accept: image/png,image/*;q=0.8,*/*;q=0.5 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive ``` The various “Accept” lines instruct the server on the types of responses the client can accept back. TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL ## Let’s look at a simplified version of a HTTP request and response TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL --- ## What is Web (HTTP) Activity This shows how a person logs on to a webpage From Port 3434* (client) Click on http://www.hotmail.com GET Request To Port 80 (Server) MSN The client’s port can be any high-numbered port, 3434 is just an example ## What is Web (HTTP) Activity --- This shows how a person logs on to a webpage From Port 3434* (client) Click on http://www.hotmail.com GET Request To Port 80 (Server) To Port 3434* (client) "Welcome to Hotmail" HTTP Response From Port 80 (Server) MSN The client's port can be any high-numbered port, 3434 is just an example ## What is Web (HTTP) Activity --- This shows how a person logs on to a webpage From Port 3434* (client) Click on http://www.hotmail.com GET Request To Port 80 (Server) To Port 3434* (client) "Welcome to Hotmail" HTTP Response From Port 80 (Server) From Port 3434* (client) Email Address: me@hotmail.com Password: Admin123 POST to the Web server To Port 80 (Server) MSN The client’s port can be any high-numbered port, 3434 is just an example --- ## What is Web (HTTP) Activity This shows how a person logs on to a webpage From Port 3434* (client) Click on http://www.hotmail.com GET Request To Port 80 (Server) To Port 3434* (client) "Welcome to Hotmail" HTTP Response From Port 80 (Server) From Port 3434* (client) Email Address: me@hotmail.com Password: Admin123 POST to the Web server To Port 80 (Server) To Port 3434* (client) "Welcome to your Inbox/homepage" HTTP Response From Port 80 (Server) MSN The client's port can be any high-numbered port, 3434 is just an example --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Activity * Real traffic, however, can be a little more complicated. * Almost all web pages are built from multiple files. * For example, every single image or banner ad on a web page is a separate file that needs to be individually requested before the server that has the file can respond TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Activity – Real World * Let’s look at the “NSA Today” home page. ``` Dynamic Page -- Highest Possible Classification is TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL NATIONAL SECURITY AGENCY UNITED STATES OF AMERICA NSA DAILY ... the best minute you'll spend today Current Conditions Weather Information Daily's Hot Links Search | Searchlight | Ext. Connections | Orgs | Projects | EA Outages 3 Oct 2009 TAKE 2 XmedMedia Agency Mass Mailers Today's EVENTS Multivision Ads Technology Expo Weight Watchers @ NSA Daily "Need to Know" Past News » (U//FOUO) NSAG Hosts the CENTCOM Senior Enlisted Leader (U//FOUO) On 9 September 2009, NSA/CSS Georgia (NSAG) had the privilege of hosting a visit from Command Sergeant Major (CSM) Marvin Hill, Command Senior Enlisted Leader for United States Central Command (CENTCOM). FOR OFFICIAL USE ONLY (U//FOUO) Pictured: NSA Georgia CSM Lori Brown welcomes Command Sergeant Major Marvin Hill, Command Senior Enlisted Leader for United States Central Command. (U//FOUO) This was CSM Hill's first visit to NSAG. The first stop during his visit was with ITD at the helpdesk. SGM Jacobs, of ITD, provided the CSM an overview of our maintenance operations and communications hub. CSM Hill told the ITD personnel, "you keep doing what you do, my comms run through here....and I need my comms" NSA IN THE NEWS From the DIRECTOR LTG Keith B. Alexander; United States Army Mission Messages Top IAD News (U) Securing the ``` TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Activity – Real World * It looks like one page, but each of the images and banners are separate data files that your browser pieces back together TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Activity – Real World * In fact, to build the NSA Today home page it takes 34 separate files from 4 different servers * However, most people probably don’t notice, because the entire page loads in <300 milliseconds. * If we had a slow internet connection, we’d notice the images would initially be missing. TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Activity Real-Word ``` + Document information Type: HTTP Printer Friendly Version DNI Display Raw Data DNI Forma + HTTP Header Information Services Barca reinstates 6-point lead over Real Isfahan to exhibit expressionist art Home Page Iran Middle East Iraq Palestine Lebanon Turkey Persian Gulf Others US Asia/Pacific Africa Europe Americas Sci/Tech Health Kuwait government 'resigns' over economy Mon, 16 Mar 2009 19:07:16 GMT The Kuwaiti government has submitted its resignation to the country's emir amid a row over the premier's handling of the economic crisis. "The resignation has been submitted formally and it's up to the amir (ruler) to decide," Reuters quoted Nasser al-Duwailah, a parlametarian, as saying on Monday. The resignation would further delay the approval of 5 billion dinars (USD 5.11 billion) rescue package which is to be injected to the Persian Gulf nation's economy to ease the impact of the global financial crisis. The government has not commented on the report. Go Latest News Kuwait govern economy Childhood diet risk 'US-Russian pa shield row' Judges want M confiscated Leader pardon Ancient book Lieberman eyes ally Intelligent peop ``` Notice that all of the images are missing. They are all separate server-to-client responses and therefore completely separate “sessions” in X-KEYSCORE or PINWALE TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Activity – Real World * It’s important to note that not all of the data on one web-page came from the same server. * For example, most of the NSA Today home page come from home.www.nsa, but the image of the current weather conditions came from wk-admiral208.corp.nsa.ic.gov TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Activity – Real World * This happens all the time on the Internet. * The cnn.com home page, may have an ad on it that was from the Google ad server and etc. * And this does have an impact on our collection! TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Activity * This is the traffic path for building the NSA today home page home.www.nsa corpweb1.nsa siteworks.nsa wk-admiral208.corp.nsa.ic.gov User TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## HTTP Activity * What happens if we only have collection on one of the paths? home.www.nsa corpweb1.nsa siteworks.nsa wk-admiral208.corp.nsa.ic.gov User TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## What would that traffic look like? ``` GET /current.jpg Host: wk-admiral208.corp.nsa.ic.gov User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 (USG-25) Firefox/3.0.10 Accept: image/png,image/*;q=0.8,*/*;q=0.5 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive Referer: http://home.www.nsa/ If-Modified-Since: Thu, 08 Oct 2009 19:31:56 GMT If-None-Match: "d945-16c1-842db643" Cache-Control: max-age=0 ``` If we only saw this one GET request and not the other 33 required to build the NSA Today home page, would we be able to determine what the user was actually doing? TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## What exactly is that telling us? * First off, we know what file they are requesting. * They want current.jpg from the wk-admiral208.corp.nsa.ic.gov server. * That’s actually a live public URL (http://wk-admiral208.corp.nsa.ic.gov/current.jpg) * Do we have any indication why they wanted that image? Answer is yes! Look at the referer field. TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## What exactly is that telling us? * They were referred from http://home.www.nsa/ * The referer is in essence, telling you what site was “linking” to the new site. * Warning! The referer can act in misleading ways. TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Referer Field * The referer field is the address of the page that links to new GET request. * However, this link could have been automatic to the user. * I.e. in the case of the current weather image, the link was automatic and the user wasn’t even aware of the action TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Referer Field * The referer field could also indicate a user action. * For example, imagine we were on the NSA Today webpage and clicked the link to the SID Today page. * What would that traffic look like? TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Referer Field ``` GET / Host: sidtoday.nsa User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 (USG-25) Firefox/3.0.10 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive Referer: http://home.www.nsa/ Cookie: CFID=565238; CFTOKEN=66534796; CFGLOBALS=urltoken%3DCFID%23%3D565238%26CFTOKEN%23%3D66534796%26jsessionid%23%3Da830dba3a04b67ae6e351b7463444f72496d%23lastvisit%3D%7Bts%20%272009%2D10%2D09%2015%3A38%3A04%27%7D%23timecreated%3D%7Bts%20%272009%2D06%2D19%2010%3A27%3A23%27%7D%23hitcount%3D13%23cftoken%3D66534796%23cfid%3D565238%23; JSESSIONID=a830dba3a04b67ae6e351b7463444f72496d ``` TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Referer Field * Now we’re seeing a request go to host “sidtoday.nsa” with the referer from http://home.www.nsa * How can we tell from the traffic that the first automatic referer we saw for the current weather was any different from the user-generated referer we saw for the SID Today article? TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL ## Cookies! TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Cookies * Cookies are small pieces of text-based data stored on your machine by your web browser. * Almost all websites have cookies enabled and they have a variety of uses, including to help the web-site track the activities of their users. * Most analysts are probably familiar with “machine specific cookies” like the Yahoo B cookie * However cookies are used for a variety of reasons TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## What can cookies be used for? * Cookies can be used to authenticate a user. * For example in many cases, the “active user” for Yahoo web-mail traffic is seen encoded in the I= part of the cookie string. ``` Y v=1 n=4edg46n653aef l=eibj0d_10o0p838to (Yahoo login id: [REDACTED]) p=f2kvvsy0120000000 (Gender: female, Birth year: 1984, Postal code: [REDACTED]) iz= r=jb lg=en-US ( Language/content: English ) intl=us ( Country: United States ) ``` TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## What can cookies be used for? * Cookies can be used to store information about the user that the website is interested in * Look at how the p= value below tells the website information about the user of this account: ``` v=1 n=4edg46n653aef l=eibj0d_10o0p838to (Yahoo login id: [REDACTED]) p=f2kvvsy0120000000 (Gender: female, Birth year: 1984, Postal code: [REDACTED]) iz= r=jb lg=en-US ( Language/content: English ) intl=us ( Country: United States ) ``` TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## What can cookies be used for? * Cookies can be used to identify a single machine from hundreds of other users on the same proxy IP address * The Yahoo B cookie is a “machine specific cookie” ``` B f6146fh596u4b b=4 d=GtdlgXBpYEQvWQEGtVnWhaxIPNw- s=9e i=8OSsR4OwqEO5oGGF2kJh ``` TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## What can cookies be used for? * Important note: All three of those examples are just subsets of the full Yahoo cookie string TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## How do we know what each cookie value is used for? * Nearly every web-site uses cookies that in most cases they designed for their own uses, so how do we know what they all mean? * Protocol Exploitation can examine the traffic to try to determine if there is any information contained in cookie strings that we might be interested, for example we’d like to know if any part of the cookie acts like a “machine specific cookie.” TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## How do we know what each cookie value is used for? * However, there are far more cookie options out in the wild than PE can possible examine. * So even if they aren’t aware of a machine specific cookie, it doesn’t mean that it doesn’t exist. * X-KEYSCORE gives you access to the full cookie string, so if you’re adventurous enough you can do your own protocol exploitation. TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Remember: Cookies are there for a reason! * Websites put cookies on people’s computers for a reason. * If the data is valuable for a website, it may be valuable to us as well. TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## How long do cookies live for? * Cookies, like any other file on a computer, can be deleted by the user. * Almost all browsers give you the option to view, manage and delete your cookies ``` Cookies Search: The following cookies are stored on your computer: Site Cookie Name bblux110.gp.proj.nsa.ic.gov brain.r6.r.nsa calendar.ops.s.nsa centerjohn.hqs.do.nsa csrc.do.nsa cybertrans-secure.eitc.nsa.ic.gov journal.ibd.nsa Name: Content: Host: Path: Send For: Expires: Remove Cookie Remove All Cookies Close ``` TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Cookies You can see what cookies have been stored on your machine by going into the "options" window of your browser and selecting ‘show cookies” ``` Options Main Tabs Content Applications Privacy Security Advanced History Keep my history for at least 90 days Remember what I enter in forms and the search bar Remember what I've downloaded Cookies Accept cookies from sites Accept third-party cookies Keep until: they expire Exceptions... Show Cookies... Private Data Always clear my private data when I close Firefox Ask me before clearing private data Settings... Clear Now OK Cancel Help ``` TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL ## Searches TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Searching the Internet When a user searches the Internet from one of the many web-based search engines (Google, Bing, etc.) what does the traffic look like? TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Searching the Internet: Client-to-Server * In most cases, the client-to-server traffic is a GET request where the search term is passed in the URL Arguments: ``` GET /search?hl=fr&q=iran&lr= HTTP/1.1 Host: www.google.com Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-powerpoint, application/vnd.ms-excel, application/msword, */* Cookie: PREF=ID=74f6d7addf51ccd4:U=ccbee9ee665a7dde:TB=2:TM=1255354439:LM=1255433264:S=_ M1i4RfO2ohI81maNID=27=cMFLkpovJCIWI0FC5E3Pu2C6-8_nsMS2zztfvOew9-QYDPWUza4AscyoglQRGNSkDZsi2jL65 flM-R4HgovMBEa66bfiTXn8TH3Ukm-X5hp45rLAb_Y3rNZ42HGIzyne Accept-Encoding: gzip, deflate User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Connection: Keep-Alive Cache-Control: no-cache ``` TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Searching the Internet: Client-to-Server * Notice how the URL Path is /search and one part of the URL argument is q=iran * Each website can configure their URL’s differently, so while with Google the search term is contained in the q= part of the URL, a different search form might have it as query= or search\_term= etc. http://www.youtube.com/results?search_query=iran&search_type=&aq=o TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Searching the Internet: Client-to-Server * X-KEYSCORE tries to account for all the variations of search terms contained in the URL Argument for what it extracts for the “Search Term” column. * However, there are always other varieties out there that we haven’t built it hooks for yet, so anytime you see something that you think should be extracted, please contact the team ([REDACTED]) TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## “Referer Searches” * What happens when a user clicks on a search result? * Let’s start by showing the query itself, in this example, we’re going to query the NSANet Google for “XKEYSCORE” TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## What does that GET request look like? ``` GET /search?q=xkeyscore&btnG=Google Host: google4.q.nsa User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 (USG-25) Firefox/3.0.10 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive ``` We know from this session that the client is requesting the data from the host ‘google4.q.nsa” and we see the search term in the URL Argument TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## “Referer Searches” * What happens when a user clicks on a search result? ``` GET /redmine Host: xkeyscore.r1.r.nsa User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 (USG-25) Firefox/3.0.10 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive Cookie: _session_id=ffd87ac8682e8fa8f421b4ffdf9693ae Referer: http://google4.q.nsa/search?q=xkeyscore&btnG=Google+Search ``` First, we can determine the full URL by adding the GET line to the host: http://xkeyscore.r1.r.nsa/redmine/ xkeyscore.r1.r.nsa/redmine XKEYSCORE - Overview - XKEYSCO... TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## “Referer Searches” * Secondly, we get some hints as to why the user was requesting that page from the Referer line: Referer: http://google4.q.nsa/search?q=xkeyscore&btnG=Google+Search * Note that it was the same URL that we were at immediately before we clicked the “result” link TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## “Referer Searches” * Let’s look at that process again: google4.q.nsa xkeyscore.r1.r.nsa First, a client-to-server request is sent that contains the query on "xkeyscore" TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Referer Searches * Let’s look at that process again: google4.q.nsa xkeyscore.r1.r.nsa Second, the server responds back with the search results TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## “Referer Searches” * Let’s look at that process again: google4.q.nsa xkeyscore.r1.r.nsa Third, by clicking on one of the results, a new GET request is issued to retrieve the XKEYSCORE home page. In this request, the location of the original search is listed as the "referer" TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL google4.q.nsa xkeyscore.r1.r.nsa What will happen if we only have collection on this link? TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## “Referer Searches” * When XKEYSCORE sees a search contained in the “referer” field, we still extract it out as meta-data into the “search terms” but we append it with (referer) to denote where it was originally found: ``` Search Terms (referer) the legal status of the caspian sea ``` | HTTP Type | Host | URL Path | URL Args | | --- | --- | --- | --- | | get | www.parstimes.com | /law/caspian_status.html | | ``` Referer: http://www.google.com/search?hl=fa&source=hp&q=the+legal+status+of+the+caspian+sea&lr= ``` TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## “Referer Searches” ``` GET /law/caspian_status.html HTTP/1.1 Accept: */* Host: www.parstimes.com Referer: http://www.google.com/search?hl=fa&source=hp&q=the+legal+status+of+the+caspian+sea&lr= Accept-Language: fa Accept-Encoding: gzip, deflate User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; InfoPath.2) Cache-Control: max-stale=0 Connection: close X-BlueCoat-Via: 0A6F53530F3F63EE ``` Can we guess what happened here? TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Another example ``` TOP SECRET//COMINT//20320108 ID: sess_orig_proc Type: HTTP-GET Printer Friendly Version DNI Display | Raw Data | DNI Format Services GET /Hezbollah-Terrorism-Judith-Palmer-Harik/dp/1860648932 HTTP/1.1 User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/1.0.154.48 Safari/525.19 Referer: http://www.google.com.pk/search?hl=en&q=written books on hizbollah&btnG=Google Search&meta= Accept: text/xml,application/xml,application/xhtml xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 Accept-Encoding: gzip, deflate, bzip2, sdch Cookie: ubid-main=185-5525816-8765531 apn-user-id=P1YXY7QF1PUYQ5 Accept-Language: en-US,en Accept-Charset: ISO-8859-1,*,utf-8 Host: www.amazon.com Connection: Keep-Alive ``` TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL ## Proxy Information TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Proxy Information * In a lot of cases we’re going to see HTTP Activity from behind a proxy or proxies. * What is a proxy? * A proxy is a server that is acting as an intermediary for HTTP requests from clients * Why do proxies exists? * Performance: Proxy can cache responses for static pages * Censorship: Proxy can filter traffic * Security: Proxy can look for malware * Access-Control: Proxy can control access to restricted content TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Proxy Information * Routinely, we’re going to see ISP level proxies. * That is, instead of having each individual user request web pages directly from the web servers, the ISP is going to collect all of those requests first, and then proxy them out through a handful of proxy IP addresses. * When the response is returned, the proxy passes it on to the appriopriate user TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Proxy Information * Why would the ISP want to proxy traffic? * In many cases the ISP won’t have to supply public IP addresses to all it’s users * It can simply give them a private IP address, and then use a handful of public IP addresses for its proxies which are the machines actually requesting the traffic from the web-servers TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Proxies on the Internet Web-Server Short-lived connections Single-user Web-Servers Short-lived connections Multiple-users multiplexed Web-Servers Long-lived connections Multiple-users multiplexed The Internet Direct-Connect Mixed-Gateway National-Level Proxy Cache Proxy-to-Proxy User-to-Proxy TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Identifying a Proxy * How do you know that the IP address that you think is your target is really a proxy? * First step, check NKB. * They have services that attempt* to automatically detect proxies * These services are in no way 100% accurate so this is only the first step in checking to see if the IP Address is a proxy TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Identifying a Proxy: NKB ``` Query: IP Address [REDACTED] Date: 2009-10-27@09:21:50 ``` | Description | Value | Confidence | Classification | | :---------- | :------------------------------------------- | :--------- | :----------------------------- | | Location Add Analyst Input | | | | | IP Range | [REDACTED] | [REDACTED] | (TS//SI//REL TO USA, FVEY) | | Lat/Long (precision) | (none found) | | (TS//SI//REL TO USA, FVEY) | | City | ZAHEDAN | 20 | (TS//SI//REL TO USA, FVEY) | | Country | (IR) IRAN | 91 | (TS//SI//REL TO USA, FVEY) | | Provider Add Analyst Input | | | | | IP Owner | RAYANE FARAZ IRANSHAHR COMPANY, INTERNET SERVICE PROVIDER | 82 | (U//FOUO) | | Autonomous System Route Prefix | [REDACTED]0.0/17 | 59 | (U//FOUO) | | Autonomous System Number | 12880 | 95 | (U//FOUO) | | Autonomous System Name | DCI-AS DCI Autonomous System | 95 | (U//FOUO) | | FQDN | (none found) | | (U//FOUO) | | Domain | [REDACTED] | 30 | (U//FOUO) | | Service | PROXY | | (U//FOUO) | | Service | TRANSPARENT PROXY | | (U//FOUO) | TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Identifying a Proxy * Other things to be on the look out for: * X-Forwarded-For IP Address * What is it? * An X-Forwarded-For IP address the proxy passing on to the server what it thinks is the IP address of the user * Think of it as the proxy telling the server “this is who I think this request came from” * It’s important to note that multiple proxies can, and often, are present, so one proxy might just be reporting the IP address of another proxy TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Identifying a Proxy * X-Forwarded-For IP Address as seen in traffic: ``` GET /HTTP/1.0 User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Host: www.ebay.com Pragma: no-cache Via: 1.0 s.jonoobnet.com (squid/3.0.STABLE10) X-Forwarded-For: [REDACTED] Cache-Control: max-age=259200 Connection: keep-alive ``` TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Some Examples of X-Forwarded-For headers: ``` X-Forwarded-For: [REDACTED] X-Forwarded-For: [REDACTED] X-Forwarded-For: [REDACTED] X-Forwarded-For: [REDACTED] X-Forwarded-For: [REDACTED] X-Forwarded-For: 192.168.1.10, 10.0.0.22,[REDACTED] X-Forwarded-For: [REDACTED] X-Forwarded-For: 127.0.0.1,[REDACTED] X-Forwarded-For: [REDACTED]google.com,[REDACTED] ``` Multiple-Layers of Proxies! In-general, the first IP is the one closet to the original requestor Keep in mind – these can be totally fake TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Identifying a Proxy * Similar to the X-Forwarded-For Tag is the “VIA tag” * The VIA tag is the proxy identify itself ``` GET /HTTP/1.0 User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Host: www.ebay.com Pragma: no-cache Via: [REDACTED] (squid/3.0.STABLE10) X-Forwarded-For: 217.219.95.135 Cache-Control: max-age=259200 Connection: keep-alive ``` TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Identifying a Proxy * The Via: tag may even contain some good information about the proxy * Be careful though because this information could be falsified: Via: 1.0 tehran-proxy-srv:3128 (squid/2.5.STABLE1) TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Identifying a Proxy * Remember though that the X-Forwarded-For and VIA lines can be falsified and don’t have to be present! * If they’re not present, how can you tell the IP address is a proxy? * Test it in MARINA! TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Testing IP Addresses in MARINA * The primary side effect of a proxy is too many users online at the same time * So if all else fails, try querying on the IP address (assuming its USSID18 compliant of course!) in MARINA to see how many users were active within an hour time frame * It’s not scientific but generally it will help TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL --- TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL ## Testing IP Addresses in MARINA For example look at these results: ``` Specify Date Range (YYYYMMDD [hhmmss]): 200910260100 to 200910260200