Introduction

A survey of Alexa's top 10,000 websites on the Internet was conducted to measure the usage of security-related HTTP headers, mobile awareness and potential information leakage. We did this by grabbing the banners of those websites with 18 different user agents (see Appendix) using a modified UATester script, and analyzing the resulting HTTP headers.

The latest raw data can be downloaded from the website in CSV format at: http://www.shodanhq.com/research/infodisc/download_latest.

This report was generated from the data collected and parsed on March 14th, 2011.

Security Headers

Overview

We looked at how many of the banners used available security HTTP headers, such as: X-XSS-Protection, X-Frame-Options, Access-Control-Allow-Origin and Strict-Transport-Security.

X-XSS-Protection

The XSS Filter was introduced by Internet Explorer 8 to reduce the effectiveness of non-persistent XSS attacks. The X-XSS-Protection header can be used by web developers to control or disable the XSS Filter.

Of the banners containing the X-XSS-Protection header, 8.7% of them explicitly disabled the header (value of "0") while the remaining 91.3% had the value of "1; mode=block". The disabling of this feature can be traced to a flaw in IE8 that was discovered in late 2009.

X-Frame-Options

The X-Frame-Options header was introduced by Microsoft in Internet Explorer 8 as a counter-measure to Clickjacking. The possible values for the header are: SAMEORIGIN and DENY. The SAMEORIGIN policy forces the page to only be displayed in a frame on the same origin as the page itself. And the DENY policy prevents the page from being displayed in a frame at all. The X-Frame-Options header has since also been adopted by the other major browsers. In the survey though, only 0.54% of the banners contained this feature.

Access-Control-Allow-Origin

The Access-Control-Allow-Origin header is part of the Cross-Origin Resource Sharing recommendation by the Web Applications Working Group. The header is used to allow cross-site AJAX requests and is supported by most modern browsers (learn more).

61% of the banners had a value of "*", 19.5% had "https://ymodules.yammer.com" and the remaining 19.5% had "https://apps.boostmobile.com".

Strict-Transport-Security

HTTP Strict Transport Security (HSTS) is a proposed web security policy mechanism where a web server declares that a web browser should interact with the server only using a secure connection (such as HTTPS).

The HSTS policy is meant to only apply to HTTPS requests, as such the websites in the survey that contained the HSTS header are actually HTTPS banners. They automatically redirected our UATester script from a non-secure HTTP to a secure HTTPS connection.

Note: Only 2 websites used this header, which makes it the least used security-related HTTP header amongst the top 10,000 websites.

Analyzing the Server Header

Overview

In an attempt to hide server information such as the web server software (Apache, Microsoft-IIS etc.), many websites take the step to obfuscate their Server header. Based on our data, those attempts aren't always thorough enough and sometimes server information can still be obtained by sending different user agent requests. The same method can also be used to learn more about the network topology of a web backend (i.e. reverse proxies) and possibly identify various servers that are in use. The graph below breaks down the number of websites that contained a given number of unique Server headers. As indicated, the majority return only 1 unique header but about 10% of the websites surveyed had more than 1 distinct value.

Example: live.com

The website live.com contains 3 distinct Server headers:

Microsoft-IIS/6.0
Microsoft-IIS/7.0
Microsoft-IIS/7.5

All desktop browsers and most mobile browsers received their data from the Microsoft-IIS/6.0 server, while the popular command-line tools cURL and Wget were sent to the Microsoft-IIS/7.5 server. The browser on the Symbian OS platform was the only user-agent that received data from a Microsoft-IIS/7.0 daemon.

Tip: Discover new hosts in your penetration test by using various user agents in your penetration test.

Example: qiyi.com

Qiyi.com provides 3 different Server headers depending on the user agent:

nginx
nginx/
nginx/0.8.34

For the majority of browsers the website returned a non-descript Server header, but for Firefox 3.6, Opera and Nikto the banner contained specific version information.

Tip: Identify missed obfuscation and potential information leaks by sending different user agents.

Blocked User Agents

Overview

Not all web requests returned banners, and below is a break-down of the most common user agents that didn't return anything from the website. With so much data gathering there is a problem of noise, as such only websites that returned fewer than 3 empty banners were considered in the analysis.

Example: Wget

GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc.

Notable websites that didn't return a banner:

Example: Nikto

Nikto is an Open Source (GPL) web server scanner which performs comprehensive tests against web servers for multiple items, including over 6400 potentially dangerous files/CGIs, checks for outdated versions of over 1000 servers, and version specific problems on over 270 servers.

Notable websites that didn't return a banner:

Example: cURL

cURL is a command line tool for transferring data with URL syntax, supporting DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, kerberos...), file transfer resume, proxy tunneling and a busload of other useful tricks.

Notable websites that didn't return a banner:

joomla.org

Appendix

List of User Agents

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0)
Mozilla/5.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)
Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100922 Firefox/4.0.1
Mozilla/5.0 (Windows; U; Windows NT 5.2; rv:1.9.2) Gecko/20100101 Firefox/3.6
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.514.0 Safari/534.7
Opera/9.99 (Windows NT 5.1; U; pl) Presto/9.9.9
Googlebot/2.1 (+http://www.google.com/bot.html)
Mozilla/2.0 (compatible; Ask Jeeves)
msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)
Mozilla/4.75 (Nikto/2.01)
curl/7.7.2 (powerpc-apple-darwin6.0) libcurl 7.7.2 (OpenSSL 0.9.6b)
Wget 1.9cvs-stable
Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543a Safari/419.3
Mozilla/5.0 (iPad; U; CPU iPhone OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B314 Safari/531.21.10
Mozilla/5.0 (Linux; U; Android 2.1-update1; en-at; HTC Hero Build/ERE27) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17
jBrowser-WAP
Nokia7650/1.0 Symbian-QP/6.1 Nokia/2.1