The Role of HTTPS Inspection in Google Search and YouTube Reports

Google Search and YouTube contain a wealth of information, and it makes sense to allow your users to access this information in order to conduct research and perform their jobs effectively. But you also want to ensure this privilege is not abused by users spending all day watching cat videos or hours researching their next holiday while at work.

We are commonly asked:

• How do I get a report showing a list of videos people have watched on YouTube?
• How can I get a report that shows all Google searches a user has made?
• Can I get alerts sent to me when specific keywords are entered into Google search?

Unfortunately, since Google made the decision to implement HTTPS across all their web properties (including YouTube), the information required to answer these questions is often lacking from your web gateway or firewall's log files. You can report that someone has been to youtube.com, but not what videos they were watching. Likewise, you can report that someone has visited google.com, but not what they searched for.

The problem is, when your firewall logs access to a HTTPS website, only the domain is logged. For example:

https://www.google.com https://www.youtube.com

But to ensure these websites are not being abused, we really need the firewall to log the full URL as the interesting parts, such as the video that was watched or the term that was searched, are contained in the query string of the URL. For example:

https://www.google.com/search?q=My+Search+Term https://www.youtube.com/watch?v=GDDDaya2mM4

With more of the web adopting Google's stance on HTTPS (for good reasons), this directly impacts the usefulness of your web activity reports, beyond just Google and YouTube.

Fortunately, many Next Gen Firewalls, UTMs and Secure Web Gateways have a HTTPS Inspection feature that gets around this problem and forces the full URLs to be logged again.

In this article I will show you how to selectively apply HTTPS Inspection to Google and YouTube using Sophos UTM, and how to report on YouTube videos and Google searches using Fastvue Sophos Reporter.

Using Website Tags To Apply HTTPS Inspection to Specific Sites

We have a comprehensive article on how to configure HTTPS Inspection on Sophos UTM here, and it also describes its affect on logging and reporting.

However, deploying HTTPS Inspection is not without its issues. If you don't want to apply HTTPS inspection to all web traffic, Sophos UTM enables you to tag certain websites (google.com and youtube.com for example), and then add the tag to your HTTPS scanning settings.

In the screenshot below, I am creating a Website Tag called 'ScanThis' and adding the google.com and youtube.com domains.

When enabling HTTPS Inspection, simply select the website tags containing the websites you want to scan.

You could achieve the same result by adding the Streaming Media and Search Engines category to the list of categories to scan on the right hand side.

One advantage of scanning the youtube.com domain instead of the entire Streaming Media category is that Sophos UTM will not scan the full video streams, as these are served from another domain (currently this is googlevideo.com). It will only decrypt content served from the youtube.com domain - the one that includes the video URL you want to see in your reports.

Update: Thanks to Nick in the comments, implementing HTTPS Inspection can cause issues for certain apps and devices, such as Chromebooks. To help ease these headaches, Google have published a handy list of domains to whitelist from HTTPS Inspection. See our article on implementing exceptions in Sophos UTM without relaxing security, for information on how to do this.

Reporting on YouTube videos with Fastvue Reporter

Once HTTPS Inspection is implemented, you can retrieve the list of YouTube video URLs.

Update: Since originally writing this article, we've since discovered a better technique to report on YouTube videos utilizing Referrer URLs. This provides a report that better shows the video URLs along with more accurate Start and End Times. The new information follows, with the old information in strikethrough below.

To retrieve a report showing all the YouTube videos accessed, run an Activity Report with the filter: Referrer with Query 'Contains' youtube.com/watch, youtube.com/watch?

Why have a filter for both "youtube.com/watch" OR "youtube.com/watch?" The Activity Reports dynamically change the columns in the reports depending on what you're filtering on. If you just run a report filtered by Referrer with Query 'Contains' youtube.com/watch, you'll just see a list of users accessing that URL. By adding two 'things' to filter on (youtube.com/watch OR youtube.com/watch?) the Activity Reports will add another column showing the Referrer URL with Query – which in this case is the URL of the video being watched.

Here's what the final report should look like.

Note that the calculated 'Browsing Time' is not a column in this report (Start and End Times are though). If you'd like the Browsing Time as well, add another filter to the above for: Origin Domain 'Equal to' youtube.com

So the full filter would be: Referrer with Query 'Contains' youtube.com/watch, youtube.com/watch? AND Origin Domain 'Equal to' youtube.com

I know this is not very intuitive, and ideally you shouldn't have to worry about hacking around with the filter to display the columns you want to see in the report. I recommend clicking the Save Filter button once you've defined the above and saving it as 'YouTube Videos'. You can then click the Load filter button anytime you want to run these reports in the future.

Once HTTPS Inspection is implemented, you can retrieve the list of YouTube video URLs by running an Activity Report with the following filter:

URL 'Contains' youtube.com/watch

Here's the resulting report:

If you want to find the users that have accessed a specific video, for example: https://www.youtube.com/watch?v=GDDDaya2mM4, run an Activity Report with the following filter:

Site Query 'Contains' GDDDaya2mM4 And Site Domain 'Equal to' youtube.com

Reporting on Google Searches with Fastvue Reporter

Likewise, now that HTTPS Inspection is enabled, you will start to see Google searches appear in the 'Search Terms' section of User Overview Reports.

You can also run Activity Reports to search for specific search terms or keywords. For example:

Search Terms 'Contains' Guns, Weapons

The columns displayed in an Activity Report change depending on the fields and operators you use in your filters. You can force the 'Search Terms' column into the report by using a Search Terms 'Contains' filter. Therefore, to extract all Search Terms, you can use this (ugly) filter:

Search Terms 'Contains' a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z

Here's the resulting report:

Conclusion

Setting up HTTPS Inspection on Google and YouTube will help satisfy these two common reporting requirements. But with more of the web moving to HTTPS, you will eventually need to expand the list of sites, or perhaps just scan everything.

When choosing a new firewall / gateway, performance with HTTPS Inspection enabled should be one of the features at the top of your list. It is not only critical for identifying and removing malware within secure connections, but also for giving you the detail you need in your reports to manage and control your network effectively.