“Hey Uilson, please help us! We are getting an error to access the internet!”

OK! Time to stop dreaming about weekend plans and find out what is going on!

I quickly confirmed from my notebook that internet access was down and Internet Explorer was returning the error message below:

Error
FW-1 at fw6057: Access denied.

Requests were being redirected to our edge firewall.

Network Configuration

The network used two Forefront TMG in Network Load Balanced (NLB) configuration and all browsers received proxy details via WPAD.DAT script, delivered by GPO from our Active Directory servers.

Troubleshooting

When setting the web proxy details manually in Internet Explorer using the IP and port of the Forefront TMG proxy server,  Internet access was restored! This narrowed the problem down to an issue with the WPAD.DAT script.

Investigating WPAD

I went to Internet Explorer and tried to download the WPAD script by typing its address into my browser:
http://server.domain.com/wpad.dat

I found I could not access this link. Then, remembering some advice I received from one of our Field Analysts, I tried accessing the script via port 8080:
http://server.domain.com:8080/wpad.dat 

Success! I could download script.

I tried manually setting one of the workstations to download the script using port 8080, and it was able to access the internet again!

OK my friends! I’ve found what was wrong! The Forefront TMG Server was refusing requests to the WPAD.dat script on port 80.

Solution

The reason why Internet access suddenly dropped was that someone made a change to Forefront TMG’s Internal network properties and disabled the access via port 80 by unchecking the “Publish automatic discovery information for this network” option, as shown in the image below:

When checking this option again, all users got their Internet access back!

About Web Proxy Auto Discovery (WPAD)

The ‘Publish automatic discovery information for this network’ option in Forefront TMG allows access to the Web Proxy Automatic Discovery (WPAD) protocol. All you need to do is configure a host record in DNS called WPAD that resolves to the IP address of your Forefront TMG’s internal network interface.

The WPAD method can pose potential security issues, so Microsoft added WPAD to the default Global Query Block List in Windows Server 2008. This means that the DNS service will not respond to WPAD queries by default. It is possible to turn this method on by following some steps that my friend Richard Hicks describes in his post: DNS Security Enhancements and Proxy Auto Discovery.

The best way to deploy the WPAD script is keeping the default link provided by Forefront TMG. In case you want to set up a customized link, always create it using port 8080 as default. For example: http://proxy.uilson.com:8080/wpad.dat

Using an address like the one above won’t impact users if someone unchecks the Automatic Discovery publishing option.

You also need to be sure the script address on Forefront TMG matches what you have specified in Active Directory GPO.

Further WPAD Troubleshooting

Luckily, my issue was easily solved by re-publishing the Auto Discovery service on Forefront TMG. If you are having other issues with WPAD on Forefront TMG and this article does not help, here are some other WPAD troubleshooting resources you may find useful:

Troubleshooting Automatic Detection (Forefront Operations Documentation)
http://technet.microsoft.com/en-us/library/cc302643.aspx

Forefront TMG Web Proxy Auto Detect Fails (Richard Hicks)
http://tmgblog.richardhicks.com/2011/05/23/forefront-tmg-2010-web-proxy-auto-detect-fails/

Troubleshooting Windows Proxy Auto Discover – WPAD (Infratalk)
http://infratalk.wordpress.com/2011/09/10/troubleshooting-windows-proxy-autodiscovery-wpad/

Automatic Discovery Woes (Forefront TMG Product Team Blog)
http://blogs.technet.com/b/isablog/archive/2006/01/04/416887.aspx

WPAD is Working Or Not (Suraj Singh MSFT)
http://blogs.technet.com/b/sooraj-sec/archive/2011/07/07/wpad-is-working-or-not.aspx

The post Troubleshooting Forefront TMG Web Proxy Auto Discovery (WPAD) Issues appeared first on Forefront TMG Reporting.

This video takes you through how to use Fastvue TMG Reporter to report on YouTube.com, and how to create a report that provides a simple list of all the videos that people have watched. It also dives a little deeper into how YouTube streaming works to help you better understand the detailed information in TMG Reporter’s activity reports.

Reporting on YouTube with Fastvue TMG Reporter

If you have any questions about reporting on YouTube.com, or if you have other specific reporting requirements, don’t hesitate to get in touch.

If you haven’t yet tried Fastvue TMG Reporter, download our free 30 day trial today.

The post How to Report on YouTube Activity with Fastvue TMG Reporter appeared first on Forefront TMG Reporting.

Amazingly, very few people know anything about it and even fewer people including some IT professionals know how it works. The aim of this articles is to briefly explain the concepts of how this technology works. With your new understanding you should be able to detect and avoid SSL related problems and warnings.

Terminology

There are a few common terms used to refer to different aspects of the technology, but in general they are all interchangeable and refer to an encrypted data session between a client browser and a secure web server:

• HTTPS – Hyper Text Transfer Protocol Secure
• SSL – Secure Sockets Layer
• TLS – Transport Layer Security

Other terms related to this technology include:

• Keys – Text cypher
• Public Key -Known text cypher
• Private Key – unknown text cyper
• Symmetrical or Session Key – Pair of matching keys on either side
• Certificates – Text containers containing cypher and other identifying information

The Basic Steps

Below are the main steps involved in creating and maintaining an encrypted data session. The initial SSL hand shake is covered in steps 1 through 5, and the data transmission that continually reoccurs is covered in steps 6 and 7.

1.  –> Browser requests secure site using HTTPS header
2. <– Secure web server sends certificate containing its public key
3. –> Browser validates the certificate  with request to validation servers
4. –> Browser uses the public key and creates a symmetric key that will only be valid for that session and sends it to the web server
5. Web server decrypts the symmetric key with it’s private key
6. <– Web server returns data encrypted with the symmetric session key
7. Browser decrypts data using the symmetric session key

Depending on the client and the web server configuration, there may be additional steps (such as authentication / verification) involved.

Deeper dive into the steps

Step 1.  Request secure site

URLs for normal (un-encrypted) web browsing start with http://. When the URL starts with https:// it indicates to the web server that the browser is requesting secure content.

The actual header data for these two are methods contain three parts.

<Protocol> :// <Fully Qualified Domain Name (FQDN)> : <Port number>

An example of this would be

• http://www.google.com:80
• https://www.google.com:443

The reason most users are not aware of this is because most modern browsers hide the http:// and :80 portions of the URL. For secure URLs, modern browsers usually show the https:// portion but they still  hide the port number portion.

Step 2.  Secure web server sends certificate

Once the web server receives the request that indicates a secure connection is required, it sends its certificate to the client browser.  The certificate contains the following bits of information that will be used in the following step.

• The site name to which the certificate is issued to
• The Certificate Authority (CA) that issued the certificate
• The dates for which the certificate is valid.
• Thumbprint
• CRL points

You can check these yourself by clicking on the certificate and examining the various fields (Open Windows Certificate Manager  with Start | Run | certmgr.msc).

Step 3. Browser validates the certificate

Before the browser will accept the certificate and use it to establish the session keys, it validates that the certificate is valid and trusted. To do this:

• It compares the name to which the certificate was issued and checks that it matches the Request and Response header.
• Checks the current date against the certificate validity period dates.
• It uses the Certificate Revocation List (CRL) points to see if the certificate’s thumbprint has been revoked or if it is still valid.
• It also checks the local system to see if it trusts the Certificate Authority that issued the certificate.

If any of these validations fail the browser will alert you to it and provide the option to accept the invalid certificate and proceed to the following step. If the certificate is trusted and no alerts appear, a secure padlock indicator appears in the address bar.

There is another category of certificates that are “extra trusted”.  These certificates require additional validation steps between the Certificate Authority and the client. These certificates are typically valid for shorter duration and are subject to annual re-evaluation.  You can identify the Extended Validation (EV) certificates by how the browser “Lights Up Green” when it encounters one of these.

Step 4. Browser creates the symmetric key

The browser uses the web server certificate’s public key and and creates a unique session key. The session key will only be valid for the duration of the session and is therefore referred to as a session key.

Only the client browser can decrypt the data since it contains the private key that was used  in conjunction with the public key. The browser then send the session key back to the web server. At this point it is called the pre-master secret.

Step 5. Web server decrypts the session key

The web server is in possession of its own private key that can be used to decrypt any message encrypted using its public key. It uses its private key to decrypt the pre-master secret and the session key received from the client browser.

This is then used for all further encryption for the duration of the session. The the same key will now be used at the client and server side as it is a symmetric key pair. The process up to now is known as the SSL handshake.

Step 6. Data is encrypted before sending to the client

The web server now uses the symmetric key to encrypt the data portion of all packets being sent back to the client. It is important to note that the HTTP header information is not encrypted, allowing for correct routing and reporting of the encrypted traffic.

Step 7. The client browser decrypts the response from the web server

The client browser receives the encrypted data from the web server. It uses its session key to decrypt the data and render the content. An important concept to grasp here is that the tunnel is between the client browser and the web server, not the entire client computer. Even if data packets are intercepted on the client with a packet sniffer such as Wire Shark it cannot be decrypted.

Something to think about

With the Web 2.0 and cloud services, more information than ever before is being transmitted over the web. Increasingly, a lot of it is confidential and should not simply be transmitted in clear text.

As consumers we should be vigilant and take note of the SSL status (or lack thereof) before we post sensitive information to a web site.

As IT professionals we have an obligation to ensure that applications we publish over the Internet are secure and encrypted.

Implementing SSL and using proper certificates that do not generate a certificate warning is not difficult or expensive. Secure, validated and encrypted web communication becomes more important everyday.

As a starting point, make sure your TMG Reporter installation is secured by following this guide: How To Secure And Publish the Fastvue TMG Reporter Web Site

The post What everyone should know about HTTPS, SSL, TLS and Certificates appeared first on Forefront TMG Reporting.

Richard Hicks (MVP)

Richard Hicks is a guy worth listening to when it comes to Forefront TMG. He has been installing, configuring, and managing TMG and its predecessors ISA Server and Proxy Server for more than 15 years. He has also built a great resource of information that Forefront TMG administrators use every day on his own TMG Blog site and on ISAServer.org.

Throughout his experience with Forefront TMG, Richard has picked up some great tips and tricks that you can apply in your daily workflow to help make your TMG Firewall easier to manage and even more performant.

You can read all about these tips and tricks is his latest article Forefront TMG 2010 Policy and Configuration Management Tips and Tricks.

Forefront TMG Tips and Tricks

More tips from the Archives

Richard has also been kind enough to write some guest posts on our very own TMG Reporter blog. Here are some of his past articles from our archives:

• Block Instant Messaging Traffic Using Forefront TMG’s HTTP Filter
• Forefront TMG Enterprise Logging with Remote SQL Server
• Tips for Healthy Logging and Reporting in Forefront TMG 2010
• Enterprise Reporting Challenges with Forefront TMG 2010
• Logging Improvements in Forefront TMG 2010

The post Forefront TMG Tips and Tricks appeared first on Forefront TMG Reporting.

TMG Reporter 2.1 features detailed Activity Reports, Scheduled Custom Reports, Comprehensive Filtering and more!

See What’s New

Or read on below…

Activity Reports

Activity Reports provide a chronological list of all activity showing full URLs and timestamps. Step through daily web browsing activity and expand sites to view full URLs, categories and actions.

Click the Activity Report button on the Reports tab to get started.

Scheduled Reports

Schedule any report and email it to the right person. Send the Marketing manager an unproductive browsing report for their department every week, or receive an automated daily email of the day’s top threats.

Click the clock button on the Reports tab to get started.

Comprehensive Filtering

Report on exactly what you need with the new filtering interface.  Run reports on unacceptable browsing in the marketing department (yeah, we like to pick on the marketing folks!), but don’t include Web Ads or blocked traffic.

Click the Filter button on the Reports tab to get started.

Drilldown Reporting

Drilldown on activity by launching new reports while keeping the existing filters. Create a report filtered by Unproductive browsing, and then drilldown into ‘Allowed’ traffic while keeping your unproductive filter.

Hover over the magic green arrow that appears in reports to get started.

Performance!

TMG Reporter 2.1 generates reports almost twice as fast as version 2.0 and uses less memory. 2.1 also comes with a long list of minor bug fixes and tweaks making the overall experience even smoother.

Import Forefront TMG’s Remote SQL Logs

If you have configured Forefront TMG to log to a remote SQL database (there are some great reasons to do so), you can now import directly from the database. No need for the Fastvue Arbiter!

Go to Settings | Sources | Add Source | SQL Server to get started.

Server 2012 & Windows 8

TMG Reporter installs and configures all the prerequisites (IIS and .NET 3.5) on Windows Server 2012 and Windows 8.

Get Started

FAQs

How much does it cost?

Upgrading to TMG Reporter 2.1 is free for anyone with an active subscription to TMG Reporter. Otherwise, all pricing information can be found on our Pricing Page.

Can I upgrade without losing data?

Yes, absolutely. Just run the new TMG Reporter installer over the top of your existing installation. Your existing settings will be picked up by the installer, so just hit next throughout the wizard without changing any settings. Once the installer has completed, head to the site and clear your browser cache (ctrl + F5, or cmd + R on Mac). You can then run reports on all your previously imported data.

Note: The Dashboard and Alerts screens will be reset, and will start rebuilding as new data is imported.

Do I need to upgrade the Fastvue Arbiter?

If you are running a relatively recent build of TMG Reporter (2.0.1.14 or above), then you do not need to upgrade the Arbiter on your TMG Server. No changes have been made to the Arbiter since July 2012.

How many chickens do I need to stack to reach the moon?

You’ll need about 1,164,848,484 chickens.

Can someone help me?

We won’t stack chickens for you, but we’re more than happy to help with any questions or issues related to TMG Reporter. Get in touch at support.fastvue.co. We’re here to help.

The post TMG Reporter 2.1 Out Now! appeared first on Forefront TMG Reporting.

The Tunnel

We all know that HTTPS traffic is encrypted. A virtual secure tunnel is created between the client and the server over which the “payload” of the conversation is transmitted, but it is important to know that the ‘control data’ is not encrypted.

To illustrate this, you can look at a WireShark capture of an unencrypted HTTP browser session. Here you can see the server responding with a 302 redirect. The actual content of the response is visible is clear text. Note that this capture is at the network level.

If this response were sent from the server to the client in an encrypted SSL session, the packet would essentially be the same except that the 302 response data would be hashed. Only the browser on the client’s machine is capable of decrypting the hash and reading the data.

It is important to note that the secure SSL tunnel is between the client browser application and the web server application.

Forefront TMG is capable of scanning traffic flowing through it for malware, but when the conversation happens over an encrypted SSL tunnel, Forefront TMG cannot perform any layer 7 deep packet inspection. A download that contains a piece of malware will pass straight through Forefront TMG if the communication is encrypted with HTTPS. Fortunately, Forefront TMG has a way around this via its HTTPS Inspection feature.

How HTTPS Inspection Works

When HTTPS inspection is enabled in Forefront TMG, the single secure tunnel between the client browser and the web server is split in two. Forefront TMG becomes the tunnel termination point, establishing a secure tunnel with the web server. This enables it to decrypt and inspect the data coming from the web server. To keep everything secure, Forefront TMG establishes a second tunnel from itself to the client machine.

HTTPS Concerns and Misconceptions

Privacy

Because the Forefront TMG server effectively breaks the trusted secure connection between the client and the web server, it introduces privacy concerns. These can be a prohibiting factor that limits the use of HTTPS inspection due to law or policy.

Blocking Secure Sites

One common misconception is that HTTPS inspection needs to be enabled in order to block secure sites. URL filtering and access rules work on the unencrypted ‘control data’, and are therefore unaffected by HTTPS inspection. See Blocking Sites with Forefront TMG.

Reporting

Web sites listed in Forefront TMG’s Reports, as well as the reports in Fastvue TMG Reporter are also unaffected by HTTPS inspection. These reports are built from log data that includes the full URL of the site, not the site’s content. Details such as the client IP, authenticated username, access rules and site name remain the same regardless of the conversation being encrypted or not.

Reporting on HTTPS Traffic

It is useful to report on your organization’s HTTPS traffic to get an idea of the HTTPS risk profile, especially if you are trying to make the decision about whether or not to enable HTTPS Inspection.

The easiest way to report purely on HTTPS traffic is using Fastvue TMG Reporter 2.1 (currently in Beta, download here). The new filtering feature allows you to filter your reports by Protocol Equal to SSL-tunnel. It is important to specify SSL-tunnel (as opposed to HTTPS) as this is the way the HTTPS protocol is recorded in TMG’s Web Proxy Logs.

Another option is to filter by Destination Port Equal to 443

Here is a screenshot of the new Activity Report in TMG Reporter 2.1 when filtered by Destination Port Equal to 443.

TMG Reporter’s Activity and Overview reports will show you the sites, categories and productivity ratings of SSL sites, as well as the percentage being blocked/allowed, and the current policy rules in play. This helps you identify and quantify the potential risk associated with SSL tunnels, providing justification to enable the added security of HTTPS inspection.

Separate Access Rules for HTTPS traffic

If you need fine-grain control over HTTPS traffic, a good idea is to create separate Forefront TMG web access rules for HTTPS. To do this:

1. Open the Forefront TMG Management Console
2. Select Web Access Policy
3. Locate your rule that allows internet access for users
4. Right click copy | Right click paste
5. Open the new rule and rename it to indicate HTTP traffic only
6. Select the Protocols tab and remove all protocols except HTTP
7. Click OK to save the changes.
8. Select the original rule
9. Rename the rule to indicate HTTPS traffic only
10. Select the Protocols tab and remove all protocols except HTTPS
11. Click OK to save the changes
12. Verify that the access rules are below the URL filtering deny rules.
13. Click the Apply Button to apply the changes.

Once some traffic has been filtered through the new rules, you will see the results on TMG Reporter’s live dashboard under the Firewall section. You can drill down into these rules to view more details about the traffic, and use this information to customize the rules over time.

You can also see this information in Company Overview or User Overview reports in the Firewall Rules section.

Summary

HTTPS inspection is a very useful feature in Forefront TMG 2010, in order to guard against malware transmitted over an encrypted SSL Tunnel.

HTTPS inspection does not need to be enabled to block sites that use HTTPS (such as facebook.com), nor is it required for reporting on HTTPS (SSL) sites.

To get a more accurate picture of your organization’s HTTPS traffic, use Fastvue TMG Reporter 2.1 to run a filtered report by Protocol Equal to SSL-tunnel, or Destination Port Equal to 443.

These reports can help you determine your HTTPS risk profile and justify whether or not you need to enable HTTPS inspection in Forefront TMG.

The post HTTPS Inspection in Forefront TMG – Concerns and Misconceptions appeared first on Forefront TMG Reporting.

If you contact Microsoft to purchase or renew Forefront TMG, you will be told that the product is no longer available.

The OEM License Loophole

Fortunately, Microsoft has not pulled OEM licenses from their Forefront TMG product listing. This means you can still purchase Forefront TMG from OEM appliance vendors such as Winfrasoft, Celestix, Iron Networks (previously nAppliance), IVO Networks, or SecureGuard.

This is still not ideal as most Forefront TMG customers are comfortable with their software only deployment model and do not want yet-another-appliance in their server room or datacenter.

Virtual TMG Appliances

Forefront TMG Virtual Appliances are a great option for ‘software only’ deployments. You simply download an ISO image and deploy it on your VMware or HyperV infrastructure.

At this point in time, Winfrasoft is the only vendor that offers Forefront TMG virtual appliances in addition to their popular hardware appliances.

If you are looking at purchasing Forefront TMG, or need to renew your license before you have found a suitable TMG replacement, definitely check out the Winfrasoft Virtual TMG Appliances to mimimize the deployment headache.

Download a Forefront TMG Virtual Appliance

Fastvue is also a Gold Winfrasoft partner, so if you’re interested in purchasing a physical or virtual Forefront TMG appliance, contact us!

Don’t Forget Reporting!

Once you are up and running with your Forefront TMG appliance (virtual or hardware), make life that little bit easier and grab yourself a copy of TMG Reporter. Live network dashboards, overview and detailed activity reporting, seamless AD integration, scheduled reporting, customizable alerts and more!  Check out the new features in TMG Reporter 2.1 Beta.

The post How To Purchase Forefront TMG After January 2013 appeared first on Forefront TMG Reporting.

To reduce the amount of unauthenticated traffic recorded by Forefront TMG:

1. Use TMG Reporter to run a report on your anonymous user (hover over the anonymous user and click the green arrow to ‘Run report on anonymous’)
2. Go to the Firewall Rules section of the report and you will see all the rules that are allowing the unauthenticated traffic.
3. Edit these rules in Forefront TMG and set them from ‘All Users’ to ‘All Authenticated Users’.

There are certain system defined rules that allow unauthenticated traffic, and unfortunately these rules cannot be edited via the TMG Management Console.

One such rule is the SafeSearch rule that gets created when using Forefront TMG’s SafeSearch enforcement feature. This rule effectively allows all browsing to search engines to pass through unauthenticated. If you want to identify the user that was responsible for a specific search, bad luck!

Fortunately there is a way to set this rule to ‘All Authenticated Users’ but it needs to be done using a script. Richard Hicks explains how to do this in his post, Enable Authentication for SafeSearch Enforcement Rule in Forefront TMG 2010.

Thanks for the tip Richard!

UPDATE!

Richard Hicks has made a follow up post outlining some of the challenges associated with the ideal goal of ‘authenticating everything’. I recommend checking it out here Identifying and Reducing Anonymous Traffic Allowed by Forefront TMG 2010

On this point, we often see authenticating BYOD (bring your own devices) such as mobile phones and tablets as a major headache for Forefront TMG Administrators. The recommended course of action in this situation is to create a separate network for these devices, and create an access rule for this network in Forefront TMG. This rule can allow ‘All Users’ (unauthenticated), and then it is easy to include or exclude this traffic using a Rule Equal to ‘my unauthenticated traffic’ Filter in TMG Reporter. Make sure you’re using the latest 2.1 Beta to access this comprehensive filtering feature.

The post Reducing Anonymous (Unauthenticated) Traffic in Forefront TMG appeared first on Forefront TMG Reporting.

New Features Demo

Installation and upgrade instructions will be presented once you download (it’s really simple). We look forward to hearing your feedback!

The post TMG Reporter 2.1 Now Available appeared first on Forefront TMG Reporting.

The downside of grep/findstr/grepToolXYZ is they generate a new file that does not necessarily adhere to the format or structure of the original log file. The output is often stripped of all W3C log headers, and many tools even replace delimiting characters such as tabs with spaces. The end result is that you have a file containing your search result, but the file is incompatible with your log analysis application (such as WebSpy Vantage).

Fastvue Log Grep Utility

To solve this problem, we developed a simple Grep tool for Windows. It takes a folder of W3C text logs (or any text files), and produces new files containing only the log lines matching your search criteria. The new files are unaltered in format and structure and retain the original log headers. The Fastvue Log Grep Utility is perfect for cutting down a folder of Forefront TMG or ISA Server W3C text logs to just the information in your search string. You can then import these reduced files into WebSpy Vantage, or any other log analysis app.

The Fastvue Log Grep Utility supports plain text search, regular expressions, inverted match (find everything except), and can merge results from multiple files into a single output file, or create multiple output files.

Fastvue Grep Tool for W3C Logs. Retains log headers and structure of the log file.

Although it was designed for keeping the structure of W3C text logs, it can also be used to search any type of text file. Just ignore the option to ‘Always include header lines’.

Download Fastvue Log Grep Utility

We’re making the Fastvue Log Grep Utility tool available absolutely free, so please go ahead and downloaded it here:

Hopefully it makes your life a little bit easier. Either way, we’d love to hear your comments below.

The post Free Windows Grep for W3C Log Files appeared first on Forefront TMG Reporting.