Web traffic logs and data sources for Office 365 Cloud App Security

Evaluation > Planning > Deployment > Utilization
Start evaluating
Start planning
Start deploying
You are here!
Next steps

You can use a wide range of web traffic log files and data sources with Office 365 Cloud App Security. However, your web traffic log files must include specific information and be formatted a certain way so that they will work with Office 365 Cloud App Security app discovery reports and the Cloud Discovery dashboard. Use this article as a reference guide for the web traffic logs and data sources you'll use with Office 365 Cloud App Security.

Note

You must be a global administrator, security administrator, or security reader to access the Security & Compliance Center and Office 365 Cloud App Security portal. See Permissions in the Office 365 Security & Compliance Center.

Web traffic log requirements

Office 365 Cloud App Security uses data in your web traffic logs to help you understand which apps people in your organization are using. The more details that are included in the log files, the better visibility you'll have into user activity.

The following table lists the requirements and attributes that are needed for your web traffic logs to work correctly with Office 365 Cloud App Security:

Attributes Additional requirements
Date of the transaction
Source IP
Source user (recommended)
Destination IP address
Destination URL (recommended: URLs provide higher accuracy for cloud app detection than IP addresses)
Total amount of data (recommended)
Amount of uploaded or downloaded data (recommended: provides insights about cloud app usage patterns)
Action taken (allowed or blocked)
The data source for the log files must be supported.
The format the log files use must match the standard format. When the file is uploaded, app discovery will verify this.
The events in the log must have taken place no more than 90 days ago.
The log file must include outbound traffic information that can be analyzed for network activity.

If attributes aren't included in the logs that are loaded, Office 365 Cloud App Security can't show or analyze the information for you. For example, Cisco ASA Firewall's standard log format does not include the amount of uploaded bytes per transaction, the username, or a target URL (only a target IP). Because that information isn't in the Cisco log files, Office 365 Cloud App Security won't include it when analyzing your organization's network traffic.

Note

For some kinds of firewalls, you must set an information level for web traffic logs to include the required attributes. For example, Cisco ASA firewalls must have the information level set to 6. Make sure to confirm that your firewalls are set to deliver the correct information in your web traffic logs.

Data attributes for different vendors

The following table summarizes the information in web traffic logs from various vendors. Be sure to check with your vendor for the most current information.

Data source Target app URL Target app IP Username Origin IP Total traffic Uploaded bytes
Barracuda
Yes
Yes
Yes
Yes
No
No
Blue Coat
Yes
No
Yes
Yes
Yes
Yes
Checkpoint
No
Yes
No
Yes
No
No
Cisco ASA
No
Yes
No
Yes
Yes
No
Cisco FWSM
No
Yes
No
Yes
Yes
No
Cisco Ironport WSA
Yes
Yes
Yes
Yes
Yes
Yes
Cisco Meraki
Yes
Yes
No
Yes
No
No
Clavister NGFW (Syslog)
Yes
Yes
Yes
Yes
Yes
Yes
Dell SonicWall
Yes
Yes
No
Yes
Yes
Yes
Fortigate
No
Yes
No
Yes
Yes
Yes
Juniper SRX
No
Yes
No
Yes
Yes
Yes
Juniper SSG
No
Yes
No
Yes
Yes
Yes
McAfee SWG
Yes
No
No
Yes
Yes
Yes
Meraki (Cisco)
Yes
Yes
No
Yes
No
No
Microsoft Threat Management Gateway
Yes
No
Yes
Yes
Yes
Yes
Palo Alto Networks
Yes
Yes
Yes
Yes
Yes
Yes
Sophos
Yes
Yes
Yes
Yes
Yes
No
Squid (Common)
Yes
No
Yes
Yes
No
Yes
Squid (Native)
Yes
No
Yes
Yes
No
Yes
Websense - Investigative detail report (CSV)
Yes
Yes
Yes
Yes
Yes
Yes
Websense - Internet activity log (CEF)
Yes
Yes
Yes
Yes
Yes
Yes
Zscaler
Yes
Yes
Yes
Yes
Yes
Yes

Supported vendor firewalls and proxies

Office 365 Cloud App Security supports the following firewalls and proxies.

  • Barracuda - Web App Firewall (W3C)

  • Blue Coat Proxy SG - Access log (W3C)

  • Check Point

  • Cisco ASA Firewall (note that you must set the information level to 6)

  • Cisco IronPort WSA

  • Cisco ScanSafe

  • Cisco Merkai - URLs log

  • Dell Sonicwall

  • Fortinet Fortigate

  • Juniper SRX

  • Juniper SSG

  • McAfee Secure Web Gateway

  • Microsoft Forefront Threat Management Gateway (W3C)

  • Palo Alto series Firewall

  • Sophos SG

  • Sophos Cyberoam

  • Squid (Common)

  • Squid (Native)

  • Websense - Web Security Solutions - Investigative detail report (CSV)

  • Websense - Web Security Solutions - Internet activity log (CEF)

  • Zscaler

Note

If a data source that you'd like to use is not included here, you can request that it be added to app discovery. To do that, when you're creating a report, select Other for Data source. Then type the name of the data source that you're trying to upload. We'll review the log, and let you know if we add support for that log type.

Troubleshoot errors when log files are uploaded

After you upload web traffic log files, check the governance log to see if there were any errors. If there are errors, use the information in the following table to resolve those errors.

Error Description Resolution
Unsupported file type
The file uploaded is not a valid log file. For example, an image file.
Upload a text, zip, or gzip file that was directly exported from your firewall or proxy.
Internal error
An internal resource failure was detected.
Click Retry to re-run the task.
The log format does not match
The log format you uploaded does not match the expected log format for this data source.
Verify that the log is not corrupt. Compare and match the log file format to the sample format shown on the upload page.
Transactions are more than 90 days old
All transaction are more than 90 days old and therefore are being ignored.
Export a new log with recent events and re-upload it.
No transactions to catalogue cloud apps
No transaction to any recognized cloud apps are found in the log.
Verify that the log contains outbound traffic information.
Unsupported log type
When you select Data source = Other (unsupported), the log is not parsed. Instead, it is sent for review to the Microsoft Cloud App Security technical team.
The Microsoft Cloud App Security technical team builds a dedicated parser for each data source. Most popular data sources are already supported. When an unsupported data source is uploaded, it is reviewed and added to the list of potential new data source parsers.
When a new parser is added to the feature, a notification is included in the Microsoft Cloud App Security release notes.

Next steps