Web traffic logs and data sources for Office 365 Cloud App Security
|Evaluation >||Planning >||Deployment >||Utilization|
||You are here!
You can use a wide range of web traffic log files and data sources with Office 365 Cloud App Security. However, your web traffic log files must include specific information and be formatted a certain way so that they will work with Office 365 Cloud App Security app discovery reports and the Cloud Discovery dashboard. Use this article as a reference guide for the web traffic logs and data sources you'll use with Office 365 Cloud App Security.
You must be a global administrator, security administrator, or security reader to access the Security & Compliance Center and Office 365 Cloud App Security portal. See Permissions in the Office 365 Security & Compliance Center.
Web traffic log requirements
Office 365 Cloud App Security uses data in your web traffic logs to help you understand which apps people in your organization are using. The more details that are included in the log files, the better visibility you'll have into user activity.
The following sections list the necessary attributes and additional requirements for your web traffic logs to work correctly with Office 365 Cloud App Security.
Office 365 Cloud App Security can't show or analyze attributes that aren't included in your web traffic logs. For example, Cisco ASA Firewall's standard log format does not have the number of uploaded bytes per transaction, the username, or a target URL (only a target IP). Therefore, those attributes are not shown in Cloud Discovery data, and visibility into the cloud apps is limited. For Cisco ASA firewalls, the information level must be set to 6.
Your web traffic logs should include the following attributes:
- Date of the transaction
- Source IP
- Source user (highly recommended)
- Destination IP address
- Destination URL (recommended; URLs provide higher accuracy for cloud app detection than IP addresses)
- Total amount of data (recommended; data information is highly valuable)
- Amount of uploaded or downloaded data (recommended; provides insights about cloud app usage patterns)
- Action taken (allowed or blocked)
In addition to including the attributes listed earlier in this article, your web traffic logs should meet the following requirements:
- The data source for the log files must be supported.
- The format the log files use must match the standard format. When the file is uploaded, app discovery will verify this.
- The events in the log must have taken place no more than 90 days ago.
- The log file must include outbound traffic information that can be analyzed for network activity.
Data attributes for different vendors
The following table summarizes the information in web traffic logs from various vendors. Be sure to check with your vendor for the most current information.
|Data source||Target App URL||Target App IP||Username||Origin IP||Total traffic||Uploaded bytes|
|Cisco ASA (Syslog)||No||Yes||No||Yes||Yes||No|
|Cisco ASA with FirePOWER||Yes||Yes||Yes||Yes||Yes||Yes|
|Cisco Ironport WSA||Yes||Yes||Yes||Yes||Yes||Yes|
|Clavister NGFW (Syslog)||Yes||Yes||Yes||Yes||Yes||Yes|
|SonicWall (formerly Dell)||Yes||Yes||No||Yes||Yes||Yes|
|Digital Arts i-FILTER||Yes||Yes||Yes||Yes||Yes||Yes|
|Palo Alto Networks||No||Yes||Yes||Yes||Yes||Yes|
|Websense - Investigative detail report (CSV)||Yes||Yes||Yes||Yes||Yes||Yes|
|Websense - Internet activity log (CEF)||Yes||Yes||Yes||Yes||Yes||Yes|
Supported vendor firewalls and proxies
Office 365 Cloud App Security supports the following firewalls and proxies.
- Barracuda - Web App Firewall (W3C)
- Blue Coat Proxy SG - Access log (W3C)
- Check Point
- Cisco ASA Firewall (make sure to set the information level to 6)
- Cisco ASA with FirePOWER
- Cisco IronPort WSA
- Cisco ScanSafe
- Cisco Merkai - URLs log
- Clavister NGFW (Syslog)
- Digital Arts i-FILTER
- Fortinet Fortigate
- iboss Secure Cloud Gateway
- Juniper SRX
- Juniper SSG
- McAfee Secure Web Gateway
- Microsoft Forefront Threat Management Gateway (W3C)
- Palo Alto series Firewall
- Sonicwall (formerly Dell)
- Sophos SG
- Sophos XG
- Sophos Cyberoam
- Squid (Common)
- Squid (Native)
- Websense - Web Security Solutions - Investigative detail report (CSV)
- Websense - Web Security Solutions - Internet activity log (CEF)
If a data source that you'd like to use is not included here, you can request that it be added to app discovery. To do that, when you're creating a report, select Other for Data source. Then type the name of the data source that you're trying to upload. We'll review the log, and let you know if we add support for that log type. Alternatively, you can define a custom parser that matches your format.
Troubleshoot errors when log files are uploaded
After you upload web traffic log files, check the governance log to see if there were any errors. If there are errors, use the information in the following table to resolve those errors.
|Unsupported file type
||The file uploaded is not a valid log file. For example, an image file.
||Upload a text, zip, or gzip file that was directly exported from your firewall or proxy.
||An internal resource failure was detected.
||Click Retry to re-run the task.
|The log format does not match
||The log format you uploaded does not match the expected log format for this data source.
|Verify that the log is not corrupt. Compare and match the log file format to the sample format shown on the upload page.|
|Transactions are more than 90 days old
||All transaction are more than 90 days old and therefore are being ignored.
||Export a new log with recent events and re-upload it.
|No transactions to catalogue cloud apps
||No transaction to any recognized cloud apps are found in the log.
||Verify that the log contains outbound traffic information.
|Unsupported log type
||When you select Data source = Other (unsupported), the log is not parsed. Instead, it is sent for review to the Microsoft Cloud App Security technical team.
||The Microsoft Cloud App Security technical team builds a dedicated parser for each data source. Most popular data sources are already supported. When an unsupported data source is uploaded, it is reviewed and added to the list of potential new data source parsers.
When a new parser is added to the feature, a notification is included in the Microsoft Cloud App Security release notes.