您现在访问的是微软AZURE全球版技术文档网站,若需要访问由世纪互联运营的MICROSOFT AZURE中国区技术文档网站,请访问 https://docs.azure.cn.

Azure Monitor 中的 Log Analytics 代理收集自定义日志Collect custom logs with Log Analytics agent in Azure Monitor

Azure Monitor 中的 Log Analytics 代理的自定义日志数据源使你可以从 Windows 和 Linux 计算机上的文本文件中收集事件。The Custom Logs data source for the Log Analytics agent in Azure Monitor allows you to collect events from text files on both Windows and Linux computers. 许多应用程序将信息记录到文本文件,而不是标准日志记录服务(例如 Windows 事件日志或 Syslog)。Many applications log information to text files instead of standard logging services such as Windows Event log or Syslog. 在收集后,可以将数据分析到查询中的各个字段,或者在收集期间将数据提取到各个字段。Once collected, you can either parse the data into individual fields in your queries or extract the data during collection to individual fields.

重要

本文介绍如何使用 Log Analytics 代理 (即 Azure Monitor 使用的代理之一)收集自定义日志。This article covers collecting custom logs with the Log Analytics agent which is one of the agents used by Azure Monitor. 其他代理收集不同的数据,并以不同的方式进行配置。Other agents collect different data and are configured differently. 请参阅 Azure Monitor 代理概述 ,了解可用代理的列表及其可收集的数据。See Overview of Azure Monitor agents for a list of the available agents and the data they can collect.

自定义日志收集

要收集的日志文件必须符合以下条件。The log files to be collected must match the following criteria.

  • 每行日志必须有单个条目,或在每个条目的开头使用时间戳,时间戳可匹配以下任一种格式。The log must either have a single entry per line or use a timestamp matching one of the following formats at the start of each entry.

    YYYY-MM-DD HH:MM:SSYYYY-MM-DD HH:MM:SS
    M/D/YYYY HH:MM:SS AM/PMM/D/YYYY HH:MM:SS AM/PM
    Mon DD, YYYY HH:MM:SSMon DD, YYYY HH:MM:SS
    yyMMdd HH:mm:ssyyMMdd HH:mm:ss
    ddMMyy HH:mm:ssddMMyy HH:mm:ss
    MMM d hh:mm:ssMMM d hh:mm:ss
    dd/MMM/yyyy:HH:mm:ss zzzdd/MMM/yyyy:HH:mm:ss zzz
    yyyy-MM-ddTHH:mm:ssKyyyy-MM-ddTHH:mm:ssK

  • 日志文件不允许会以新条目覆盖文件的循环日志记录或日志轮换。The log file must not allow circular logging or log rotation, where the file is overwritten with new entries.

  • 日志文件必须使用 ASCII 或 UTF-8 编码。The log file must use ASCII or UTF-8 encoding. 不支持其他格式,如 UTF-16。Other formats such as UTF-16 are not supported.

备注

如果日志文件中存在重复项,Azure Monitor 将收集这些项。If there are duplicate entries in the log file, Azure Monitor will collect them. 但是,查询结果将不一致,其中过滤结果显示的事件比结果计数更多。However, the query results will be inconsistent where the filter results show more events than the result count. 重要的是,你要验证日志以确定创建它的应用程序是否是导致该行为的原因,并在可能的情况下对其进行处理,然后再创建自定义日志收集定义。It will be important that you validate the log to determine if the application that creates it is causing this behavior and address it if possible before creating the custom log collection definition.

备注

Log Analytics 工作区支持以下限制:A Log Analytics workspace supports the following limits:

  • 最多只能创建 500 个自定义日志。Only 500 custom logs can be created.
  • 一个表最多仅支持 500 个列。A table only supports up to 500 columns.
  • 列名称的最大字符数为 500。The maximum number of characters for the column name is 500.

重要

自定义日志收集要求编写日志文件的应用程序定期将日志内容刷新到磁盘。Custom log collection requires that the application writing the log file flushes the log content to the disk periodically. 这是因为自定义日志收集依赖于要跟踪的日志文件的文件系统更改通知。This is because the custom log collection relies on filesystem change notifications for the log file being tracked.

定义自定义日志Defining a custom log

使用以下步骤定义自定义日志文件。Use the following procedure to define a custom log file. 请在本文末尾查看添加自定义日志的演示示例。Scroll to the end of this article for a walkthrough of a sample of adding a custom log.

步骤 1。Step 1. 打开自定义日志向导Open the Custom Log Wizard

自定义日志向导在 Azure 门户中运行,使用它可以定义要收集的新自定义日志。The Custom Log Wizard runs in the Azure portal and allows you to define a new custom log to collect.

  1. 在 Azure 门户中,选择“Log Analytics 工作区”> 你的工作区 >“高级设置” 。In the Azure portal, select Log Analytics workspaces > your workspace > Advanced Settings.
  2. 单击“数据” > “自定义日志” 。Click on Data > Custom logs.
  3. 默认情况下,所有配置更改均会自动推送到所有代理。By default, all configuration changes are automatically pushed to all agents. 对于 Linux 代理,配置文件会发送到 Fluentd 数据收集器。For Linux agents, a configuration file is sent to the Fluentd data collector.
  4. 单击“添加+” ,打开自定义日志向导。Click Add+ to open the Custom Log Wizard.

步骤 2.Step 2. 上载和分析示例日志Upload and parse a sample log

首先上载自定义日志示例。You start by uploading a sample of the custom log. 该向导将分析并显示此文件中的条目,以便进行验证。The wizard will parse and display the entries in this file for you to validate. Azure Monitor 将使用指定的分隔符标识每个记录。Azure Monitor will use the delimiter that you specify to identify each record.

“换行” 是默认分隔符,用于每行包含单个条目的日志文件。New Line is the default delimiter and is used for log files that have a single entry per line. 如果行以日期和时间开头且格式符合要求,则可以指定“时间戳” 分隔符,它可支持跨多行的条目。If the line starts with a date and time in one of the available formats, then you can specify a Timestamp delimiter which supports entries that span more than one line.

如果使用时间戳分隔符,则存储在Azure Monitor 中的每个记录的 TimeGenerated 属性将填充为日志文件中为该条目指定的日期/时间。If a timestamp delimiter is used, then the TimeGenerated property of each record stored in Azure Monitor will be populated with the date/time specified for that entry in the log file. 如果使用换行分隔符,则 TimeGenerated 将填充为 Azure Monitor 收集此条目的日期和时间。If a new line delimiter is used, then TimeGenerated is populated with date and time that Azure Monitor collected the entry.

  1. 单击“浏览” ,浏览到示例文件。Click Browse and browse to a sample file. 请注意,此按钮在某些浏览器中可能标记为“选择文件” 。Note that this may button may be labeled Choose File in some browsers.
  2. 单击“下一步”。 Click Next.
  3. 自定义日志向导将上传文件,并列出其标识的记录。The Custom Log Wizard will upload the file and list the records that it identifies.
  4. 更改用于标识新记录的分隔符。根据日志文件中的记录,选择标识效果最好的分隔符。Change the delimiter that is used to identify a new record and select the delimiter that best identifies the records in your log file.
  5. 单击“下一步”。 Click Next.

步骤 3.Step 3. 添加日志集合路径Add log collection paths

必须在可查找自定义日志的代理上定义一个或多个路径;You must define one or more paths on the agent where it can locate the custom log. 可以提供日志文件的特定路径和名称,也可以使用通配符为名称指定路径。You can either provide a specific path and name for the log file, or you can specify a path with a wildcard for the name. 这样,应用程序就可以每天创建新文件,或者在某个文件达到一定大小时创建新文件。This supports applications that create a new file each day or when one file reaches a certain size. 还可以为单个日志文件提供多个路径。You can also provide multiple paths for a single log file.

例如,应用程序可能会每天创建日志文件,将日期包括在如 log20100316.txt 的名称中。For example, an application might create a log file each day with the date included in the name as in log20100316.txt. 此类日志的模式可能是 log*.txt,它将按照应用程序命名方案应用于任何日志文件。A pattern for such a log might be log*.txt which would apply to any log file following the application’s naming scheme.

下表提供了有效模式示例,用来指定不同的日志文件。The following table provides examples of valid patterns to specify different log files.

说明Description 路径Path
Windows 代理上的 C:\Logs 中带 .txt 扩展名的所有文件All files in C:\Logs with .txt extension on Windows agent C:\Logs\*.txtC:\Logs\*.txt
Windows 代理上的 C:\Logs 中具有以 log 开头的名称和 .txt 扩展名的所有文件All files in C:\Logs with a name starting with log and a .txt extension on Windows agent C:\Logs\log*.txtC:\Logs\log*.txt
Linux 代理上的 /var/log/audit 中带 .txt 扩展名的所有文件All files in /var/log/audit with .txt extension on Linux agent /var/log/audit/*.txt/var/log/audit/*.txt
Linux 代理上的 /var/log/audit 中名称以 log 开头并带 .txt 扩展名的所有文件All files in /var/log/audit with a name starting with log and a .txt extension on Linux agent /var/log/audit/log*.txt/var/log/audit/log*.txt
  1. 选择 Windows 或 Linux,指定正在添加的路径格式。Select Windows or Linux to specify which path format you are adding.
  2. 键入路径,并单击 + 按钮。Type in the path and click the + button.
  3. 其他任何路径请重复此步骤。Repeat the process for any additional paths.

步骤 4.Step 4. 提供日志名称及描述Provide a name and description for the log

指定的名称用于日志类型,如上所述。The name that you specify will be used for the log type as described above. 它将始终以 _CL 结尾,以将其与自定义日志区分开来。It will always end with _CL to distinguish it as a custom log.

  1. 为日志键入名称。Type in a name for the log. 系统会自动提供 _CL 后缀。The _CL suffix is automatically provided.
  2. 添加可选“说明” 。Add an optional Description.
  3. 单击“下一步” ,保存自定义日志的定义。Click Next to save the custom log definition.

步骤 5。Step 5. 验证是否正在收集自定义日志Validate that the custom logs are being collected

新自定义日志的初始数据可能需要一个小时才能在 Azure Monitor 中出现。It may take up to an hour for the initial data from a new custom log to appear in Azure Monitor. 它将从指定路径的日志中,收集在自定义日志的定义时间之后生成的条目。It will start collecting entries from the logs found in the path you specified from the point that you defined the custom log. 它不会在自定义日志创建过程中保留上传的条目,但是它将收集它找到的日志文件中的现有条目。It will not retain the entries that you uploaded during the custom log creation, but it will collect already existing entries in the log files that it locates.

Azure Monitor 开始从自定义日志收集后,它的记录将可用于日志查询。Once Azure Monitor starts collecting from the custom log, its records will be available with a log query. 将为自定义日志指定的名称用作查询中的类型Use the name that you gave the custom log as the Type in your query.

备注

如果查询中缺少 RawData 属性,则可能需要关闭并重新打开浏览器。If the RawData property is missing from the query, you may need to close and reopen your browser.

步骤 6.Step 6. 分析自定义日志条目Parse the custom log entries

全部日志条目将存储在名为 RawData 的单个属性中。The entire log entry will be stored in a single property called RawData. 你很可能希望将每个条目中信息的不同部分分离到每条记录的各个属性中。You will most likely want to separate the different pieces of information in each entry into individual properties for each record. 请参考在 Azure Monitor 中分析数据来了解用于将 RawData 分析到多个属性中的选项。Refer to Parse text data in Azure Monitor for options on parsing RawData into multiple properties.

删除自定义日志Removing a custom log

在 Azure 门户中使用以下过程删除以前定义的自定义日志。Use the following process in the Azure portal to remove a custom log that you previously defined.

  1. 从工作区的“高级设置”中的“数据” 菜单 中选择“自定义日志” ,以便列出所有自定义日志。From the Data menu in the Advanced Settings for your workspace, select Custom Logs to list all your custom logs.
  2. 单击要删除的自定义日志旁边的“删除” 。Click Remove next to the custom log to remove.

数据收集Data collection

Azure Monitor 大概每隔 5 分钟就会从每个自定义日志中收集新条目。Azure Monitor will collect new entries from each custom log approximately every 5 minutes. 代理会在从中进行收集的每个日志文件中记录其位置。The agent will record its place in each log file that it collects from. 如果代理在一段时间内处于脱机状态,则 Azure Monitor 将从其上次脱机的位置收集条目,即使这些条目是在代理脱机期间创建的。If the agent goes offline for a period of time, then Azure Monitor will collect entries from where it last left off, even if those entries were created while the agent was offline.

日志条目的全部内容写入到名为 RawData 的单个属性中。The entire contents of the log entry are written to a single property called RawData. 请参阅在 Azure Monitor 中分析文本数据来了解用于将每个导入的日志条目分析到多个属性中的方法。See Parse text data in Azure Monitor for methods to parse each imported log entry into multiple properties.

自定义日志记录属性Custom log record properties

自定义日志记录的类型与提供的日志名称一致,且具有下表中的属性。Custom log records have a type with the log name that you provide and the properties in the following table.

propertiesProperty 说明Description
TimeGeneratedTimeGenerated Azure Monitor 收集该记录时的日期和时间。Date and time that the record was collected by Azure Monitor. 如果日志使用基于时间的分隔符,则此时间是从条目中收集的时间。If the log uses a time-based delimiter then this is the time collected from the entry.
SourceSystemSourceSystem 从中收集记录的代理类型。Type of agent the record was collected from.
OpsManager – Windows 代理,直接连接或 System Center Operations ManagerOpsManager – Windows agent, either direct connect or System Center Operations Manager
Linux - 所有 Linux 代理Linux – All Linux agents
RawDataRawData 收集的条目的完整文本。Full text of the collected entry. 你很可能希望将此数据分析到各个属性中You will most likely want to parse this data into individual properties.
ManagementGroupNameManagementGroupName System Center Operations Manager 代理的管理组名称。Name of the management group for System Center Operations Manage agents. 对于其他代理,这是 AOI-<workspace ID>For other agents, this is AOI-<workspace ID>

添加自定义日志的演示示例Sample walkthrough of adding a custom log

以下部分是创建自定义日志的演示示例。The following section walks through an example of creating a custom log. 收集的示例日志在每行有单个条目,以日期和时间开头,然后是逗号分隔的代码、状态和消息字段。The sample log being collected has a single entry on each line starting with a date and time and then comma-delimited fields for code, status, and message. 几个示例条目如下所示。Several sample entries are shown below.

2019-08-27 01:34:36 207,Success,Client 05a26a97-272a-4bc9-8f64-269d154b0e39 connected
2019-08-27 01:33:33 208,Warning,Client ec53d95c-1c88-41ae-8174-92104212de5d disconnected
2019-08-27 01:35:44 209,Success,Transaction 10d65890-b003-48f8-9cfc-9c74b51189c8 succeeded
2019-08-27 01:38:22 302,Error,Application could not connect to database
2019-08-27 01:31:34 303,Error,Application lost connection to database

上载和分析示例日志Upload and parse a sample log

我们提供其中一个日志文件,然后可以看到它将收集的事件。We provide one of the log files and can see the events that it will be collecting. 在这种情况下,换行是有效的分隔符。In this case New Line is a sufficient delimiter. 如果日志中的单个条目跨过多行,则需要使用时间戳分隔符。If a single entry in the log could span multiple lines though, then a timestamp delimiter would need to be used.

上载和分析示例日志

添加日志集合路径Add log collection paths

日志文件位于 C:\MyApp\LogsThe log files will be located in C:\MyApp\Logs. 每天将创建一个新文件,名称为包括日期的 appYYYYMMDD.log 模式。A new file will be created each day with a name that includes the date in the pattern appYYYYMMDD.log. 此日志的有效模式是 C:\MyApp\Logs\*.logA sufficient pattern for this log would be C:\MyApp\Logs\*.log.

日志集合路径

提供日志名称及描述Provide a name and description for the log

我们使用 名称“MyApp_CL”,然后键入“说明” 。We use a name of MyApp_CL and type in a Description.

日志名称

验证是否正在收集自定义日志Validate that the custom logs are being collected

我们使用简单的查询 MyApp_CL 来从收集的日志中返回所有记录。We use a simple query of MyApp_CL to return all records from the collected log.

没有自定义字段的日志查询

自定义日志的替代方法Alternatives to custom logs

尽管自定义日志在您的数据符合上面列出的条件时很有用,但在某些情况下,您需要另一个策略:While custom logs are useful if your data fits the criteria listed above, there are cases such as the following where you need another strategy:

  • 数据不符合所需的结构,如具有不同格式的时间戳。The data doesn't fit the required structure such as having the timestamp in a different format.
  • 日志文件不符合要求,如文件编码或不受支持的文件夹结构。The log file doesn't adhere to requirements such as file encoding or an unsupported folder structure.
  • 集合之前需要对数据进行预处理或筛选。The data requires preprocessing or filtering before collection.

在不能使用自定义日志收集数据的情况下,请考虑下列备用策略:In the cases where your data can't be collected with custom logs, consider the following alternate strategies:

后续步骤Next steps