您现在访问的是微软AZURE全球版技术文档网站,若需要访问由世纪互联运营的MICROSOFT AZURE中国区技术文档网站,请访问 https://docs.azure.cn.

在 Azure 中使用网络观察程序与 Graylog 来管理和分析网络安全组流日志Manage and analyze network security group flow logs in Azure using Network Watcher and Graylog

网络安全组流日志提供可用来了解 Azure 网络接口入口和出口 IP 流量的信息。Network security group flow logs provide information that you can use to understand ingress and egress IP traffic for Azure network interfaces. 流日志根据网络安全组规则显示出站和入站流、流应用到的网络接口、有关流的 5 元组信息(源/目标 IP、源/目标端口、协议),以及是允许还是拒绝了流量。Flow logs show outbound and inbound flows on a per network security group rule basis, the network interface the flow applies to, 5-tuple information (Source/Destination IP, Source/Destination Port, Protocol) about the flow, and if the traffic was allowed or denied.

可以在启用了流日志记录的网络中使用许多的网络安全组。You can have many network security groups in your network with flow logging enabled. 使用多个已启用流日志记录的网络安全组,可能会导致从日志中分析和获取见解变得非常麻烦。Several network security groups with flow logging enabled can make it cumbersome to parse and gain insights from your logs. 本文提供一种解决方法,即使用 Graylog(开源日志管理和分析工具)和 Logstash(开源服务器端数据处理管道)来集中管理这些网络安全组流日志。This article provides a solution to centrally manage these network security group flow logs using Graylog, an open source log management and analysis tool, and Logstash, an open source server-side data processing pipeline.

警告

以下步骤适用于流日志版本 1。The following steps work with flow logs version 1. 有关详细信息,请参阅针对网络安全组的流日志记录简介For details, see Introduction to flow logging for network security groups. 以下说明在未修改的情况下不适用于版本 2 的日志文件。The following instructions will not work with version 2 of the log files, without modification.

场景Scenario

已使用网络观察程序启用网络安全组流日志。Network security group flow logs are enabled using Network Watcher. 流日志流入 Azure Blob 存储。Flow logs flow in to Azure blob storage. Logstash 插件用于连接和处理 Blob 存储中的流日志并将其发送到 Graylog。A Logstash plugin is used to connect and process flow logs from blob storage and send them to Graylog. 将流日志存储到 Graylog 中之后,可对其进行分析,并在自定义的仪表板中将其可视化。Once the flow logs are stored in Graylog, they can be analyzed and visualized into customized dashboards.

Graylog 工作流

安装步骤Installation Steps

启用网络安全组流日志记录Enable network security group flow logging

对于本方案,必须在帐户的至少一个网络安全组上启用网络安全组流日志记录。For this scenario, you must have network security group flow logging enabled on at least one network security group in your account. 有关启用网络安全组流日志的说明,请参阅以下文章: 网络安全组流日志记录简介For instructions on enabling network security group flow logs, refer to the following article Introduction to flow logging for network security groups.

安装 GraylogSetting up Graylog

对于本示例,需在 Azure 中部署的 Ubuntu 14.04 服务器上配置 Graylog 和 Logstash。In this example, both Graylog and Logstash are configured on an Ubuntu 14.04 Server, deployed in Azure.

  • 请参阅 Graylog 提供的文档,获取有关在 Ubuntu 上进行安装的分步说明。Refer to the documentation from Graylog, for step by step instructions on how install onto Ubuntu.
  • 另请确保遵循文档配置 Graylog Web 界面。Make sure to also configure the Graylog web interface by following the documentation.

本示例使用 Graylog 最低安装要求(即This example uses the minimum Graylog setup (i.e 单个 Graylog 实例),但可将 Graylog 构建为根据系统和生产需求跨资源进行缩放。a single instance of a Graylog), but Graylog can be architected to scale across resources depending on your system and production needs. 有关体系结构注意事项或深层体系结构指南的详细信息,请参阅 Graylog 的 文档体系结构指南For more information on architectural considerations or a deep architectural guide, see Graylog’s documentation and architectural guide.

可根据平台和偏好,以多种方式安装 Graylog。Graylog can be installed in many ways, depending on your platform and preferences. 有关可能的安装方法的完整列表,请参阅 Graylog 的官方文档For a full list of possible installation methods, refer to Graylog's official documentation. Graylog 服务器应用程序在 Linux 分发版上运行,附带以下先决条件:The Graylog server application runs on Linux distributions and has the following prerequisites:

安装 LogstashInstall Logstash

Logstash 用于将 JSON 格式的流日志平展到流元组级别。Logstash is used to flatten the JSON formatted flow logs to a flow tuple level. 平展流日志可使 Graylog 中的日志组织和搜索变得更轻松。Flattening the flow logs makes the logs easier to organize and search in Graylog.

  1. 若要安装 Logstash,请运行以下命令:To install Logstash, run the following commands:

    curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-5.2.0.deb
    sudo dpkg -i logstash-5.2.0.deb
    
  2. 配置 Logstash,以分析流日志并将其发送到 Graylog。Configure Logstash to parse the flow logs and send them to Graylog. 创建 Logstash.conf 文件:Create a Logstash.conf file:

    sudo touch /etc/logstash/conf.d/logstash.conf
    
  3. 将以下内容添加到该文件。Add the following content to the file. 更改突出显示的值,以反映存储帐户详细信息:Change the highlighted values to reflect your storage account details:

     input {
         azureblob
         {
             storage_account_name => "mystorageaccount"
             storage_access_key => "NrUZmx7pJSKaRJzvQbeiZWi5nBRWOTr7Wwr9DrvK7YtDBrADYxT1y0oEExtSlkDnGRt7qcRiZzEBCCyRYND8SxSt"
             container => "insights-logs-networksecuritygroupflowevent"
             registry_create_policy => "start_over"
             codec => "json"
             file_head_bytes => 21
             file_tail_bytes => 9
             # Possible options: `do_not_break`, `with_head_tail`, `without_head_tail`
             break_json_down_policy  => 'with_head_tail'
             break_json_batch_count => 2
             interval => 5
         }
     }
    
     filter {
         split { field => "[records]" }
         split { field => "[records][properties][flows]"}
         split { field => "[records][properties][flows][flows]"}
         split { field => "[records][properties][flows][flows][flowTuples]"
     }
    
      mutate {
         split => { "[records][resourceId]" => "/"}
         add_field =>{
                     "Subscription" => "%{[records][resourceId][2]}"
                     "ResourceGroup" => "%{[records][resourceId][4]}"
                     "NetworkSecurityGroup" => "%{[records][resourceId][8]}"
         }
         convert => {"Subscription" => "string"}
         convert => {"ResourceGroup" => "string"}
         convert => {"NetworkSecurityGroup" => "string"}
         split => { "[records][properties][flows][flows][flowTuples]" => ","}
         add_field => {
                     "unixtimestamp" => "%{[records][properties][flows][flows][flowTuples][0]}"
                     "srcIp" => "%{[records][properties][flows][flows][flowTuples][1]}"
                     "destIp" => "%{[records][properties][flows][flows][flowTuples][2]}"
                     "srcPort" => "%{[records][properties][flows][flows][flowTuples][3]}"
                     "destPort" => "%{[records][properties][flows][flows][flowTuples][4]}"
                     "protocol" => "%{[records][properties][flows][flows][flowTuples][5]}"
                     "trafficflow" => "%{[records][properties][flows][flows][flowTuples][6]}"
                     "traffic" => "%{[records][properties][flows][flows][flowTuples][7]}"
         }
         add_field => {
                     "time" => "%{[records][time]}"
                     "systemId" => "%{[records][systemId]}"
                     "category" => "%{[records][category]}"
                     "resourceId" => "%{[records][resourceId]}"
                     "operationName" => "%{[records][operationName}}"
                     "Version" => "%{[records][properties][Version}}"
                     "rule" => "%{[records][properties][flows][rule]}"
                     "mac" => "%{[records][properties][flows][flows][mac]}"
         }
         convert => {"unixtimestamp" => "integer"}
         convert => {"srcPort" => "integer"}
         convert => {"destPort" => "integer"}
         add_field => { "message" => "%{Message}" }
     }
         date {
             match => ["unixtimestamp" , "UNIX"]
         }
     }
     output {
         stdout { codec => rubydebug }
         udp {
             host => "127.0.0.1"
             port => 12201
         }
     }
    

    提供的 Logstash 配置文件由三个部分组成:input、filter 和 output。The Logstash config file provided is composed of three parts: the input, filter, and output. 输入部分指定 Logstash 要处理的日志的输入源 - 在本例中,你将使用 Azure 博客输入插件(将在后续步骤中安装),使我们可以访问 Blob 存储中存储的网络安全组流日志 JSON 文件。The input section designates the input source of the logs that Logstash will process – in this case, you are going to use an Azure blog input plugin (installed in the next steps) that allows us to access the network security group flow log JSON files stored in blob storage.

然后,filter 部分将平展每个流日志文件,以便使每个单独的流元组及其关联属性成为单独的 Logstash 事件。The filter section then flattens each flow log file so that each individual flow tuple and its associated properties becomes a separate Logstash event.

最后,output 节将每个 Logstash 事件转发到 Graylog 服务器。Finally, the output section forwards each Logstash event to the Graylog server. 若要满足特定需要,可根据需要修改 Logstash 配置文件。To suit your specific needs, modify the Logstash config file, as required.

备注

前面的配置文件假定已在本地主机环回 IP 地址 127.0.0.1 上配置 Graylog 服务器。The previous config file assumes that the Graylog server has been configured on the local host loopback IP address 127.0.0.1. 如果没有,请确保将输出部分中的主机参数更改为正确的 IP 地址。If not, be sure to change the host parameter in the output section to the correct IP address.

有关安装 Logstash 的其他说明,请参阅 Logstash 文档For further instructions on installing Logstash, see the Logstash documentation.

安装适用于 Azure Blob 存储的 Logstash 输入插件Install the Logstash input plug-in for Azure blob storage

使用该 Logstash 插件可以直接从指定的 Blob 存储帐户访问流日志。The Logstash plugin allows you to directly access the flow logs from their designated blob storage account. 若要安装该插件,请从默认的 Logstash 安装目录(在本例中为 /usr/share/logstash/bin)运行以下命令:To install the plug-in, from the default Logstash installation directory (in this case /usr/share/logstash/bin), run the following command:

cd /usr/share/logstash/bin
sudo ./logstash-plugin install logstash-input-azureblob

有关该插件的详细信息,请参阅文档For more information about this plug in, see the documentation.

设置从 Logstash 到 Graylog 的连接Set up connection from Logstash to Graylog

在使用 Logstash 建立了与流日志的连接并设置了 Graylog 服务器之后,现在需要将 Graylog 配置为接受传入的日志文件。Now that you have established a connection to the flow logs using Logstash and set up the Graylog server, you need to configure Graylog to accept the incoming log files.

  1. 使用针对 Graylog 服务器 Web 界面配置的 URL 导航到该界面。Navigate to your Graylog Server web interface using the URL you configured for it. 可以通过将浏览器定向到 http://<graylog-server-ip>:9000/ 来访问该界面You can access the interface by directing your browser to http://<graylog-server-ip>:9000/

  2. 若要导航到配置页,请在顶部导航栏的右侧选择“系统”下拉菜单,并单击“输入”。 To navigate to the configuration page, select the System drop-down menu in the top navigation bar to the right, and then click Inputs. 或者导航到 http://<graylog-server-ip>:9000/system/inputsAlternatively, navigate to http://<graylog-server-ip>:9000/system/inputs

    入门

  3. 若要启动新输入,请在“选择输入”下拉列表中选择“GELF UDP”,并填写表单。To launch the new input, select GELF UDP in the Select input drop-down, and then fill out the form. GELF 是“Graylog 扩展日志格式”(Graylog Extended Log Format) 的缩写。GELF stands for Graylog Extended Log Format. GELF 格式由 Graylog 开发。The GELF format is developed by Graylog. 若要详细了解此格式的优点,请参阅 Graylog 文档To learn more about its advantages, see the Graylog documentation.

    确保将输入绑定到配置 Graylog 服务器的 IP。Make sure to bind the input to the IP you configured your Graylog server on. IP 地址应与 Logstash 配置文件 UDP 输出中的 host 字段匹配。The IP address should match the host field of the UDP output of the Logstash configuration file. 默认端口应是 12201The default port should be 12201. 确保端口与 Logstash 配置文件中指定的 UDP 输出中的 port 字段匹配。Ensure the port matches the port field in the UDP output designated in the Logstash config file.

    屏幕截图显示了 Graylog 输入,其中包含用于启动和查找输入的选项。

    启动输入后,应会看到它显示在“本地输入”部分中,如下图所示:Once you launch the input, you should see it appear under the Local inputs section, as shown in the following picture:

    显示包含所启动的输入的“本地输入”部分的屏幕截图。

    若要详细了解 Graylog 消息输入,请参阅文档To learn more about Graylog message inputs, refer to the documentation.

  4. 完成这些配置后,可使用以下命令启动 Logstash 来开始读入流日志:sudo systemctl start logstash.serviceOnce these configurations have been made, you can start Logstash to begin reading in flow logs with the following command: sudo systemctl start logstash.service.

搜索整个 Graylog 消息Search through Graylog messages

等待一段时间让 Graylog 服务器收集消息后,可以搜索整个消息。After allowing some time for your Graylog server to collect messages, you are able to search through the messages. 若要检查发送到 Graylog 服务器的消息,请在 " 输入 " 配置页中,单击所创建的 GELF UDP 输入的 "显示收到的消息" 按钮。To check the messages being sent to your Graylog server, from the Inputs configuration page click the “Show received messages” button of the GELF UDP input you created. 随后会定向到如下图所示的屏幕:You are directed to a screen that looks similar to the following picture:

屏幕截图显示显示搜索结果、直方图和消息的 Graylog 服务器。

单击蓝色的“%{Message}”链接会展开每条消息,显示每个流元组的参数,如下图所示:Clicking on the blue “%{Message}” link expands each message to show the parameters of each flow tuple, as shown in the following picture:

屏幕截图显示来自 Graylog 服务器的消息详细信息。

默认情况下,如果未选择要搜索的特定消息字段,会将所有消息字段包含在搜索中。By default, all message fields are included in the search if you don’t select a specific message field to search for. 如果想要搜索特定的消息(例如,If you want to search for specific messages (i.e –来自特定源 IP 的流元组) 你可以使用 Graylog 搜索查询语言(如所– flow tuples from a specific source IP) you can use the Graylog search query language as documented

使用 Graylog 分析网络安全组流日志Analyze network security group flow logs using Graylog

在设置并运行 Graylog 之后,现在可以使用它的某些功能来更好地了解流日志数据。Now that Graylog it set up running, you can use some of its functionality to better understand your flow log data. 操作方法之一是使用仪表板创建数据的特定视图。One such way is by using dashboards to create specific views of your data.

创建仪表板Create a dashboard

  1. 在顶部导航栏中,选择“仪表板”或导航到 http://<graylog-server-ip>:9000/dashboards/In the top navigation bar, select Dashboards or navigate to http://<graylog-server-ip>:9000/dashboards/

  2. 在此处,请单击绿色的“创建仪表板”按钮,并在简短表单中填写仪表板的标题和说明。From there, click the green Create dashboard button and fill out the short form with the title and description of your dashboard. 单击“保存”按钮创建新仪表板。Hit the Save button to create the new dashboard. 随后会出现如下图所示的仪表板:You see a dashboard similar to the following picture:

    屏幕截图显示 Graylog 服务器仪表板,其中包含用于创建和编辑仪表板的选项。

添加小组件Add widgets

可以单击仪表板的标题来查看该仪表板,但目前它是空的,因为尚未添加任何小组件。You can click the title of the dashboard to see it, but right now it's empty, since we haven’t added any widgets. 可添加到仪表板中的简易且有用的小组件类型是“快速值”图表,其中显示所选字段的值列表及其分布。An easy and useful type widget to add to the dashboard are Quick Values charts, which display a list of values of the selected field, and their distribution.

  1. 通过从顶部导航栏中选择 " 搜索 ",导航回到接收流日志的 UDP 输入的搜索结果。Navigate back to the search results of the UDP input that’s receiving flow logs by selecting Search from the top navigation bar.

  2. 在屏幕左侧的“搜索结果”窗格下,找到“字段”选项卡,其中列出了每个传入流元组消息的各个字段。 Under the Search result panel to the left side of the screen, find the Fields tab, which lists the various fields of each incoming flow tuple message.

  3. 选择要从中进行可视化的任何所需参数(本示例选择了 IP 源)。Select any desired parameter in which to visualize (in this example, the IP source is selected). 若要显示可能的小组件列表,请单击字段左侧的蓝色下拉箭头,并选择“快速值”生成小组件。To show the list of possible widgets, click the blue drop-down arrow to the left of the field, then select Quick values to generate the widget. 应会看到下图所示的内容:You should see something similar to the following picture:

    源 IP

  4. 在此处,可以选择小组件右上角的“添加到仪表板”按钮,并选择要添加的相应仪表板。From there, you can select the Add to dashboard button at the top right corner of the widget and select the corresponding dashboard to add.

  5. 导航回到仪表板可以看到刚刚添加的小组件。Navigate back to the dashboard to see the widget you just added.

    可将其他各种小组件(例如直方图和计数)添加到仪表板,以跟踪重要指标,如下图中所示的示例仪表板:You can add a variety of other widgets such as histograms and counts to your dashboard to keep track of important metrics, such as the sample dashboard shown in the following picture:

    流日志仪表板

    有关仪表板和其他类型的小组件的更多说明,请参阅 Graylog 的 文档For further explanation on dashboards and the other types of widgets, refer to Graylog’s documentation.

通过将网络观察程序与 Graylog 集成,现在能够以方便、集中的方式管理和可视化网络安全组流日志。By integrating Network Watcher with Graylog, you now have a convenient and centralized way to manage and visualize network security group flow logs. Graylog 提供其他许多强大功能,例如流和警报,使用这些功能也能进一步管理流日志,以及更好地了解网络流量。Graylog has a number of other powerful features such as streams and alerts that can also be used to further manage flow logs and better understand your network traffic. 设置 Graylog 并将其连接到 Azure 之后,可以继续尽情了解它所提供的其他功能。Now that you have Graylog set up and connected to Azure, feel free to continue to explore the other functionality that it offers.

后续步骤Next steps

访问 使用 Power BI 可视化网络安全组流日志,了解如何使用 Power BI 可视化网络安全组流日志。Learn how to visualize your network security group flow logs with Power BI by visiting Visualize network security group flows logs with Power BI.