DIY Client Monitoring - Setting up Tiered Event Forwarding

 I'm a big fan of using built-in features to solve your everyday IT admin problems. One feature that’s been around for a long time but is in my opinion highly under-rated, is Windows Event Forwarding (WEF). My colleague Mark Renoden posted some great material on The Security Log Haystack – Event Forwarding and You a while back and there is another good article written way back in 2008 called Quick and Dirty Large Scale Eventing for Windows. Nothing recent.

These posts do a great job in explaining  WEF setup in a simple environment scenario, but in this post I want to specifically focus on the setup in a large-scale environment where there are geographically dispersed sites or slow/intermittent WAN  links to work with or around around that necessitate a tiered setup. Note you would also need to design a tiered WEF configuration if you want to collect events from more then 100,000 machines as this is the supported limit per event collector.

 STEP 1 - Configure Endpoints

  1. Create a new Group Policy to be linked to your client machines (Event sources).  This Group Policy is going to do two things:
      One - Make sure that the Windows Remote Management (WS-Management Service is running), and
      Two - Tell the clients where to connect to for their WEF configuration.

Computer Configuration\Policies\Windows Settings\Security Settings\System Services\Windows Remote Management (WS-Management) Service
Setting: Automatic

Computer Configuration\Policies\Administrative Templates\Windows Components\Event Forwarding\Configure Target Subscription Manager
Setting: "Server=http://<FQDN of your upstream Windows Event Collector Server>:5985/wsman/SubscriptionManager/WEC"

Tip: If could change other optional transport parameters in this string too (For example if you want to use HTTPS instead of HTTP or specify the Refresh Interval for clients)

2. Duplicate this group policy and change the subscriptionmanager string so that each GPO points at a different Site Event Collection Server.  Depending on your OU structure setup, You will likely want to create a new GPO for each AD Site that contains an Event Collector Server. In this example I have four regional sites and a group policy configured for each:


STEP 2 - Configure the first Site Event Collector (Manually)

Now that the client computers have been told where they need to connect to, we need to configure an event subscription on the first member server that tells the clients what events they should push and how often they should do it.  You will only have to do this on the first event collector server - once you have done that you can export the configuration and import it later to apply the rest of the Site Event Collectors.

  1. Launch and elevated command prompt on your first Site Event Collector Server and run Winrm Quickconfig. This will configure the server with the default configuration to allow it to start accepting http://* WS-Management Requests from other machines.


2.Launch Eventvwr.msc and go to the Subscriptions node to create a new subscription. (You will get a popup if the WEF Service is not already running on the Server, Click Yes to start the service and set it to automatic)


3. Right Click Subscriptions and Create Subscription

4. Create a Subscription Name and Description for your subscription

5. Choose "Source computer initiated" (As we have already configured a group policy to make the clients contact this server for subscriptions)

6. Select Computer Groups and choose which computers may use this Subscription. (We'll use "Domain Computers" here as we will control the scope of each subscription using different client Group Policy Objects)

7. Choose which events should be collected using Select Events. (In this example I'm choosing to forward all Critical events from the Application event log. For finer control of the events use the XML Query Section.)

8. Select Advanced for finer control of the Network optimization settings. (see the "Tips for Scaling/Performance" section of this post to see what each setting does under the hood)   

 9. Click Ok to finish creating the subscription. You should see an indicator button next to the subscription telling you if everything is ok. (Notice the "0" in the "Source Computer Column" - It will take some time for the source computers to pull down the configuration)

10. We will also manually increase the size of the "forwarded events" log file (This will depend on your expected load - here I am setting Site Server to 500MB)

 11. At this point it would be a good idea to check that event forwarding is working on the client side. Jump onto one of your event clients and look for Event 100 in the Microsoft-Windows-Forwarding/Operational log of the Event Source Machine. You should also see the "Source Computer" column from the screenshot above increase)

TIP: If you want to speed this up run GPUPDATE /FORCE on the client


 10. Now that your first site event collector is created, you can use WECUTIL to export the configuration to re-use in Step 3.

To get the Subscription name run:

wecutil es

To export the subscription:

wecutil gs "%subscriptionname%" /f:xml >>"%subscriptionname%.xml"

STEP 3 - Scale out the Site Event Collectors

Now that you have the exported event subscription information you can use group policy to automate the configuration of the remaining Site Event Collectors.

  1. Create a new GPO and apply it only to the Site Event Collector Servers (Good Group Policy Practice is to create a Security group populated with the Site Event Collector Server Computer accounts for this purpose but note if you do that a reboot would be needed before the server will apply the GPO).

2. Configure the WinRM Service GPO Settings

Computer Configuration\Policies\Windows Settings\Security Settings\System Services\Windows Remote Management (WS-Management) Service

Setting: Automatic

Computer Configuration\Administrative Templates\Windows Components\Windows Remote Management (WinRM)\WinRM Service – Allow Remote Server Management through WinRM

Setting: Ipv4 filter *, Ipv6 filter


3. Point the Site Event Collector at the upstream Central Site event collector

Computer Configuration\Policies\Administrative Templates\Windows Components\ Event Forwarding\Configure Event Forwarding\Configure Target Subscription Manager

Setting: "Server=http://<FQDN of CENTRAL Event Collector>:5985/wsman/SubscriptionManager/WEC"


4. Add the following commands into a batch file, and drop in into SYSVOL scripts (DomainName\NETLOGON) share along with the event subscription XML file.

::Start WinRM Service (if not already running), create a default listener and a firewall exception for http
WinRm quickconfig -quiet

::Start WEF Service
wecutil qc -quiet

::Create Subscription from XML files
wecutil cs "\\contoso\NETLOGON\SiteEventSubscription.xml"

::Optional - Set the size of the forwarded events log to 500MB - Here I'm going to use 500MB but you will need to test how often your logs are rolling over and tune as necessary.)
wevtutil sl forwardedevents /ms:500000000

5. Configure a Group Policy Preferences Scheduled task to execute the batch file



STEP 4 -  Setup the CENTRAL event collector

  1. The final piece of the puzzle is to configure the event subscription on the Central Event Collector. This step can be done in very much the same way as Steps 2 or 3 (Depending if you want to use the GUI or not).

If you re-use the exported XML file used to configure the Site Event Collector Servers, Just be sure to make the following changes along the way:

 SubscriptionId - Change the Name of the Subscription to "Central Event Hub"
 Description - Change the Description to something descriptive
 QueryList - Change the Query Filter to collect All Events from the "Forwarded Events" Source log
 AllowedSourceDomainComputers - (Optionally) filter this a security group which contains only the Site Event Collector Servers

Then execute WECUTIL CS

::Optional - Create Subscription from XML files
wecutil cs "\\Path\ServerEventSubscription.xml"

Here is an example  of the GUI and XML file (Note the Query Filter is changed to Forward ALL events from the "Forwarded Events" Log


2. Remember to run the following commands on the Central Event Collector Server as done previously for the site servers

::Start/Setup WinRm Service (if not already configured)
WinRm quickconfig -quiet

::Setup and start the WEC Service (If not already configured)
wecutil qc -quiet

::Optional - Set the size of the forwarded events log to 1GB
wevtutil sl forwardedevents /ms:1000000000

3. Check on the progress of each of your Event Collector Site Servers picking up the event subscription (or use wecutil gr <subscription name>)

4. All done! now go test that events get forwarded.

 Testing Tips


Use Powershell to create some fake events on your downstream event clients, then follow the subscription chain back to the Central Event Collector Server

New-EventLog -LogName Application -source "WEF Testing"
Write-EventLog -LogName Application -EventId 1 -Source "WEF Testing" -EntryType error -Message "Testing Windows Event Forwarding is Working"

Troubleshooting Tips

Windows Event Logs

 If event forwarding is not working, Use the EventLog-ForwardingPlugin\Operational Event Log on the downstream clients to work out why


 Network Connectivity

If you are concerned about hardware/windows firewall blocking communication between Event Client and Event Collector use powershell to test:

Test-NetConnection <Upstream Event Collector Server> -Port 5985

Note: The steps in this post are all based on the assumption that your Event Clients are using WinRm 2.0. If you want to grab events from older clients running WinRm 1.1 (Vista/2008 or XP/2003) you can either download and install the WinRM 2.0 client or make some additional configuration changes on the Event Collector Server. For this step, just know that WinRm 1.1 clients use "compatibility" ports for HTTP/HTTPS ports (80/443).

Performance & Scalability Tips


1. Disk IO: Performance of the Site and Central Event Collector Servers needs to be kept in mind. WEF servers will typically be disk-bound as they write like crazy to the forwarded events file (%SystemRoot%\System32\Winevt\Logs\ForwardedEvents.evtx)

Tip: Consider moving the destination event channel (usually ForwardedEvents) to a separate drive and array if you have one, otherwise it has to share disk I/o with the main OS drive and cause everything else to slow down.

Tip: Consider splitting out the WimRM and WEC service out into their own process's. This will help you keep a closer eye on the impact of WEF on your server with your server monitoring tools 

SC config WinRM type= own
SC config WecSvc type= own

2. Event Viewer Performance: As Event Collectors join the event subscription, you will likely notice that opening eventvwr.msc gets slower. As the subscription information is stored in the registry, and it grows each time an event client joins the subscription, the time to enumerate all of the WEF clients increases so it will be slower on start up.  

Tip: Use WecUtil to manage the subscription. It is NOT recommend to change registry values directly – it can (and likely will) cause corruption of the subscription.

3. Throttling events from clients: There are two ways that you can control the frequency of events being forwarded from clients;

a: Through the  "Event Delivery Optimization"  settings of the event subscription (Note this is configured per subscription); or

b: Configure a Throttle limit on the client via GPO setting (affects ALL forwarders)


Change the "Event Delivery Optimization settings"

Tip: Use Minimize Bandwidth or Custom Settings as described below to throttle events

Normal Mode
This option ensures reliable delivery of events and does not attempt to conserve bandwidth. It is the appropriate choice unless you need tighter control over bandwidth usage or need forwarded events delivered as quickly as possible. It downloads five events at a time unless 15 minutes pass, in which case it downloads any events that are available (less than 5).

Normal Mode Settings:
ConfigurationMode: Normal
DeliveryMode: Push
DeliveryMaxLatencyTime: 900000 (15 mins)
HeartbeatInterval:900000 (15 mins)

Minimize Bandwidth Mode
This option reduces the network bandwidth consumed by event delivery and is a good choice if you are using event forwarding across a wide area network or on a large number of computers on a local area network. It uses push delivery mode (where the forwarding computer contacts the collecting computer) to forward events every six hours.

Minimize Bandwidth Mode Settings:
ConfigurationMode: MinBandwidth
DeliveryMode: Push
DeliveryMaxLatencyTime: 21600000 (6 hours)
HeartbeatInterval: 21600000 (6 hours)

 Minimize Latency Mode
This option ensures that events are delivered with minimal delay. It is an appropriate choice if you are collecting alerts or critical events. It uses push delivery mode and sets a  batch timeout of 30 seconds.

Minimize Latency Settings:
ConfigurationMode: MinLatecy
DeliveryMode: Push
DeliveryMaxLatencyTime: 30000 (30sec)
HeartbeatInterval: 3600000 (6 minutes)

Custom Mode
If you have changed any Subscription setting manually, then it will become Custom.

Custom Settings:

DeliveryMaxItems – maximum number of items, which can be transferred per job (attempt). By default = 5. (wecutil ss SUBSCRIPTION_NAME /dmi:VALUE)

DeliveryMaxLatencyTime - Determines the time WinRM service will wait (in milliseconds) for maximum number of items before sending the existing number of items. For example if DeliveryMaxItems = 100 and DeliveryMaxLatencyTime = 30000 and we have only 55 events collected, then WinRM will wait 30 seconds and then send existing 55 items. (wecutil ss SUBSCRIPTION_NAME /dmlt:VALUE)

HeartbeatInterval - Frequency of events transfers in milliseconds. For example, each hour. (wecutil ss SUBSCRIPTION_NAME /hi:VALUE)


Configure a Throttle limit on the client

In some busy environments it may be necessary to control the number of events sent from event client computers. It is also possible to have multiple event forwarders configured on the same client and you need a way to control all events leaving event sources. 

Tip: Use the group policy setting "Configure Forwarder Resource Usage"  This GPO controls resource usage for the forwarder (Source Computer) by controlling the Events per second sent to the Event Collector. This setting applies across all subscriptions for the forwarder (Source Computer).

Computer Configuration\Policies\Administrative Templates\Windows Components\Event Forwarding\Configure Forwarder Resource Usage 

 I hope this post is helpful in getting you to start using Windows Built-in stuff to solve your IT problems. I've also attached the two GPO's we created here, along with a sample subscription XML file and setup .bat file that you can use to expedite your own setup. (Just remember to change all the server names and paths in all of the config).

In my next post I plan to talk about some of the great things that you can do with these events once you've got em - Once again using built-in and/or free Microsoft stuff. 


Tiered WEF