Managing RID Issuance in Windows Server 2012
Hi all, Ned here again to talk further about managing your RID pool.
By default, a domain has capacity for roughly one billion security principals, such as users, security groups, managed service accounts, and computers. If you run out, you can’t create any more.
There aren’t any domains with that many active objects, of course, but we've seen:
- Provisioning software or administrative scripts accidentally bulk created users, groups, and computers
- Many unused security and distribution groups created by delegated users
- Many domain controllers demoted, restored, or metadata cleaned
- Forest recoveries with an inappropriately set lower RID pool
- The InvalidateRidPool operation performed too frequently
- The RID Block Size registry value increased incorrectly
- RID Master seized
All of these situations use up RIDs unnecessarily, often by mistake. Over many years, a few environments ran out of RIDs and this forced customers to migrate to a new domain or revert with domain and forest recoveries.
Windows Server 2012 addresses issues with RID allocation that have become more likely with the age and ubiquity of Active Directory. These include better event logging, more appropriate limits, and the ability to - in an emergency - increase the overall RID pool allocation by one bit.
Let's get to it.
Periodic Consumption Warnings
Windows Server 2012 adds global RID space event tracking that provide early warning when major milestones are crossed. The model computes the ten (10) percent used mark in the global pool and logs an event when reached. Then it computes the next ten percent used of the remaining and the event cycle continues. As the global RID space is exhausted, events will accelerate as ten percent hits faster in a decreasing pool (but event log dampening will prevent more than one entry per hour). The System event log on every domain controller writes Directory-Services-SAM warning event 16658.
Assuming a default 30-bit global RID space, the first event logs when allocating the pool containing the 107,374,182ND RID. The event rate accelerates naturally until the last checkpoint of 100,000, with 110 events generated in total. The behavior is similar for an unlocked 31-bit global RID space: starting at 214,748,365 and completing in 117 events.
Important
Understand that these events are never "expected": investigate the user, computer, and group creation processes immediately in the domain if you see the event. Creating more than 100 million AD DS objects is quite out of the ordinary!
RID Pool Invalidation Events
There are new event alerts that a local DC RID pool was discarded. These are Informational and could be expected, especially due to the new virtualized domain controller functionality. See the event list later for details on the event.
RID Block Size Cap
Ordinarily, a domain controller requests RID allocations in blocks of 500 RIDs at one time. You can override this default using the following registry REG_DWORD value on a domain controller:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\RID Values
RID Block Size
Prior to Windows Server 2012, there was no maximum value enforced in that registry key, except the implicit DWORD maximum (which has a value of 0xffffffff or 4294967295). This value is considerably larger than the total global RID space. Administrators sometimes inappropriately or accidentally configured RID Block Size with values that exhausted the global RID at a massive rate.
In Windows Server 2012, you cannot set this registry value higher than 15,000 decimal (0x3A98 hexadecimal). This prevents massive unintended RID allocation.
If you set the value higher than 15,000, the value is treated as 15,000 and the domain controller logs event 16653 in the Directory Services event log at every reboot until the value is corrected.
Global RID Space Size Unlock
Prior to Windows Server 2012, the global RID space was limited to 230 (or 1,073,741,823) total RIDs. Once reached, only a domain migration or forest recovery to an older timeframe allowed new SIDs creation - disaster recovery, by any measure. Starting in Windows Server 2012, the 231 bit can be unlocked in order to increase the global pool to 2,147,483,647 RIDs.
AD DS stores this setting in a special hidden attribute named SidCompatibilityVersion on the RootDSE context of all domain controllers. This attribute is not readable using ADSIEdit, LDP, or other tools. To see an increase in the global RID space, examine the System event log for warning event 16655 from Directory-Services-SAM or use the following Dcdiag command:
Dcdiag.exe /TEST:RidManager /v | find /i "Available RID Pool for the Domain"
If you increase the global RID pool, the available pool will change to 2,147,483,647 instead of the default 1,073,741,823. For example:
Warning!
This unlock is intended only to prevent running out of RIDS and is to be used only in conjunction with RID Ceiling Enforcement (see next section). Do not "preemptively" set this in environments that have millions of remaining RIDs and low growth, as application compatibility issues potentially exist with SIDs generated from the unlocked RID pool.
This unlock operation cannot be reverted or removed, except by a complete forest recovery to an earlier backup.
Windows Server 2003 and Windows Server 2008 Domain Controllers cannot issue RIDs when the global RID pool 31st bit is unlocked. Windows Server 2008 R2 domain controllers can use 31st bit RIDs but only if they install hotfix KB2642658. Unsupported and unpatched domain controllers treat the global RID pool as exhausted when unlocked.
Implementing Unlocked Global RID space
To unlock the RID pool to the 31st bit after receiving the RID ceiling alert perform the following steps:
1. Ensure that the RID Master role is running on a Windows Server 2012 domain controller. If not, transfer it to a Windows Server 2012 domain controller
2. Run LDP.exe
3. Click the Connection menu and click Connect for the Windows Server 2012 RID Master on port 389, and then click Bind as a domain administrator
4. Click the Browse menu and click Modify
5. Ensure that DN is blank
6. In Edit Entry Attribute, type:
SidCompatibilityVersion
7. In Values, type:
1
8. Ensure that Add is selected in Operation and click Enter. This updates the Entry List
9. Select the Synchronous option, then click Run:
10. If successful, the LDP output window shows:
***Call Modify...
ldap_modify_ext_s(Id, '(null)',[1] attrs, SvrCtrls, ClntCtrls);
modified "".
11. Confirm the global RID pool increased by examining the System Event Log on that domain controller for Directory-Services-SAM Informational event 16655.
RID Ceiling Enforcement
To afford a measure of protection and elevate administrative awareness, Windows Server 2012 introduces an artificial ceiling on the global RID range at ten (10) percent remaining RIDs in the global space. When within one (1) percent of the artificial ceiling, domain controllers requesting RID pools write Directory-Services-SAM warning event 16657 to their System event log. When reaching the ten percent ceiling on the RID Master FSMO, it writes Directory-Services-SAM event 16657 to its System event log and will not allocate any further RID pools until overriding the ceiling. This forces you to assess the state of the RID Master in the domain and address potential runaway RID allocation; this also protects domains from exhausting the entire RID space.
This ceiling is hard-coded at ten percent remaining of the available RID space. I.e. the ceiling activates when the RID master allocates a pool that includes the RID corresponding to ninety (90) percent of the global RID space.
- For default domains, the first trigger point is 230-1 * 0.90 = 966,367,640 (or 107,374,183 RIDs remaining).
- For domains with an unlocked 31-bit RID space, the trigger point is 231-1 * 0.90 = 1,932,735,282 RIDs (or 214,748,365 RIDs remaining).
You can hit this event twice in the lifetime of a domain - once with a default-sized RID pool and once when you unlock. Preferably never, of course.
When triggered, the RID Master sets AD attribute msDS-RIDPoolAllocationEnabled (common name ms-DS-RID-Pool-Allocation-Enabled) to FALSE on the object:
CN=RID Manager$,CN=System,DC= <domain>
This writes the 16657 event and prevents further RID block issuance to all domain controllers. Domain controllers continue to consume any outstanding RID pools already issued to them.
To remove the block and allow RID pool allocation to continue, set that value to TRUE. On the next RID allocation performed by the RID Master, the attribute will return to its default NOT SET value. After that, there are no further ceilings and eventually, the global RID space runs out, requiring forest recovery or domain migration.
Important
Do not just arbitrarily remove the ceiling once hit - after all, something weird and potentially bad has happened here and your RID Master is trying to tell you. Stop and take stock, find out what caused the increase, and don’t proceed until you are darned sure that you are not going to run out immediately due to some sort of run-away process or procedure in your environment.
Removing the Ceiling Block
To remove the block once reaching the artificial ceiling, perform the following steps:
1. Ensure that the RID Master role is running on a Windows Server 2012 domain controller. If not, transfer it to a Windows Server 2012 domain controller
2. Run LDP.exe
3. Click the Connection menu and click Connect for the Windows Server 2012 RID Master on port 389, and then click Bind as a domain administrator
4. Click the View menu and click Tree, then for the Base DN select the RID Master's own domain naming context. Click Ok
5. In the navigation pane, drill down into the CN=System container and click the CN=RID Manager$ object. Right click it and click Modify
6. In Edit Entry Attribute, type:
MsDS-RidPoolAllocationEnabled
7. In Values, type (in upper case):
TRUE
8. Select Replace in Operation and click Enter. This updates the Entry List.
9. Enable the Synchronous option, then click Run:
10. If successful, the LDP output window shows:
***Call Modify...
ldap_modify_ext_s(ld, 'CN=RID Manager$,CN=System,DC= <domain> ',[1] attrs, SvrCtrls, ClntCtrls);
Modified "CN=RID Manager$,CN=System,DC= <domain> ".
Events and Error Messages
The following new messages log in the System event log on Windows Server 2012 domain controllers. Automated AD health tracking systems, such as System Center Operations Manager, should monitor for these events; all are notable, and some are indicators of critical domain issues.
Event ID |
16653 |
Source |
Directory-Services-SAM |
Severity |
Warning |
Message |
A pool size for account-identifiers (RIDs) that was configured by an Administrator is greater than the supported maximum. The maximum value of 15,000 will be used when the domain controller is the RID master. See https://go.microsoft.com/fwlink/?LinkId=225963 for more information. |
Notes and resolution
The maximum value for the RID Block Size is now 15000 decimal (3A98 hexadecimal). A domain controller cannot request more than 15,000 RIDs. This event logs at every boot until the value is set to a value at or below this maximum.
Event ID |
16654 |
Source |
Directory-Services-SAM |
Severity |
Informational |
Message |
A pool of account-identifiers (RIDs) has been invalidated. This may occur in the following expected cases: 1. A domain controller is restored from backup. 2. A domain controller running on a virtual machine is restored from snapshot. 3. An administrator has manually invalidated the pool. See https://go.microsoft.com/fwlink/?LinkId=226247 for more information. |
Notes and resolution
If this event is unexpected, contact all domain administrators and determine which of them performed the action. The Directory Services event log also contains further information on when one of these steps was performed.
Event ID |
16655 |
Source |
Directory-Services-SAM |
Severity |
Informational |
Message |
The global maximum for account-identifiers (RIDs) has been increased to %1. See https://go.microsoft.com/fwlink/?LinkId=233329 for more information including important operating system interoperability requirements. |
Notes and resolution
If this event is unexpected, contact all domain administrators and determine which of them performed the action. This event notes the increase of the overall RID pool size beyond the default of 230 and will not happen automatically; only by administrative action.
Event ID |
16656 |
Source |
Directory-Services-SAM |
Severity |
Warning |
Message |
Action required! An account-identifier (RID) pool was allocated to this domain controller. The pool value indicates this domain has consumed a considerable portion of the total available account-identifiers. A protection mechanism will be activated when the domain reaches the following threshold of total available account-identifiers remaining: %1. The protection mechanism prevents the allocation of account-identifier (RID) pools needed to allow existing DCs to create additional users, computers and groups, or promote new DCs into the domain. The mechanism will remain active until the Administrator manually re-enables account-identifier allocation on the RID master domain controller. See https://go.microsoft.com/fwlink/?LinkId=228610 for more information. |
Notes and resolution
Contact all domain administrators and inform them that the domain is close to preventing any further principal creation. Interrogate all administrators to find out who or what is creating principals lately and examine the Diagnosis section here for more inventory steps.
Event ID |
16657 |
Source |
Directory-Services-SAM |
Severity |
Error |
Message |
Action required! This domain has consumed a considerable portion of the total available account-identifiers (RIDs). A protection mechanism has been activated because the total available account-identifiers remaining is approximately: %1. The protection mechanism prevents the allocation of account-identifier (RID) pools needed to allow existing DCs to create additional users, computers and groups, or promote new DCs into the domain. The mechanism will remain active until the Administrator manually re-enables account-identifier (RID) allocation on the RID master domain controller. It is extremely important that certain diagnostics be performed prior to re-enabling account creation to ensure this domain is not consuming account-identifiers at an abnormally high rate. Any issues identified should be resolved prior to re-enabling account creation. Failure to diagnose and fix any underlying issue causing an abnormally high rate of account-identifier consumption can lead to account-identifier (RID) pool exhaustion in the domain after which account creation will be permanently disabled in this domain. See https://go.microsoft.com/fwlink/?LinkId=228610 for more information |
Notes and resolution
Contact all domain administrators and inform them that no further security principals can be created in this domain until this protection is overridden. Interrogate all administrators to find out who or what is creating principals lately and examine the Diagnosis section here for more inventory steps. Use the steps above to unlock the 31st RID bit only after you have determined that any runaway issuance is not going to continue.
Event ID |
16658 |
Source |
Directory-Services-SAM |
Severity |
Warning |
Message |
This event is a periodic update on the remaining total quantity of available account-identifiers (RIDs). The number of remaining account-identifiers is approximately: %1. Account-identifiers are used as accounts are created, when they are exhausted no new accounts may be created in the domain. See https://go.microsoft.com/fwlink/?LinkId=228745 for more information |
Notes and resolution
Contact all domain administrators and inform them that RID consumption has crossed a major milestone; determine if this is expected behavior or not by reviewing security trustee creation patterns. To ever see this event would be highly unusual, as it means that at least ~100 million RIDS have been allocated.
These are just some of the excellent supportability changes available in Windows Server 2012 AD DS. For more info, check out the TechNet library starting at:
https://technet.microsoft.com/en-us/library/hh831484
I hope to have more of these kinds of posts coming along soon, as the gloves were taken off this week for Windows Server 2012. You know me though – something shiny goes by and I vanish for weeks. We’ll see…
Ned “The Chronicles of RID” Pyle