Event ID: 4138 - An index corruption was detected in component ShadowMerge in catalog ....
Recently I faced several issues regarding the corruption of the index right after a reboot of the server following a patch installation. The issue, as I could observe it, occurs whenever the server is crawling and needs to reboot at the same time.The "techinal details" of the issue written in an event entry logged every 15 minutes or so on the index server
Event Type: Error
Event Source: Office Server Search
Event Category: Content index server
Event ID: 4138
Time: 12:24:46 PM
An index corruption was detected in component ShadowMerge in catalog AnchorProject. Stack trace is .Component: <GUID>
What actually happens on the server that triggers the corruption, is the failure of the indexer (due to reboot) to flush all the shadow index files created during a crawl.
The only clean and recommended way to recover from an index corruption is to completely rebuild the index on all the servers in the farm.
This has as effect the unavailability of the search until the new index is rebuilt. And if you have a big index, that could mean hours, days, even weeks.
Before you go ahead and rebuild the index, there might be a few steps you can try and recover your index by this avoiding the long unavailability time.
IMPORTANT NOTE: this procedure requires specific conditions and might not be successful and therefore you can still be in the position of still having to rebuild the index.
Conditions (if anyone is not met, the restore might fail):
- Event ID:4138 ( naturally )
- The error should not be logged on all servers, specifically, you need at least one Query server where this error is not logged
- The error should be referring only to component Shadow Merge (if Master Merge error is displayed, the procedure will fail )
Actual restore steps:
- Disable and stop OSSsearch service on the indexer.
- Check if the propagation status is idle
- Copy the Portal_Content catalog (<Your_index_location>\<SSPapplicationGUID>\Projects\Portal_Content\Indexer\CiFiles\*.*) from the query server to the indexer in the same location.
- Enable and start OSSsearch service on the indexer.
IMPORTANT . At the time when this error is logged, the corrupted index components are already scheduled for a merge in the master index so the procedure might still fail after all (with additional errors), if some of the errors are commited, so, no warranties here.
I hope this procedure saves you the trouble of rebuilding the entire index,but if it does not, just stop the service on all the servers and go ahead and re-build the index entirely (the cleanest way to rebuild it).
Please see as reference: Index Propagation Protocol Specification
A hotfix was released to prevent this issue from occurring in the future.