Managing a folder with a million files in an ASP application

Lately, I've been working on an interesting issue of IIS6 hosting an ASP application where we see that the ASP requests take long time to process requests and ends up hanging the whole worker process.

 

Well, in such scenarios, the best way to find out the root cause is to take the debugging approach. Using Debugdiag 1.1 we got some memory dumps  and ran the analysis against them.

 

When looking at the report, it seems that all ASP worker threads are waiting on kernel for a file IO operations. We could see worker threads that look like below:

    1:  02eceb68 7c827339 ntdll!KiFastSystemCallRet
    2:  02eceb6c 77e6b974 ntdll!ZwOpenFile+0xc
    3:  02ecee60 77e6bb5f kernel32!FindFirstFileExW+0x1c3
    4:  02ecee80 736f20df kernel32!FindFirstFileW+0x16
    5:  02ecf0ec 736f20a1 scrrun!GetComponent+0x32
    6:  02ecf10c 736f2acd scrrun!MakePathCanonical+0xb8
    7:  02ecf45c 736f2a1f scrrun!PathObject::InternalGetName+0xab
    8:  02ecf468 77d05186 scrrun!FileObject::get_Name+0x13
    9:  02ecf484 77d14acf oleaut32!tPushValJmpTab+0x102
   10:  02ecf514 736f1cf7 oleaut32!CTypeInfo2::Invoke+0x234
   11:  02ecf540 73463ef2 scrrun!FileObject::Invoke+0x2e
   12:  02ecf57c 73463e62 vbscript!IDispatchInvoke2+0xb2
   13:  02ecf5b8 73463f23 vbscript!IDispatchInvoke+0x59
   14:  02ecf6cc 73463c80 vbscript!InvokeDispatch+0x13a
   15:  02ecf6f0 73463e04 vbscript!InvokeByName+0x42
   16:  02ecf7fc 73465576 vbscript!CScriptRuntime::RunNoEH+0x10b4
   17:  02ecf844 734654fa vbscript!CScriptRuntime::Run+0x62
   18:  02ecf93c 73465988 vbscript!CScriptEntryPoint::Call+0x51
   19:  02ecf998 73465beb vbscript!CSession::Execute+0xc8
   20:  02ecf9e8 73464d1b vbscript!COleScript::ExecutePendingScripts+0x144
   21:  02ecfa04 709e2bfa vbscript!COleScript::SetScriptState+0x14d
   22:  02ecfa30 709e2bbc asp!CActiveScriptEngine::TryCall+0x19
   23:  02ecfa6c 709e2af2 asp!CActiveScriptEngine::Call+0x31
   24:  02ecfa88 709e29f6 asp!CallScriptFunctionOfEngine+0x5b
   25:  02ecfadc 709e2921 asp!ExecuteRequest+0x17e
   26:  02ecfb44 709e24e0 asp!Execute+0x24c
 ...
 ...
  

or this

 

    1:  ChildEBP RetAddr  
    2:  02e0ee3c 7c826d49 ntdll!KiFastSystemCallRet
    3:  02e0ee40 77e6c003 ntdll!ZwClose+0xc
    4:  02e0ee84 736f2102 kernel32!FindClose+0x5c
    5:  02e0f0ec 736f20a1 scrrun!GetComponent+0x57
    6:  02e0f10c 736f2acd scrrun!MakePathCanonical+0xb8
    7:  02e0f45c 736f2a1f scrrun!PathObject::InternalGetName+0xab
    8:  02e0f468 77d05186 scrrun!FileObject::get_Name+0x13
    9:  02e0f484 77d14acf oleaut32!tPushValJmpTab+0x102
   10:  02e0f514 736f1cf7 oleaut32!CTypeInfo2::Invoke+0x234
   11:  02e0f540 73463ef2 scrrun!FileObject::Invoke+0x2e
   12:  02e0f57c 73463e62 vbscript!IDispatchInvoke2+0xb2
   13:  02e0f5b8 73463f23 vbscript!IDispatchInvoke+0x59
   14:  02e0f6cc 73463c80 vbscript!InvokeDispatch+0x13a
   15:  02e0f6f0 73463e04 vbscript!InvokeByName+0x42
   16:  02e0f7fc 73465576 vbscript!CScriptRuntime::RunNoEH+0x10b4
   17:  02e0f844 734654fa vbscript!CScriptRuntime::Run+0x62
   18:  02e0f93c 73465988 vbscript!CScriptEntryPoint::Call+0x51
   19:  02e0f998 73465beb vbscript!CSession::Execute+0xc8
   20:  02e0f9e8 73464d1b vbscript!COleScript::ExecutePendingScripts+0x144
   21:  02e0fa04 709e2bfa vbscript!COleScript::SetScriptState+0x14d
   22:  02e0fa30 709e2bbc asp!CActiveScriptEngine::TryCall+0x19
   23:  02e0fa6c 709e2af2 asp!CActiveScriptEngine::Call+0x31
   24:  02e0fa88 709e29f6 asp!CallScriptFunctionOfEngine+0x5b
   25:  02e0fadc 709e2921 asp!ExecuteRequest+0x17e
   26:  02e0fb44 709e24e0 asp!Execute+0x24c

...

...

 

At first, since the files being accessed seemed to be local to the server, it seemed that the issue most likely may be caused by a filter driver affecting the normal file system operations. 

Antiviruses, Backup softwares do install filter drivers and if they are not configured properly they may slow down normal file system operations. But after making sure that no filter driver is intervening, we focused on the files that the application is managing and I was surprised to learn that the application is managing about 2TB of files, well the size is not actually the problem but the number of files in one single folder is the catch here. So about 1 million files were in one single folder that the ASP application is managing.

 

Accessing 1 million files on one single folder will definitely slow the file system, and hence the ASP application managing it. The best way to fix this issue is to re-architect the data into several folders or better move all the data into a database.