question

DavidLechevalier-2080 avatar image
0 Votes"
DavidLechevalier-2080 asked GaryNebbett commented

Issue reported when doing write action on smb share

Hello,

On Windows server 2019, I often have the error 59: An unexpected network error occurred.
I did a python sample script which reproduce the issue doing write test on a Windows smb share.
I got the following errors:


The error returned by the script can be

Traceback (most recent call last):
File "Test_integrity.py", line 153, in <module>
test_integrity()
File "Test_integrity.py", line 149, in test_integrity
file_write_content(path, info)
File "Test_integrity.py", line 125, in file_write_content
os.remove(path)
WindowsError: [Error 59] An unexpected network error occurred: 'U:\\data\\Desktop\.\\pref.ini'

or

Traceback (most recent call last):
File "Test_integrity.py", line 153, in <module>
test_integrity()
File "Test_integrity.py", line 149, in test_integrity
file_write_content(path, info)
File "Test_integrity.py", line 140, in file_write_content
win32file.WriteFile(handle, data, None)
pywintypes.error: (59, 'WriteFile', 'An unexpected network error occurred.')

To reproduce

  1. Create a share on another Windows server

  2. Map a share using the letter U:

  3. Install python + pywin32 (pip install pywin32)

  4. Start the script test.py


Actual result

  • The above error occur

I made some tests:
The issue seems to only occurs from a Windows server 2019.
It occurs with many samba version, (tested on centos7:4.10.16, bionic:4.10.13, xenial:4.3.11), many windows server share (Windows server 2012R2, Windows server 2019)
It occurs even if I force dialect to 3.10, 3.02 or 3.00.
The Windows server is fully updated (Build 17763.rs5_release.180914-1434).
* On procmon, I have the following error

78235-windows1.png

I wanted to join the wireshark trace of my issue (U:\\data\\Desktop\.\\pref.ini) and the script used to reproduce my issue.
But The interface does not allow it.


I would like to understand more this issue and eventually find a workaround.

Thank you for your help.

David Lechevalier.


windows-server-2019windows-server-storage
windows1.png (171.7 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @DavidLechevalier-2080,

Sharing the WireShark trace would help. The normal way of doing this is to place the trace file on a file service like OneDrive, Google Drive, etc. and then post the URL to the file here.

Gary

0 Votes 0 ·

Hello @GaryNebbett

Here are the link to wireshark trace:
https://drive.google.com/file/d/1jkIjGiWbHIBHRodQQ7YhUfgJvLKu6r12/view?usp=sharing => full trace
https://drive.google.com/file/d/1ATUG9BXjbPgATICBmUZNRMDJzwsp4TY2/view?usp=sharing => an extract with the file which does not work (U:\\data\\Desktop.\\pref.ini)




Then you for your help
David.

0 Votes 0 ·
CandyLuo-MSFT avatar image
0 Votes"
CandyLuo-MSFT answered

Hi ,

Based on my understanding, you might have the error 59: An unexpected network error occurred when you create / write / close files. Is that right? Please feel free to let me know if I have any misunderstanding.

In order to get a more concreted idea of this issue, I would appreciate your help to collect the following information for further troubleshooting:

1.Noticed that you have mapped the share. Let's narrow down whether the issue is related with mapping. What's the result when you access the share via UNC patch directly? Will the issue still occur?

2.When the issue happens can you ping SMB server's IP address successfully?

You can run ping IP addresss -t to test network connection.

Best Regards,

Candy


If the Answer is helpful, please click "Accept Answer" and upvote it.

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.



5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

DavidLechevalier-2080 avatar image
0 Votes"
DavidLechevalier-2080 answered DavidLechevalier-2080 published

Hi @CandyLuo-MSFT

I use a script which delete, create, write, close multiple files one by one. (no threading).
1. I have unmaped the drive and updated my script in order to use UNC path. I still have the same issue.
2. The ping did not mention connectivity issue

C:\Users\Administrator>ping 10.90.0.156 -t

Pinging 10.90.0.156 with 32 bytes of data:
Reply from 10.90.0.156: bytes=32 time<1ms TTL=64
Reply from 10.90.0.156: bytes=32 time<1ms TTL=64
Reply from 10.90.0.156: bytes=32 time<1ms TTL=64
Reply from 10.90.0.156: bytes=32 time<1ms TTL=64
Reply from 10.90.0.156: bytes=32 time<1ms TTL=64
Reply from 10.90.0.156: bytes=32 time<1ms TTL=64
Reply from 10.90.0.156: bytes=32 time<1ms TTL=64
Reply from 10.90.0.156: bytes=32 time<1ms TTL=64
Reply from 10.90.0.156: bytes=32 time<1ms TTL=64
Reply from 10.90.0.156: bytes=32 time<1ms TTL=64
Reply from 10.90.0.156: bytes=32 time<1ms TTL=64
Reply from 10.90.0.156: bytes=32 time<1ms TTL=64
Reply from 10.90.0.156: bytes=32 time<1ms TTL=64
Reply from 10.90.0.156: bytes=32 time<1ms TTL=64
Reply from 10.90.0.156: bytes=32 time<1ms TTL=64

Ping statistics for 10.90.0.156:
Packets: Sent = 15, Received = 15, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms


Best Regards,

David.


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

GaryNebbett avatar image
0 Votes"
GaryNebbett answered DavidLechevalier-2080 commented

Hello @DavidLechevalier-2080,

Thanks for the trace files. The trace data does not directly reveal the reason for the error message but it does give hints for further investigation.

There are a vast number of anonymous logon sessions being (successfully) created in the trace (probably one for each of the individual steps in your "delete, create, write" loop), but no session logoffs. The problem might be caused by hitting some limit.

There are probably more efficient ways of performing the steps (the "delete" operation could be eliminated by using appropriate flags to the "create" operation (CREATE_ALWAYS)).

Did you intend to make the Python script available (it is formatted as a hyperlink in your message)? It might throw some light on way so many sessions are established.

My suggestion for a next step would be to use Event Tracing for Windows (ETW) and Windows Performance Recorder (WPR) to get a deeper view of what is happening.

Save the XML below to a file (say "smb.wprp") and then start WPR with the command "wpr -start smb.wprp!SMB -filemode". Then reproduce the problem and stop WPR with the command "wpr -stop smb.etl".

The smb.etl resulting file can probably be substantially compressed (if necessary). If you can make that file available, then I would take a look at it.

Gary

<?xml version="1.0" encoding="utf-8"?>
<!-- wpr -start smb.wprp!SMB -filemode -->
<!-- wpr -stop ?.etl -->
<WindowsPerformanceRecorder Version="1.0" Author="Gary">
  <Profiles>
    <EventCollector Id="ETW" Name="ETW">
      <BufferSize Value="64" />
      <Buffers Value="128" />
    </EventCollector>
    <EventProvider Id="SMBClient" Name="Microsoft-Windows-SMBClient" NonPagedMemory="true">
      <Keywords>
        <Keyword Value="0x403000003C4" />
      </Keywords>
    </EventProvider>
    <EventProvider Id="MRxSMB20" Name="E4AD554C-63B2-441B-9F86-FE66D8084963" NonPagedMemory="true">
      <Keywords>
        <Keyword Value="0xFFFFFFFF" />
      </Keywords>
    </EventProvider>
    <Profile Id="SMB.Verbose.File" Name="SMB" Description="SMB" DetailLevel="Verbose" LoggingMode="File">
      <Collectors>
        <EventCollectorId Value="ETW">
          <EventProviders>
            <EventProviderId Value="SMBClient" />
            <EventProviderId Value="MRxSMB20" />
          </EventProviders>
        </EventCollectorId>
      </Collectors>
    </Profile>
  </Profiles>
</WindowsPerformanceRecorder>



· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @DavidLechevalier-2080,

Just a few more observations after a longer look at the data.

All of the "delete, create, write" loop operations use the same SMB session. No operations are ever performed on the rapidly created anonymous/guest SMB sessions.

The procmon image that you posted shows that the STATUS_UNEXPECTED_NETWORK_ERROR problems occur in the context of the MsMpEng.exe process (Windows Defender) rather than python.exe.

Gary

0 Votes 0 ·

Hi @garynebbett,

Interesting I didn't noticed that. Thanks :)

Actually, when I disable live protection, The issue seems to not occurs.
Here is the link of the procmon trace https://drive.google.com/file/d/1jFanwDn39FLccoOrdwnZarxEeNDH3D4A/view?usp=sharing

What does that mean, there is an issue in the live protection?
is there some best practice to avoid this situation and keep protection ?

Thank you.
I appreciate your help

David.

0 Votes 0 ·
GaryNebbett avatar image GaryNebbett DavidLechevalier-2080 ·

Hello @DavidLechevalier-2080,

Disabling live protection is not desirable and should not be necessary.

In this new (smb.etl) trace, the anonymous/guest logon sessions are not successfully created - the request fails with the error STATUS_INSUFFICIENT_RESOURCES.

The smb.etl trace did not actually "capture" a STATUS_UNEXPECTED_NETWORK_ERROR event; we might have to expand the data included in traces (i.e. add additional providers) to see if we can capture it.

We need to think about next steps. The STATUS_INSUFFICIENT_RESOURCES condition might mean that there are still open sessions on the SAMBA server from previous runs of the script. We might need to restart the SAMBA service in order to return to a "baseline" state for new tests.

I will make some tests on my systems to see what behaviour is to be expected of Windows Defender in this type of scenario.

Gary

0 Votes 0 ·
Show more comments
DavidLechevalier-2080 avatar image
0 Votes"
DavidLechevalier-2080 answered DavidLechevalier-2080 edited

Hi @garynebbett,

Thanks for the instruction.

The etl trace are there https://drive.google.com/file/d/1yJnb9cZmgGv_Ahq-cmibXo2PFTe_fTxz/view?usp=sharing.
I added the python script here https://drive.google.com/file/d/1y9fWl5Ko5srY7r2BwyYA89dHHZYhZwiF/view?usp=sharing

The script actually uses the CREATE_ALWAYS flags. but the issue was more often with the remove.
The main part of the script is only doing this

The beginning of the script is for generating a list of files to test.
Then, there is the main part which do write operation

          if os.path.exists(path):
                  os.remove(path)
     
          try:
                  handle = win32file.CreateFile(
                          path,
                          win32con.GENERIC_WRITE,
                          0,
                          None,
                          win32con.CREATE_ALWAYS,
                          0,
                          0)
          except Exception as err:
                  print(err)
                  sys.exit(1)
     
          win32file.WriteFile(handle, data, None)
          win32file.CloseHandle(handle)

Best regards,
David.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

GaryNebbett avatar image
0 Votes"
GaryNebbett answered

Hello @DavidLechevalier-2080,

I performed some simple tests between two Windows 10 systems (one acting as the SMB server) and the behaviour of Windows Defender seemed similar to that in your traces: Defender "opens" the new files on the remote file share, possibly just to canonicalize/normalize the path to the new file. I also see a lot of Defender initiated attempts to create a guest session with the remote server (which fail in my case because guest access is not enabled).

You mentioned in your first message that the problem occurs for you also with some Windows servers. Is guest access enabled on those servers?

If the problem can be reproduced when the server does not allow guest access then that would help resolve concerns that the large number of successful guest SMB sessions is causing some limit to be hit.

It would also be helpful/interesting to actually find evidence of a STATUS_INSUFFICIENT_RESOURCES error in an ETW/WPR trace. There are lots of things that could be traced and the tricky thing is finding a balance between the amount of trace data produced and its usefulness. I am happy to make suggestions about what to trace (by providing updated XML for the .wprp file) - if you are happy to share the resulting trace data.

Gary

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

DavidLechevalier-2080 avatar image
0 Votes"
DavidLechevalier-2080 answered

Hello @GaryNebbett

I tried again between 2 windows servers 2019. The guest access is not activated The registry value AllowInsecureGuestAuth is set to 0 (default value not modified)
The trace is https://drive.google.com/file/d/1XtI97giBLA2v4n0fiv1dhDUWY_W_Reyq/view?usp=sharing

If you can give me the XML file, I can do more tests with pleasure. My goal is to find solution to this issue.

Thank you

David.


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

GaryNebbett avatar image
0 Votes"
GaryNebbett answered

Hello @DavidLechevalier-2080,

There is still a lot to puzzle over in the existing traces, but this XML might provide some more insight. It adds tracing of file operations, so it should see the same events and errors that procmon records; it also adds tracing of the "Redirected Drive Buffering SubSystem" (rdbss.sys - a driver that can report a STATUS_UNEXPECTED_NETWORK_ERROR and is part of the layered implementation of remote file systems).

Gary

 <?xml version="1.0" encoding="utf-8"?>
 <!-- wpr -start smb.wprp!SMB -filemode -->
 <!-- wpr -stop ?.etl -->
 <WindowsPerformanceRecorder Version="1.0" Author="Gary">
   <Profiles>
     <SystemCollector Id="NTKL" Name="NT Kernel Logger">
       <BufferSize Value="64" />
       <Buffers Value="128" />
     </SystemCollector>
     <EventCollector Id="ETW" Name="ETW">
       <BufferSize Value="64" />
       <Buffers Value="128" />
     </EventCollector>
     <SystemProvider Id="Map">
       <Keywords>
     <Keyword Value="Loader" /> 
     <Keyword Value="ProcessThread" /> 
       </Keywords>
     </SystemProvider>
     <EventProvider Id="File" Name="Microsoft-Windows-Kernel-File" NonPagedMemory="true" Stack="true" />
     <EventProvider Id="SMBClient" Name="Microsoft-Windows-SMBClient" NonPagedMemory="true" Stack="true">
       <Keywords>
         <Keyword Value="0x403000003C4" />
       </Keywords>
     </EventProvider>
     <EventProvider Id="MRxSMB20" Name="E4AD554C-63B2-441B-9F86-FE66D8084963" NonPagedMemory="true">
       <Keywords>
         <Keyword Value="0xFFFFFFFF" />
       </Keywords>
     </EventProvider>
     <EventProvider Id="Rdbss" Name="Microsoft-Windows-Remotefs-Rdbss" NonPagedMemory="true" Stack="true" />
     <EventProvider Id="RdbssLog" Name="0086EAE4-652E-4DC7-B58F-11FA44F927B4" NonPagedMemory="true">
       <Keywords>
         <Keyword Value="0xFFFFFFFF" />
       </Keywords>
     </EventProvider>
     <EventProvider Id="Packets" Name="Microsoft-Windows-NDIS-PacketCapture" NonPagedMemory="true" />
     <Profile Id="SMB.Verbose.File" Name="SMB" Description="SMB" DetailLevel="Verbose" LoggingMode="File">
       <Collectors>
         <SystemCollectorId Value="NTKL">
           <SystemProviderId Value="Map">
         <Stacks>
           <Stack Value="ProcessCreate" />
         </Stacks>
           </SystemProviderId>
         </SystemCollectorId>
         <EventCollectorId Value="ETW">
           <EventProviders>
             <EventProviderId Value="File" />
             <EventProviderId Value="SMBClient" />
             <EventProviderId Value="MRxSMB20" />
             <EventProviderId Value="Rdbss" />
             <EventProviderId Value="RdbssLog" />
             <!--<EventProviderId Value="Packets" />-->
           </EventProviders>
         </EventCollectorId>
       </Collectors>
     </Profile>
   </Profiles>
 </WindowsPerformanceRecorder>

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

DavidLechevalier-2080 avatar image
0 Votes"
DavidLechevalier-2080 answered

Hello @GaryNebbett

Here are the last trace between the two Windows server 2019.
https://drive.google.com/file/d/1N4A3vZnIghKFku54KAfccUIusKL6BamV/view?usp=sharing

I hope, It will help to understand the situation.

Thanks,
David.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

GaryNebbett avatar image
0 Votes"
GaryNebbett answered

Hello @DavidLechevalier-2080,

That was helpful, but there is still a lot for me to do. Examining the file I/O and sorting by error code, we can now quickly identify the operations that incurred a STATUS_UNEXPECTED_NETWORK_ERROR problem (0xC00000C4, 3221225668 (decimal)):

79233-image.png

Looking now at all the events that occurred from start to end of the first I/O that encountered the problem, we see:

79242-image.png

I now need to do a bit of research about what could lead to this code path being taken through RxFsdCommonDispatch.

Gary


image.png (50.1 KiB)
image.png (35.7 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

DavidLechevalier-2080 avatar image
0 Votes"
DavidLechevalier-2080 answered

Hello @GaryNebbett,

Thank you for your help.
If you need more information, ask me.

I'm really interested to know what is the root cause of this behavior.

Best regards,
David.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.