question

jtorjo avatar image
4 Votes"
jtorjo asked ·

please, either allow System.IO all over HDD, or drastically improve speed of StorageFolder queries

My preference would clearly be to just have access to System.IO, once I've requested


This is clearly possible, since a Destkop Bridge app uses System.IO and can do as it pleases. It would clearly be the best case scenario.

Otherwise, StorageFolder queries should become insanely faster.

I have a 7200rpm HDD, and a folder with 797 pictures. Using System.IO to enumerate the files and get their length is <1ms.

Using a StorageFolder query is simply worse than javascript from 20 years ago - it takes over 9 seconds.

That is >9000 slower.

No matter how you slice or dice it, I don't even have words for how slow this is.

 StorageFileQueryResult query_result = last_query_;
 if (dir != last_dir_) {
     var folder = await StorageFolder.GetFolderFromPathAsync(dir);
    
     // https://blogs.msdn.microsoft.com/adamdwilson/2017/12/20/fast-file-enumeration-with-partially-initialized-storagefiles/
     // note: does not seem to be any big diff compared to .GetFilesAsync()
     QueryOptions query = new QueryOptions() {
         FolderDepth = FolderDepth.Shallow,
         //Filter out all files that have WIP enabled on them
         ApplicationSearchFilter = "System.Security.EncryptionOwners:[] ",
         IndexerOption = IndexerOption.UseIndexerWhenAvailable,
         SortOrder = { new SortEntry { AscendingOrder = false, PropertyName = "System.DateModified" }}
     };
     query.SetPropertyPrefetch(PropertyPrefetchOptions.BasicProperties, new List());
     query_result = folder.CreateFileQueryWithOptions(query);
     last_query_ = query_result;
 }
    
 uint start = 0, len = uint.MaxValue;
 foreach (var f in (await query_result.GetFilesAsync(start,len))) {
     media.Add(new media_info {
         full_file_name = f.Path,
         is_video = is_video(f.Path),
         name = f.Name,
         write_date = f.DateCreated.Date,
         width = 0, height = 0, 
         thumbnail_source = null,
     });
 }

windows-uwp
· 1
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi Richard,

Thanks for the answer:

"I recommend using IndexerOption.OnlyUseIndexerAndOptimizeForIndexedProperties, because only this value tells the system to return partial StorageFiles to your app."

Unfortunately, that is useless for me, because then I would simply miss any file that is not indexed. That basically puts the burden on my users, they need to index their HDD, so my app can be faster.

Best,
John

P.S. For some strange reason, your answer is not visible to me on the browser. I only see it in my email.

0 Votes 0 · ·
jtorjo avatar image
2 Votes"
jtorjo answered ·

So apparently, there is a way to enumerate files using low level API.
It's been so hidden, that close to no one knew about it.

Long story short - here's the link to working code -> https://github.com/microsoft/microsoft-ui-xaml/issues/1465#issuecomment-575987737

· Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

RichardZhang-MSFT avatar image
0 Votes"
RichardZhang-MSFT answered ·

Hi,

You use IndexerOption = IndexerOption.UseIndexerWhenAvailable in your code. If you want a faster query speed, this value is not recommended.

I recommend using IndexerOption.OnlyUseIndexerAndOptimizeForIndexedProperties, because only this value tells the system to return partial StorageFiles to your app.

From document, you can see this:

This tells the system to return partial StorageFiles to your app. Any other IndexerOption value will return full StorageFiles and not see any performance gains.

The IndexerOption.OnlyUseIndexerAndOptimizeForIndexedProperties is valid on Windows 10 1709(16299).



I have a 7200rpm HDD, and a folder with 797 pictures. Using System.IO to enumerate the files and get their length is <1ms.

In the description, you find that System.IO queries are much faster than StorageFolder. If you plan to use System.IO for file queries, this is possible.

You can get a handle to a file from StorageFile using the IStorageItemHandleAccess interface. Once the file handle is obtained, you can initialize System.IO where UWP can directly access ( By default apps have direct file access only to their ApplicationData and InstalledLocations ) , and you can access all Win32 file APIs, like open a brokered handle directly with CreateFileFromApp or CreateFile2 to get a faster query speed.

· 3 · Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

If I understand you correctly, the above solution would require C++/WinRT. And then, I would use it from C#.

I'll have to figure out how to do it.

"You can get a handle to a file from StorageFile using the IStorageItemHandleAccess interface." I assume you meant a handle from StorageFolder ?

Should I understand that a StorageFolder supports the IStorageItemHandleAccess inteface?

An example would help a lot here. Thanks!

0 Votes 0 · ·

Regarding your question, you can check my answer.

1 Vote 1 · ·

Sorry, the indexer is not a good approach, it is inaccurate, can contain the wrong results, the result can change within the result set, and it also may not be available for network and usb drives.

0 Votes 0 · ·
FayWang-MSFT avatar image
2 Votes"
FayWang-MSFT answered ·

Hello,​

Welcome to our Microsoft Q&A platform!

>>Should I understand that a StorageFolder supports the IStorageItemHandleAccess inteface?

For files and folders, there are separate interfaces. The IStorageItemHandleAccess interface only work with files, so you can try the IStorageFolderHandleAccess. You can access the COM interface directly through C#, but you need to define the interface yourself. And you could get a HANDLE by using a StorageFolder and a filename under the belongs to the folder, then you can use the handle to do next. The following code is an example about how to use IStorageFolderHandleAccess.


 [ComImport]
 [Guid("DF19938F-5462-48A0-BE65-D2A3271A08D6")]
 [InterfaceTypeAttribute(ComInterfaceType.InterfaceIsIUnknown)]
 public interface IStorageFolderHandleAccess
 {
     uint Create(string filename, uint creationOptions , uint access, uint sharing, uint option, IntPtr opLockHandler, ref IntPtr handle);
 };
 ​
 ​public sealed partial class MainPage : Page​
 {​
     public MainPage()​
     {​
         this.InitializeComponent();​
     }​
 ​
     private async void MyFunction(object sender, TextControlPasteEventArgs e)​
     {​
 ​
         StorageFolder storageFolder = await KnownFolders.PicturesLibrary.GetFolderAsync("Camera Roll");
         var comInterface = Marshal.GetComInterfaceForObject(storageFolder, typeof(IStorageFolderHandleAccess));
         var storageHandleAccess = (IStorageFolderHandleAccess)Marshal.GetObjectForIUnknown(comInterface);
           
         const uint HAO_READ = 0x120089;
         const uint HAO_WRITE = 0x120116;

         IntPtr handle = IntPtr.Zero;
         storageHandleAccess.Create("fileName", 0x3, HAO_READ | HAO_WRITE, 0, 0, IntPtr.Zero, ref handle);

         var safeHandle = new SafeFileHandle(handle, true);
         ......
 ​
     }​
 }

Once you get the safeHandle handle of the specific file from its folder, you can use it to file read and write using the System.IO API.

>>How can I enumerate all files from a StorageFolder, using System.IO

If you want to enumerate files from localFolder or installedLocation, you can directly use System.IO to enumerate. For example:

 string root = Windows.ApplicationModel.Package.Current.InstalledLocation.Path;
 string path = root + @"\Assets";
    
 DirectoryInfo dirinfo = new DirectoryInfo(path);
 FileInfo[] files = dirinfo.GetFiles();

If you want to enumerate files from other folders that you do no have the direct access, you need to use win32 api to achieve it(e.g. FindFirstFile/FindNextFile).

Thanks.





· 8 · Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

This looks really awesome! Thanks!

But if I want to query for all files in a folder, I would need some code similar to what you just wrote, but for folders. Do you have something like this?

0 Votes 0 · ·

I have updated my answer, you can check it.

1 Vote 1 · ·

Thanks, I've looked at it.

I must be pretty bad at explaining things: what I want is this. Having a StorageFolder, I want to enumerate all its files, using System.IO.

You've shown me how to get a IStorageFolderHandleAccess , thanks for that. But if I read the docs correctly, "Create" will open a file, not create a query to enumerate all files.

How can I enumerate all files from a StorageFolder, using System.IO ?

0 Votes 0 · ·
Show more comments
davidhollowell avatar image
2 Votes"
davidhollowell answered ·

Hi jtorjo, We have a work item on this being considered for a future release of Windows 10. I can't guarantee when or if this will release, but I did request a release note to be added when/if it does.
You can see any updates here: https://docs.microsoft.com/en-us/windows/uwp/whats-new/windows-10-build-18362. There's a "what's new" section, typically found on the left side of that page, which will include any release notes.
Please also note, Q&A is a great place to discuss things like this, but in the future, please consider using Feedback Hub to submit similar requests. You can open feedback hub by clicking the Start button and typing "Feedback Hub".
Best Regards, -David


· 4 · Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi David,

Many thanks for the answer! I did check the page.

How will this be implemented - will this be a speed improvement, or a new API, or just access to System.IO for all the HDD?

Having said that,
1. It's really sad that it takes so long for Microsoft to tackle something as important as this - file system queries should be insanely fast, this is paramount.
2. I did use Feedback Hub a while ago, and no one even bothered to answer.
3. It's really sad that for most stuff, there's already a github page (example: WinUI), but for UWP/Winrt there's no such thing.


Best,
John

0 Votes 0 · ·

This is really great to hear!

0 Votes 0 · ·

Feedback Hub is useless. It doesn't even have a mechanism for closing issues. So there are issues from years ago taking up front page (top votes) with tens of thousands of votes, because they just sit there collecting votes years after the issues have already been resolved.

Feedback hub is not an issue tracking system. You should honestly just get rid of it if it's developers don't understand how issue tracking should work.

1 Vote 1 · ·

Yup, 100% agreed!

0 Votes 0 · ·