Hi Roy,
I'm not sure exactly what you mean by a "file broker" and "load non-signed into the file broker" (is this documented anywhere?).
UWP apps pick files using the FileOpenPicker class, and this class works through a Windows-provided PickerHost.exe process. And I can't assure you that this PickerHost.exe process does load any custom SNE - Shell Namespace Extension - again through standard Windows's Common Dialogs, IFileDialog interfaces in its process.
It's loaded when one uses PickSingleFileAsync (File Open) and PickSaveFileAsync (File Save As). Again, I can confirm this is not a guess, it's a fact. When a UWP app (such as the FilePicker sample) calls these API, a custom SNE is loaded in the PickerHost.exe process.
So, maybe what you mean is a custom SNE won't be loaded into the app process itself, and yes, if this what's you mean, you're right, but it quite irrelevant.
As I told in my remark, it works fine with PickSaveFileAsync (Save As). It's just with PickSingleFileAsync (File Open) that it reports this OutOfMemoryException (please note is not a security nor package identity error).
So, IMHO, either UPW apps have a pretty serious security issue (as you say this shouldn't be allowed at all), either the PickSingleFileAsync case should be investigated more.