question

TravisRothlisberger-5588 avatar image
0 Votes"
TravisRothlisberger-5588 asked TravisRothlisbergerAdmin-2740 answered

Advice on shared, file-based data access from Ubuntu VM

We are in the middle of collecting data that requires processing in Matlab, and are looking to improve the processing performance by running it a multi-core VM with Matlab installed. The data is currently located in SharePoint and shared via OneDrive synchronization from users' personal computers. The best solution for accessing the data from the VM is not clear, so I am looking for some advice. It's feasible that we could migrate the files away from SharePoint and into some kind of Azure storage if there is clear benefit to doing so. I'm currently exploring the following possibilities:

  1. Sync the data to storage on the same virtual drive as the VM itself using a OneDrive client on the VM. But Microsoft doesn’t produce a client for Linux, so this would use the open-source Linux client. There would still have to be a download step before doing any data processing, but hopefully that would be much faster since it should all be occurring within the Microsoft cloud backbone.

  2. Setup an Azure Files space to be mounted by the VM and have users access the space via SMB as a network drive. The data would be immediately available to the VM once its uploaded, but we would also need to also migrate about 200GB of existing data from SharePoint.

  3. Setup an Azure Files space to be mounted by the VM and have it synchronize from SharePoint via an Azure Logic App. This would allow us to keep the existing SharePoint \ OneDrive based workflow without requiring a separate synchronization step that is run from the VM.

I'm an Azure rookie and this is just what I've found from looking around. I was hoping there was a more direct way to connect from an Azure VM to SharePoint cloud storage. Is there a better option? How would file access performance differ between synchronizing the files onto the VM drive vs connecting to an Azure Files drive via SMB? Thanks in advance.





azure-virtual-machines
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Due to recent circumstances, we are now leaning strongly toward option #2 (dedicated Azure Files space), but are still looking for advice on how to best migrate the existing SharePoint data to Azure Files. I haven't been able to identify a method for server-side cloning of the data from SharePoint to Azure Files.

0 Votes 0 ·
EchoDu-MSFT avatar image
0 Votes"
EchoDu-MSFT answered TravisRothlisberger-5588 commented

Hello @travisrothlisberger-5588 ,

Welcome to Q&A Forum!

According to my research, MATLAB is third-party software.

The Q&A forum does not support the third-party software. We suggest that you can go to the corresponding three-party official website or platform for professional help.

Thanks,
Echo Du



· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

The use of Matlab is incidental. The question is how to best access data as descsribed on the Azure Ubuntu VM.

0 Votes 0 ·

@EchoDu-MSFT - You've misunderstood. Please remove the DELETED tag from this post. It's been impossible to find somebody at Microsoft who is willing to offer advice on the best option for Ubuntu VM accessible shared file storage.

0 Votes 0 ·
TravisRothlisbergerAdmin-2740 avatar image
0 Votes"
TravisRothlisbergerAdmin-2740 answered

I ended up installing rclone on an Azure VM. I then mounted the Azure Files to the VM and used rclone's OneDrive configuration to copy from SharePoint to the mount. Even though everything was running on Microsoft servers, it still seemed fairly slow and inefficient. There should really be a better way to move data between these two storage platforms. I'd still be interested in hearing about that if anybody knows a better option going forward.


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.