Async Multi File Upload Control with Metadata support for Silverlight 2 (Beta 1)
By now a lot of you must be having a lot of fun of Silverlight 2 Beta 1. If you have not downloaded, installed and played with SL 2 Beta 1, I urge you do it right away - you are missing out!!! You can download SL2 here.
I have been working on a number of projects that are using the Silverlight 2 beta bits, and several of them needed file upload functionality. I looked around, and found a few controls, but while all of them did certain things admirably well, there were a few things missing.
- A few of the controls I found allowed you to select multiple files, and upload them, but all of them did so sequentially i.e. one file at a time. Parallel uploads were a necessity for my projects, with the extent of parallelization as a controllable parameter.
- None of them really allowed you to control how a large file gets chunked i.e. at what size boundaries. This can be an important parameterization to improve efficiency.
- Since most of my projects are media related, it is common for me to work on applications where a user is allowed to attach metadata to the file being uploaded. And metadata could vary from application to application. I needed a control that would allow some sort of metadata architecture, that is not hard wired to the control and pluggable by design. I wanted to reuse the same control in many apps, with the consuming apps defining the metadata UI and serialization process, and wiring it to the control through a standard interface.
- And lastly I also needed a completely templatable control.
Needless to say I could not find a combination of all of the above in one control - so I set out to do the next best thing - write one for my own. And the point of this entry is to share it with you. The code is zipped and attached.
First a little about the solution structure. You will find several projects in the solution:
- MFUControl - This the main control project containing most of the functionality behind the control. It also contains all of the control's UI.For those of you who are just starting out with SL 2, or have not played with WPF custom controls before, the default UI for a custom control is defined as a control template that gets applied to the control through a style defined in a file called generic.xaml - this file has to be named exactly this way for the runtime to recognize it(this is similar to WPF). Also, applying a UI to a custom control by way of a control template allows the consumer of the control to later apply other templates to change the L&F.
- ProgressBar - Since SL 2 Beta 1 does not have a ProgressBar control yet, and I needed one, I created one by extending the RangeBase control. Again a default control template is applied and can be found in the generic.xaml file in the project.
- MFUTestApp - This the test harness for the app - standard SL 2 app that has one page with the control on it.
- TestWeb - An ASP.Net web app that can be used to run MFUTestApp from IIS
- UploadService - This the HTTP backend to which the files are uploaded. I have chosen to use a WCF service, using a RESTful interface (implemented using the WCF Web Programming layer), with a single operation that gets invoked using an HTTP POST accepting a file stream.
I will attempt to sparingly explain some of the key pieces of the code here. I have commented the code pretty well - so hopefully in reading through the downloaded code most of it will be obvious to you.
But first things first - below is a screen shot of the control in action, uploading some videos. By the way, if some of you are designers and other kind of creative professionals, and are snickering at the UI, you will have to forgive the paucity of my artistic skills. I can code, but let it be known - I cannot design nice UI's :-).
So the key pieces of the code as I said :
- The actual upload functionality - The upload piece is pretty simple. I use the HttpWebRequest type to do the upload. Initialize it properly, plug in the file name in the HTTP headers, get the request stream, write the file content to it, and POST - that's about it. On the service side, it is even simpler - open the stream, get the file name from the HTTP headers, create a new file at a known location and write it out. This goes on for every file that is being uploaded.
- The parallelization/multi threaded upload - Unfortunately while SL 2 allows you to spin up your own threads, it does not allow making network calls from anywhere other than the UI thread(at least not yet). So I had to look for a way around. The good thing is that the network API's in SL 2 are all async, and use the standard Begin...()/End...() async invocation pattern established in .Net. In this pattern as soon as you make the egin..() call, control returns to your code and the actual unit of work initiated with the call gets automatically shifted to a background thread from the thread pool.
In my code I break up the entire list of files of chosen by the user into sublists based on the number of background threads the user desires to use (this is specified by the MaximumUploadThreads dependency property on the control) - one sublist for each thread. As an example if you had selected 14 files and specified 4 threads, the first 3 sublists each get 4 files, and the last one gets 2.
Once these lists are broken up, I loop through them, and call BeginGetRequestStream() once for the first file in each list. Control returns and only as many threads get employed as there are sublists, and upload starts. To ensure that at any point in time, no more threads get employed, I start uploading the next file in any sublist, once the previous upload for that sublist is completed i.e. initiate the next upload in the callback handler specified in the BeginGetResponseStream() method for the previous upload, which gets invoked once a response comes back from the service after the previous upload completes (or fails).
Note that upping the number of threads will allow you to achieve more parallelization, but at the cost of more background threads on the client.
- The chunking of large files - The control has another dependency property called FileChunkBoundaryInKiloBytes. What you specify is used as the chunking size boundary. For files that are smaller in size than the size specified here are uploaded in one fell swoop, and the progress bar goes straight from 0 to 100%. This is also because the HttpWebRequest has no mechanism for reporting true progress of the network activity :-(. However for files that are larger than the chunking size, are uploaded in chunk sizes as specified. the code follows a different route for those files, and all chunks of a file get uploaded in the same background thread as being used for the sublist to which the file belonged. An internal data structure named FileChunk is used to maintain stream positions, so that successive chunk uploads can be done accurately.
Progress is reported for each chunk, so you will see the progress bar report things somewhat more realistically here. Also additional HTTP headers are passed in to the backend, for each chunk, including the chunk #, total chunks and the chunk size, to facilitate the proper appending of chunks to the file as it gets built on the server with each POST of a chunk.
Note that upping the boundary may be a lucrative option for very large file uploads to achieve faster uploads, but this is client memory, and you can soon get OutOfMemory exceptions if you are not careful.
- The metadata piece - The assumption here is that the client (the app using the control) and the server has some well known contract for the metadata passed and expected respectively, and I needed a way for the control to facilitate the collection, packaging and uploading of that metadata, without itself being aware of any specifics of the metadata - that is the only way it would be reusable across all possible metadata requirements.
The metadata UI is supplied to the control by you defining a Data Template in your app code with the appropriate UI, and supplying it to the control through the FileMetadataTemplate dependency property.
You also need to implement a custom type to act as the data source for this data template. This type needs to implement an interface named IFileMetadataBase that I defined, and that has a single method named SerializeFileMetadata() returning a string. This method is where you supply your serialization logic (in the provided sample I use Json - but you could as easily use XML) , and as long as the serialized form is a string - we are good. This type also will typically implement the metadata elements as properties that you will bind to the appropriate portions of your metadata Data Template.
So how does all this get wired up ? At runtime if the control senses that a Data Template has been specified for metadata, it raises an event called ProvideMetadataInstance and you will need to handle this. The event arguments to this event has a single property of type IFileMetadataBase (the interface I mentioned above), and you set it to a new instance of the implementing type that you created - and voila!!. The control wires up the instance to the data template. When files are uploaded, it calls the function to serialize the metadata to a string, and passes it on as another additional, well named HTTP header, so that the backend can do whatever it wants with it.
I provide a sample of this in the code, with two metadata fields called "Comments" and "Description". Clicking the little button on each file entry, displays the data template which you can fill out.
- The backend - I implement this as a WCF service. But since the interface is simple (HTTP POST + Headers) - I am certain that you ASP.Net (or even other web platform) experts out there can reimplement it using other patterns.
Also the service now puts all uploaded files in a subfolder called Assets under the service directory. This is hardwired in the service code. Change it as you see fit, but do remember to change service code accordingly as well.
- There is also remove functionality in the control - you can check the little checkboxes, a Remove button appears, and you can then remove files from the list
There are a lot of things incomplete in the control:
- Slick UI
- There should be a ceiling enforced on both the chunking size and max thread count to prevent inadvertent memory outage/resource outage on the client
- The button (or whatever else) that shows the metadata, needs to be hidden when no data template is attached
- In future revs of SL 2, as network calls and background threads become more friendly, you can take better advantage of that
- Any other bugs/incomplete features you can find (of which there may be a ton)
- Slick UI (Have I said that already ??)
In any case my goal was to give you all something to play with and certainly improve upon. Let me know if this helped.
Remember to uninstall any older versions of SL on your machine, download and install the runtime , the SDK and the tools, and enjoy!!!