IDAHO Department of Labor “Job Search” engine leveraging Windows Azure cloud platform
Idaho Department of Labor’s IT team has leveraged unique capability of Microsoft’ cloud platform Windows Azure, and it is about building “hybrid services and applications”. Hybrid application architecture implies some parts of the application are hosted and running on their on-premise infrastructure, but other modules/parts are hosted and running in the cloud. Hybrid application is another example of how government organizations can explore and adopt cloud platform on their own terms.
Job Search application engine used traditional n-layer architecture and was hosted on-premise, but the Idaho Department of Labor’s IT team was looking for creative ways drive to higher scalability and performance without investing heavily on their infrastructure. Team has come up with their next-gen version of Job Search engine with a new architecture whereby most applications and data are still hosted on their on-premise servers, but the job search engine service (performing the job search) is running in the cloud and listing/data for the open-jobs (non-sensitive data) is stored in cloud database SQL Azure.
Here’s high-level diagram of the architecture of their Job Search engine and service.
- Main ASP.NET application is running on their on-premise infrastructure, and Job Engine service is running in the cloud. Job Service engine was built with .NET Framework in C# programming language, and has both WCF (Windows Communication Foundation) and ASMX interfaces. WCF services not only performs basic function of returning job listings based on the keywords entered by the user, but it also does some interesting things like proximity checks (matching job-seekers location to the location where jobs are available). Results are also ranked by an internal numeric algorithm, distance and the job posting/listing date.
- Sync Framework 2.0 is being utilized to replicate and sync all open job listings from their on-premise database to SQL Azure database hosted in the cloud. This hybrid architecture has enabled them to leverage the cloud for what it offers (higher scalability, availability and performance on commodity servers managed by third-party).
- Open-jobs listings are retrieved from SQL Azure, and stored in the memory cache on each virtual instance for better search performance. Memory cache is updated based regularly on an algorithm. They are also looking at distributed caching capability of the up-coming version of Windows Azure AppFabric.
- Performance counters are currently stored in Azure Table Storage and search metrics are stored in an on premise MSSQL database. They have also written a monitoring application for internal use, which is built using ASP.NET and new ASP.NET 4.0 chart controls and consumes data from counters stored in Windows Azure Table Storage.
- Job Engine service is hosted in only 2 small virtual machine instances of Windows Azure, but they can dial-up or dial-down the number of instances based on schedule/time (peak days in month/week) or cpu-load. This helps them lower their cost and yet provide unlimited scalability on-demand.
- Management of the number of instances running is handled by their on-premise Windows Service (also written in the C# using .NET 4.0 Framework). It relies on the Windows Azure RESTful API and Microsoft.WindowsAzure.StorageClient library to get live statistics and manage the number of Virtual Machines.
Idaho Department of Labor’s IT team started their Windows Azure exploration early September, and deployed Job Search engine service in the Windows Azure (cloud platform) in October. Service is running successfully more than two months, and the department is now looking at other IT work-load scenarios to derive benefits of the Windows Azure cloud platform. Special thanks to Brett Richard (Development Manager) and Greg Gipson (Lead Architect and Developer) from Idaho Department of Labor for this great success story.
Hope this solution provides you with some ideas to adopt hybrid architecture with the cloud platform to drive scalability, efficiency and lower costs.
Here are some resources on learning Windows Azure and SQL Azure
- Windows Azure on MSDN: http://msdn.microsoft.com/windowsazure
- Windows Azure Training Kit: http://msdn.microsoft.com/en-us/wazplatformtrainingcourse.aspx
- Sync Framework: http://msdn.microsoft.com/en-us/sync/default.aspx