SQL Server on Linux, aka project Helsinki: Story behind the idea
As I mentioned in my previous post, I left SQL Server in 2007 to work on what ended up being a super exciting software project: The Midori Operating System. I spent almost eight years working in Midori’s kernel and runtime team. I had worked with exceptional folks on very interesting and challenging problems before but I never thought I would learn so much about languages, runtimes, and kernels as I have while working on Midori.
Late in 2014 the Midori project was completed and I had to figure out what to do next. Basically, I was looking for the next thing to work on. In my next role my goal was to utilize and expand on my prior experiences from working on both SQL Server and Midori.
At some point, during my job search, I connected back with the SQL Server team. The team had been contemplating porting SQL Server to Linux and they were looking for someone to drive the engineering side of the project. This was clearly an interesting lead and it got me curious. However, it is important to understand amount of work a such project would require. I had some serious concerns taking on a port across the full codebase to Linux.
To give you an idea of the effort involved, the SQL Server RDBMS and other services that ship with it in the SQL Server product suite account for more than 40 million lines of C++ code. Even though SQL Server has a resource management layer called SQLOS, the codebase bleeds Win32 semantics throughout. This means a pure port could take years just to get compiling and booting let alone figuring out things like performance and feature parity with SQL Server on Windows. In addition, doing a porting project while other SQL Server innovation is happening in the same codebase would have been a daunting task and keep the team in a close to endless catch-up game.
In conclusion, even though the potential job-offer intrigued me, it felt like an impossible task for one to take on.
Going back to my time in Midori for a bit, sometime in December of 2011 we were working closely with one of our strategic partners, hoping they would adopt Midori for some interesting workloads. One of their requirements was that Midori had to run some Windows applications. One snowy day I was working from home when my boss called me and asked if it is feasible to bring the Microsoft Research-project Drawbridge to Midori. At that time, Drawbridge could already host some applications. So the question was if we could bring support for Drawbridge to Midori. That day, we had a long conversation over the phone and decided to try. Within a few months we had required Win32 applications running on Midori. This is how I was introduced to Drawbridge.
The idea to use Drawbridge as the starting point for bringing SQL Server to Linux came accidentally. I was talking to one of my very close friends and a former Midorian kernel team member on the parking lot about my job search post Midori. During the conversation we were talking about the SQL Server on Linux job opportunity and I mentioned if we only could do it inside Midori then the problem would be expedited due to the Drawbridge presence on Midori. Then I pause for the second thinking out-loud: what if we had something like this on Linux. I dismissed it immediately as the most random idea ever. So we finished the conversation and split. I jumped into the car but the idea of bringing Drawbridge to Linux stuck in my head. I immediately drove to my brother, who by the way did the original Drawbridge support in Midori and shared the idea with him. His initial reaction was that it wouldn't work and I should forget about it. The next day he called me and said that if there was any chance for us to bring SQL Server to Linux in timely manner, leveraging the Drawbridge architecture was really the only feasible way to go.
To further validate the idea, I talked to several folks across Microsoft. The pattern with my brother repeated itself again and again. First folks would reject the idea, but then a day or two later they would come back to me indicating that this might just be the right approach.
During the process I reached out to the Microsoft Research folks who had worked on Drawbridge and to my huge surprise, they actually already had Linux prototype partially up and running.
So I decided to pitch this approach to the SQL Server team and I got an offer to the take the job. I started the new job on early in 2015 and within less than a month we had SQL Server booting up and able to run queries on Linux. Sometimes it would crash or hang, but it was there and running:
Well, I think we might want to remember this day…
See attached picture
I can connect and run queries.
1> select * from sys.dm_os_schedulers
scheduler_address parent_node_id scheduler_id cpu_id status
------------------ -------------- ------------ ----------- -----------------------------------
0x00000000DC190040 0 0 0 VISIBLE ONLINE
0x00000000DC1A0040 0 1 1 VISIBLE ONLINE
0x00000000DC1B0040 0 2 2 VISIBLE ONLINE
0x00000000DC1C0040 0 3 3 VISIBLE ONLINE
0x00000000DC1D0040 0 4 4 VISIBLE ONLINE
0x00000000DC1E0040 0 5 5 VISIBLE ONLINE
0x00000000DC1F0040 0 6 6 VISIBLE ONLINE
0x00000000DC580040 0 7 7 VISIBLE ONLINE
0x00000000DC590040 0 1048578 0 HIDDEN ONLINE
0x00000000DCC10040 64 1048576 0 VISIBLE ONLINE (DAC)
0x00000000DCC60040 0 1048579 1 HIDDEN ONLINE
0x00000000DCC70040 0 1048580 2 HIDDEN ONLINE
0x00000000D6460040 0 1048581 3 HIDDEN ONLINE
(13 rows affected)
1> select @@ver
Msg 137, Level 15, State 2, Server CLIENT, Line 1
Must declare the scalar variable "@@ver".
1> SELECT @@VERSION
Microsoft SQL Server 2014 - 12.0.2000.8 (X64)
Feb 20 2014 20:04:26
Copyright (c) Microsoft Corporation
Enterprise Edition (64-bit) on Windows NT 6.2 (Build 9200: ) (Hypervisor)
(1 rows affected)
SqlServer thinks it is running on HV ;-) but it really runs on L!
Have a great weekend
Having SQL Server up and running so fast helped us expedite the engineering process enormously. As I am sure you can imagine running on other platforms only a fraction of the work required to deliver the product. Now we were in a position to work on our lab automation, testing, packaging, servicing, security, high availability, performance, stress and everything else in parallel. As an example we had to make the complete test infrastructure for SQL Server run against SQL Server on Linux.
During the process we took the ownership of the Drawbridge codebase and started evolving it. That is how the SQL Server Platform Abstraction Layer (SQLPAL) was born.
So far this project has been an amazing journey with an amazing team and I am 100% certain we could never have gotten this far, this fast, without the dedication and skills of the SQL Server on Linux team!
In upcoming posts, I will be covering more technidal details of SQLPAL, SOSv2, why performance and scalability is not an issue, how we handle debugging and other interesting learnings we have been through. We strongly believe that SQL Server will run comparably on all platforms we have taken it to. In fact, we are already getting interesting evidence from some customers that it is indeed the case.
If you are still skeptical, you should be. However, after observing the pattern for so many times now, I am almost certain you will turn around pretty soon.