SUSE Linux Enterprise 11 SP2 Proudly Presents: Linux 3.0

From a developer’s point of view, changing a version number is nothing to go crazy about. Outside of the occasional significance to a version number, version numbers are typically more or less random strings – window dressing, if you will.

The Linux 3.0 kernel does not fall in the window dressing category, which is why we have upgraded SUSE Linux Enterprise 11 SP2 to this new version. This is not to say that we as a company were lagging prior to this upgrade; we have been delivering new functionality with SUSE Linux Enterprise for a long time without revving the kernel.  And it is well known that we already provided our customers with a state-of-the-art mainline kernel with valuable improvements. Don’t get me wrong:  There are some interesting things in Linux 3.0 which we believe our customers will find valuable – which Vojtech Pavlik will highlight in a follow-up post.

Rather, the reason why we feel this change is significant is that it marks a departure from a practice that we have been adhering to with almost no exceptions for more than a decade.

To be clear, the decision to update the Linux kernel via a service pack was not something we took lightly. The kernel has been a very special part of the product and although for many other packages we have done version updates in response to feature requests, to date we have been extremely conservative with the kernel. Case in point: We updated the kernel once during SUSE Linux Enterprise Server 8 (in 2001) and again in SUSE Linux Enterprise Server 11 SP1 (in 2010 – more on that in a moment). During all of the lifetime of SUSE Linux Enterprise Server 9 and SUSE Linux Enterprise Server 10, we stayed with the kernel version we had chosen for the GA version.

The kernel version upgrade as part of SUSE Linux Enterprise Server 11 SP1 was conducted under what we considered rather unique circumstances. The main motivation at this time was that the upstream X.org community was changing their driver model significantly with the adoption of KMS (Kernel Mode Setting), but there were other upstream changes in KVM, the kernel scheduler and other components that made this a compelling choice.

Travelingback in Time

You might ask, “If those circumstances were unique, then why are we doing it again in SP2?” To answer this question, I need to pose another question: What were our reasons for sticking with a stable base kernel throughout the lifetime of SUSE Linux Enterprise Server 9 and SUSE Linux Enterprise Server 10, and backport selected features only?

To a large degree, the answer to this question is a historical one. When SUSE released the first ever Linux Enterprise Server product in 2000, the Linux kernel community was using a development model that worked with two code streams: a stable stream with even minor numbers (2.2.x, and later 2.4.x) and a development stream with odd minor numbers (2.3.x, later 2.5.x). All the innovative stuff happened in the development branch, but the one intended for (and suited for) Enterprise deployments was the stable one. The development branch was supposed to be the bleeding edge, and nobody was concerned if it broke every now and then – which it did.

Work on the development branch would continue for several years, and culminate in a long stabilization phase that eventually resulted in the next major kernel release.

Naturally, this model led to an approach where vendors (like SUSE) would be required to backport a lot of functionality from mainline development kernels in order to enable the latest hardware as well as satisfy customer requests for specific features. In the past, we have been (literally) backporting hundreds if not thousands of patches for each service pack.

Needless to say, this approach certainly had its drawbacks. First and foremost: Backporting functionality from a current kernel to a code base several years old is not only a tedious process, it’s also an error-prone one. Thus, a lot of effort was devoted to ensuring that these backports were, in fact, correct.

The 2.6 Development Model

Starting with the 2.6 release in December 2003, the kernel community changed its development model, and no longer employed a dual code stream approach. All innovative work went into the 2.6 branch, which is also what every distribution is supposed to base their enterprise release on. There is a significant focus on quality, API compatibility and general stability.

The development of the 2.6 series lasted for almost eight years, serving as a catalyst for the release of the 3.0 kernel in August 2011. During this process the development was greatly accelerated, allowing new hardware to be enabled faster; the introduction of new services like ISCSI and KVM virtualization was also increased over the old model. Through all of this development, the 2.6 series has managed to maintain a high level of quality.

Despite these advances, we must keep in mind that not just any workload on a 2.6 kernel can match the requirements of an enterprise platform. That would actually be expecting a tad too much. No matter which kernel base we pick for a SUSE Linux Enterprise release, there is always a significant effort that goes into benchmarking so that we may be able to solidify the code base for optimal performance.

Quality, Stability, Performance

Speaking of stability and quality, I’d like to also address a few topics essential to understanding why we are certain that we can deliver the level of performance, robustness and stability which continue to be a hallmark of SUSE’s solutions.

For an enterprise operating system, stability of all application interfaces is absolutely crucial, and it’s through our ISV certification program – which has over 9000 certified applications – that we have been able to keep these interfaces stable across service packs.  This includes the libraries we ship, the kernel system call interfaces, its I/O control calls and most of the myriad of other ways users space interact with kernel. This also holds for SUSE Linux Enterprise 11 SP2. Our kernel teams have gone over the kernel interfaces that are exposed to user space applications and verified their compatibility with SP1.

In terms of quality, a SUSE Linux Enterprise Server release undergoes a rigorous testing process, performed by our internal QA group, as well as many of our hardware and software partners. Through an ever-growing set of automated tests, our QA group is constantly monitoring the quality of the product in general – especially when it comes to the kernel – in order to ensure its stability under a wide variety of workloads.

Last but not least, we run an assortment of benchmarks on a wide range of hardware, and our kernel teams work hard to address any performance regressions found. This is what I like to call “quality engineering,” which we are very proud of. The processes and rules we have established have served us very well, and have resulted in a succession of SUSE Linux Enterprise releases with the highest level of quality in the past. Needless to say, we have been applying the same processes to SP2, and have found the resulting product to be as robust and stable as any of our previous releases.

Why we think this is important

As we’ve upgraded the kernel in Service Pack 2, we’re now delivering a kernel with high quality, stable APIs and new functionality by upgrading to a current mainline kernel.

And we will likely do it again.

Today, we have the choice. We can either stick to the current kernel version and cherry-pick the features needed, backport them and harden the result – which is what we’ve done for 10 years now – or we can move to a recent kernel, and apply the same hardening process.

We believe that this choice is a great advantage for you, our customers, and we intend to make a conscious decision to develop every future service pack with the same approach. In the case of SP2, we decided that there is significant benefit in choosing the 3.0 kernel. In a future SUSE Linux Enterprise Server 11 service packs, we may come to the same conclusion – or we may find that the demands of enterprise customers are served equally well by just backporting a selected set of features.

In summary, it boils down to this: in the past, we were dealing with a trade-off between stability and innovation. Thanks to the SP2 release, today we can provide stability and innovation without compromising one over the other.

What could be better?!