MPAC Logo
Industry: .edu & .gov
Location: Canada
Download Full Story

Kubernetes underpins the Municipal Property Assessment Corporation’s drive to fulfil its civic duty with increased efficiency and security

Highlights

  • 40% reduction in cloud costs
  • 85% reduction in cluster deployment time—from days to minutes
  • 80% reduction in update and patch management time—from five days to hours

Products

The Municipal Property Assessment Corpo­ration (MPAC) is the largest assessment ju­risdiction in North America. It assesses more than five million properties in Ontario, worth $3 trillion, in compliance with the Assess­ment Act and provincial regulations. The Corporation’s property assessments are the foundation of Ontario’s property tax system, which generates $30 billion annually for municipalities to supply local services.

Every day, thousands of property owners access the organization’s external prop­erty-valuation application, AboutMyProp­erty. Assessors use a workflow system to update information and property owners use AboutMyProperty to view their prop­erty profiles, assessment information and comparable properties in the area. With different valuations for industrial versus residential and commercial properties, for example, secure data processing and analysis at scale are top priorities.

For the cloud operations and infrastructure team, led by IT director Gopi Balasingam and senior infrastructure architects Chruz Cruz, David Zheng and Ken Tam, security, re­silience and cost-efficiency are major pre­occupations. As custodians of public data, the company must ensure its technical in­frastructure is modern and robust. It’s this directive that has hastened MPAC’s journey to the cloud, Kubernetes and Rancher.

Overview

For the cloud operations and infrastructure team at MPAC, security, resilience and cost-efficiency are major preoccupations. As custodians of often sensitive public data, the company must ensure its technical infrastructure is modern and robust. It is this directive that has hastened MPAC’s journey to
the cloud, Kubernetes and Rancher.

The journey to containers

Like many companies, MPAC’s infrastruc­ture has run on-premise in the data center for many years. As cloud computing ma­tured, the team decided early on to migrate its data center infrastructure to the cloud on Amazon Web Services (AWS). Moving to the cloud made sense in those early days when costs were low. Consequently, the aim was to migrate its estate of standalone machines as quickly as possible with little to no disruption.

In 2017, the team migrated the entire ap­plication ecosystem by lifting and shifting Spring Boot and Java applications from the old on-premise environment to the cloud, running on standalone compute instances. Doing this manually was time consuming and onerous. Scaling and analytics, for ex­ample, required lots of manual intervention, which was highly inefficient. It also started to get expensive. On paper, five cents an hour didn’t seem much but, when the team added 300 and 400 hosts as standalone instances, the costs started to spiral.

The team started to consider container­ization as a method to streamline running workloads in the cloud, and to reduce costs. MPAC had an estate of Spring Boot applica­tions that were easy to transplant into con­tainers, then into Kubernetes. Some applica­tions ran in standalone Docker containers, load-balanced with Elastic Load Balancing (ELB). Applications were not self-healing and the team had to write scripts in order to do rolling deployments. When comparing this methodology to running containers in Ku­bernetes, there was no contest. Kubernetes was much more agile and ‘self-aware’ and soon became the team’s one-stop-shop.

MPAC trialed several management op­tions — experimenting initially with Meso­sphere, Docker Swarm, Tectonic (CoreOS) and Rancher 1.6. With Rancher 2.0 some distance away, and with a need to move quickly, the team opted to work with Kuber­netes Operations (kops). At that time kops was the standard management tool for Amazon-related Kubernetes clusters, and would allow MPAC to keep its data man­agement systems in Canada — where Am­azon Elastic Kubernetes Service (EKS) didn’t yet have a presence.

Kops performed well but, before long, the team soon noticed it had a lot of gaps, particularly in upgrade management and maintenance functionality. Typically, up­grades and maintenance took three to five days and if the team wanted to do a security patch, they were at the mercy of kops’ release cycle. This was particularly problematic when it came to achieving ISO 27001 certification — the team needed an added layer of security to prove they were on top of patch management in order to meet certification requirements.

With the launch of Rancher 2.3, MPAC re­alized many of these issues would be re­solved and in February 2020 conducted a successful two-month POC. They ran a small non-production environment and were so excited with the results they went into full production at once.

As a government-funded organization, with a clear civic duty, we have a responsibility to choose the technologies that will drive great agility and the greatest efficiencies. That’s why we work with Kubernetes and Rancher Labs.

What were the problems MPAC was trying to solve?

Achieving Major Cloud Efficiencies

AWS has been an integral part of MPAC’s infrastructure for over eight years. By early 2017, the company had closed all on-prem­ise and hosted data centers and the focus was to migrate to AWS at speed. They de­ployed all workloads quickly in AWS US East (Virginia). Then, with data residency con­cerns in mind, migrated from AWS US East region to the new AWS Canada (Central) Canadian regional service.

The team loved (and still loves) the ease of use, flexibility and tooling inherent in AWS, but as time went on, Cruz and the team no­ticed costs were accumulating. At a macro level, costs looked low, but on closer analy­sis in Rancher, an accumulation over time of small over-subscriptions and over-re­sourcing resulted in a substantial monthly bill. Rancher brought a level of operational visibility to MPAC’s AWS-based Kubernetes containers that allowed the team to closely monitor and identify inefficiencies — and take immediate action.

Suddenly, Cruz and the team could see what kind of resources were truly required to run the business and could scale this analysis down to individual applications. Through taking the simple action to moni­tor individual processes and find tiny re­source inefficiencies, the team estimates MPAC’s monthly AWS bill has reduced by 40 percent. A significant savings.

With the required granular visibility to keep costs down, Cruz and the team can’t imag­ine a time that they won’t be all-in with AWS.

Transforming Kubernetes Management

The team knew Rancher would enable a re­peatable, predictable Kubernetes deploy­ment strategy — one that could be supported collectively, throughout the business. Senior architect David Zheng knew Kubernetes in­side and out but was the only one with deep knowledge — a burden on one person. Cruz wanted every team member to be able to manage MPAC’s Kubernetes clusters, wheth­er in a typical deployment scenario or during upgrade and patching cycles.

What Rancher has brought is a central, uni­fied and intuitive Kubernetes management methodology which has democratized the use of containers across the business. For the first time, IT and development teams can work side by side, with full visibility of cluster performance, spinning up and tear­ing down new instances, in minutes.

Whereas, in kops, upgrades and mainte­nance took three to five days, in Rancher it now takes a few hours. Upgrades can take place more regularly and patch manage­ment is no longer at the mercy of kops’ release cycle. Why is this important? As a public service, MPAC’s compliance relies on its systems being fully updated, at all times. Overall update and patch management times have been reduced by over 80 per­cent. Finally, cluster deployment and scal­ing in Rancher is dramatically improved – with highly variable workloads, the team is now able to scale MPAC’s five clusters from a few nodes to hundreds in minutes.

Being a public organization, security was also a primary focus. To achieve ISO 27001, the team needed a reproduceable artifact which would prove the architecture met mean time to recovery (MTTR) require­ments. Achieving an accurate reading in kops was difficult — too many nuances and issues arising along the way. In kops, for ex­ample, there was a requirement to hand off hard-coded access tokens which could be shared among team members. A bet­ter access control method was needed, and Rancher brought this functionality. Auto­mated Role-Based Authentication Control (RBAC) has reduced complexity whilst add­ing a layer of security to the infrastructure.

Importantly, Rancher has improved MPAC’s overall security posture. Rancher’s built-in security features — CIS benchmarking, RBAC, monitoring and alerting capabilities — provide additional reassurance and are helping the team to maintain compliance in line with its civic responsibilities.

Freedom to Choose – An Agnostic Environment

Finally, having a technically agnostic envi­ronment will become increasingly impor­tant to MPAC.

MPAC’s Kubernetes landscape is a het­erogeneous one. Currently, the team runs an RKE cluster, an EKS cluster imported into Rancher and two AWS Linux clusters also imported into Rancher. Rancher gives MPAC the freedom to use EKS alongside RKE, GKE and any other technology, for that matter. It’s this agnostic, open source ap­proach that the team believes will further boost innovation and drive even greater efficiencies.