The Municipal Property Assessment Corporation (MPAC) is the largest assessment jurisdiction in North America. It assesses more than five million properties in Ontario, worth $3 trillion, in compliance with the Assessment Act and provincial regulations. The Corporation’s property assessments are the foundation of Ontario’s property tax system, which generates $30 billion annually for municipalities to supply local services.
Every day, thousands of property owners access the organization’s external property-valuation application, AboutMyProperty. Assessors use a workflow system to update information and property owners use AboutMyProperty to view their property profiles, assessment information and comparable properties in the area. With different valuations for industrial versus residential and commercial properties, for example, secure data processing and analysis at scale are top priorities.
For the cloud operations and infrastructure team, led by IT director Gopi Balasingam and senior infrastructure architects Chruz Cruz, David Zheng and Ken Tam, security, resilience and cost-efficiency are major preoccupations. As custodians of public data, the company must ensure its technical infrastructure is modern and robust. It’s this directive that has hastened MPAC’s journey to the cloud, Kubernetes and Rancher.
For the cloud operations and infrastructure team at MPAC, security, resilience and cost-efficiency are major preoccupations. As custodians of often sensitive public data, the company must ensure its technical infrastructure is modern and robust. It is this directive that has hastened MPAC’s journey to
the cloud, Kubernetes and Rancher.
The journey to containers
Like many companies, MPAC’s infrastructure has run on-premise in the data center for many years. As cloud computing matured, the team decided early on to migrate its data center infrastructure to the cloud on Amazon Web Services (AWS). Moving to the cloud made sense in those early days when costs were low. Consequently, the aim was to migrate its estate of standalone machines as quickly as possible with little to no disruption.
In 2017, the team migrated the entire application ecosystem by lifting and shifting Spring Boot and Java applications from the old on-premise environment to the cloud, running on standalone compute instances. Doing this manually was time consuming and onerous. Scaling and analytics, for example, required lots of manual intervention, which was highly inefficient. It also started to get expensive. On paper, five cents an hour didn’t seem much but, when the team added 300 and 400 hosts as standalone instances, the costs started to spiral.
The team started to consider containerization as a method to streamline running workloads in the cloud, and to reduce costs. MPAC had an estate of Spring Boot applications that were easy to transplant into containers, then into Kubernetes. Some applications ran in standalone Docker containers, load-balanced with Elastic Load Balancing (ELB). Applications were not self-healing and the team had to write scripts in order to do rolling deployments. When comparing this methodology to running containers in Kubernetes, there was no contest. Kubernetes was much more agile and ‘self-aware’ and soon became the team’s one-stop-shop.
MPAC trialed several management options — experimenting initially with Mesosphere, Docker Swarm, Tectonic (CoreOS) and Rancher 1.6. With Rancher 2.0 some distance away, and with a need to move quickly, the team opted to work with Kubernetes Operations (kops). At that time kops was the standard management tool for Amazon-related Kubernetes clusters, and would allow MPAC to keep its data management systems in Canada — where Amazon Elastic Kubernetes Service (EKS) didn’t yet have a presence.
Kops performed well but, before long, the team soon noticed it had a lot of gaps, particularly in upgrade management and maintenance functionality. Typically, upgrades and maintenance took three to five days and if the team wanted to do a security patch, they were at the mercy of kops’ release cycle. This was particularly problematic when it came to achieving ISO 27001 certification — the team needed an added layer of security to prove they were on top of patch management in order to meet certification requirements.
With the launch of Rancher 2.3, MPAC realized many of these issues would be resolved and in February 2020 conducted a successful two-month POC. They ran a small non-production environment and were so excited with the results they went into full production at once.
As a government-funded organization, with a clear civic duty, we have a responsibility to choose the technologies that will drive great agility and the greatest efficiencies. That’s why we work with Kubernetes and Rancher Labs.
What were the problems MPAC was trying to solve?
Achieving Major Cloud Efficiencies
AWS has been an integral part of MPAC’s infrastructure for over eight years. By early 2017, the company had closed all on-premise and hosted data centers and the focus was to migrate to AWS at speed. They deployed all workloads quickly in AWS US East (Virginia). Then, with data residency concerns in mind, migrated from AWS US East region to the new AWS Canada (Central) Canadian regional service.
The team loved (and still loves) the ease of use, flexibility and tooling inherent in AWS, but as time went on, Cruz and the team noticed costs were accumulating. At a macro level, costs looked low, but on closer analysis in Rancher, an accumulation over time of small over-subscriptions and over-resourcing resulted in a substantial monthly bill. Rancher brought a level of operational visibility to MPAC’s AWS-based Kubernetes containers that allowed the team to closely monitor and identify inefficiencies — and take immediate action.
Suddenly, Cruz and the team could see what kind of resources were truly required to run the business and could scale this analysis down to individual applications. Through taking the simple action to monitor individual processes and find tiny resource inefficiencies, the team estimates MPAC’s monthly AWS bill has reduced by 40 percent. A significant savings.
With the required granular visibility to keep costs down, Cruz and the team can’t imagine a time that they won’t be all-in with AWS.
Transforming Kubernetes Management
The team knew Rancher would enable a repeatable, predictable Kubernetes deployment strategy — one that could be supported collectively, throughout the business. Senior architect David Zheng knew Kubernetes inside and out but was the only one with deep knowledge — a burden on one person. Cruz wanted every team member to be able to manage MPAC’s Kubernetes clusters, whether in a typical deployment scenario or during upgrade and patching cycles.
What Rancher has brought is a central, unified and intuitive Kubernetes management methodology which has democratized the use of containers across the business. For the first time, IT and development teams can work side by side, with full visibility of cluster performance, spinning up and tearing down new instances, in minutes.
Whereas, in kops, upgrades and maintenance took three to five days, in Rancher it now takes a few hours. Upgrades can take place more regularly and patch management is no longer at the mercy of kops’ release cycle. Why is this important? As a public service, MPAC’s compliance relies on its systems being fully updated, at all times. Overall update and patch management times have been reduced by over 80 percent. Finally, cluster deployment and scaling in Rancher is dramatically improved – with highly variable workloads, the team is now able to scale MPAC’s five clusters from a few nodes to hundreds in minutes.
Being a public organization, security was also a primary focus. To achieve ISO 27001, the team needed a reproduceable artifact which would prove the architecture met mean time to recovery (MTTR) requirements. Achieving an accurate reading in kops was difficult — too many nuances and issues arising along the way. In kops, for example, there was a requirement to hand off hard-coded access tokens which could be shared among team members. A better access control method was needed, and Rancher brought this functionality. Automated Role-Based Authentication Control (RBAC) has reduced complexity whilst adding a layer of security to the infrastructure.
Importantly, Rancher has improved MPAC’s overall security posture. Rancher’s built-in security features — CIS benchmarking, RBAC, monitoring and alerting capabilities — provide additional reassurance and are helping the team to maintain compliance in line with its civic responsibilities.
Freedom to Choose – An Agnostic Environment
Finally, having a technically agnostic environment will become increasingly important to MPAC.
MPAC’s Kubernetes landscape is a heterogeneous one. Currently, the team runs an RKE cluster, an EKS cluster imported into Rancher and two AWS Linux clusters also imported into Rancher. Rancher gives MPAC the freedom to use EKS alongside RKE, GKE and any other technology, for that matter. It’s this agnostic, open source approach that the team believes will further boost innovation and drive even greater efficiencies.