Multi-Tenant AWS ECS Platform Migration

Context

Netmonk is Telkom Indonesia's network monitoring product. It started as an on-premises, Docker-based installation deployed per client, and grew the way successful products often do: one client at a time, each getting their own deployment because that was the fastest thing to do at the time. As the customer base grew, it moved onto AWS Singapore, the closest region available before AWS Jakarta existed, but kept the same one-to-one model, every new client meant provisioning a dedicated EC2 instance.

By the time I was promoted to lead the DevOps initiative, that pattern had produced 100+ monolithic EC2 deployments. The cost of that was obvious, idle capacity sitting on every one of those instances, but the operational cost was worse: 100+ servers to patch, monitor, and maintain individually, none of it shared across clients. I led a team of four engineers to re-architect it.

At that rate, the approach wasn't going to survive the business growing. Cost-wise it worked like a bucket with a hole in it: revenue came in from every new client, but a slice of it leaked straight back out as idle, per-client infrastructure that never got reused anywhere else. Signing more clients faster didn't fix that, it just poured more into the same leaking bucket.

Challenge

One instance per client is a decision that charges compound interest. Every client carries their own idle capacity. Every patch is 100+ operations. Every new signature provisions new infrastructure before it generates any margin.

Going cloud-native and multi-tenant wasn't the hard part, that direction was obvious. The dilemma was which stack to build it on. Kubernetes was the capable, expected answer, but taking it on meant a control plane and an operational learning curve the team didn't have time to absorb. The bucket was still leaking while that decision got made, so whatever we picked had to be something the team could actually run at the timeline the business needed, not just the option with the most capability on paper.

Action

The default answer to "we need container orchestration" is Kubernetes. We chose AWS ECS instead. Part of it was honest: the team didn't have deep Kubernetes expertise at the time, and taking that on mid-migration would have meant learning a control plane while also trying to hit a deadline. But it wasn't only a skills gap. Kubernetes would have given us more capability than the workload actually needed, in exchange for an operational surface a four-person team would then own permanently: control plane, upgrades, and a steeper on-call learning curve for everyone who touched it afterwards. ECS gave us exactly what the migration required, client workloads decoupled from dedicated instances onto shared infrastructure, without a specialist skill set as the price of entry. It also matched what the team already knew: every one of those 100+ EC2 instances was already running its workload through Docker Compose, so ECS's task-definition model was a short step from muscle memory the team had, not a new mental model on top of a deadline.

The platform had also been running in AWS Singapore because, when it was built, that was the nearest region. AWS Jakarta had since opened. Rather than lift-and-shift, I combined the region move with the multi-tenancy transition so the estate was rebuilt correctly once rather than moved twice. The sequence mattered here: the cloud-native, multi-tenant ECS platform went live on AWS Jakarta first and was validated end-to-end before a single client touched it. Decoupling each client off its dedicated EC2 instance only started once that new platform was proven, not in parallel with building it, so migration risk landed on individual client cutovers instead of on an unproven platform. The whole move, standing up the platform and decoupling all 100+ clients onto it, ran in about eight weeks.

Because shared infrastructure raises the stakes on deployment (a bad release now reaches every tenant rather than one), the same activity also streamlined and upgraded the Bitbucket CI/CD pipeline itself, building it with DevSecOps scanning integrated into the pipeline, following the OWASP DevSecOps Guideline, rather than bolted on after.

Tech Stack

AWS ECS runs the shared, multi-tenant workload on Docker containers, one client is now a task definition instead of a dedicated instance. Bitbucket Pipelines handles build and deploy, with Trivy scanning container images for known vulnerabilities and SonarQube scanning the code itself, both as required stages in that same pipeline rather than a separate manual step. EC2 and VPC didn't disappear, they moved down a layer: EC2 now hosts the shared ECS cluster instead of one instance per client, and VPC isolates that cluster and the surrounding network rather than isolating clients from each other.

Results

Roughly 70% lower infrastructure cost, deploys twice as fast, and, the point of the whole exercise, no more per-client instance ceiling. Onboarding a new client stopped being an infrastructure event: provisioning a client used to mean standing up and hardening a new EC2 instance from scratch, and now means adding a task definition to a cluster that's already running and already secured. Deployment became a repeatable process rather than a per-client procedure, and that repeatability is also what made the pipeline upgrade land cleanly, the same DevSecOps scanning stage now covers every client instead of being something to re-verify per instance.

The consolidation also closed a risk that had been running quietly the whole time: one EC2 instance per client meant one point of failure per client. If that instance went down, that client was down, full stop, and recovery meant someone noticing and rebuilding it by hand. On the shared ECS platform, a client's tasks run across multiple availability zones, so a bad node or a failed deploy gets rescheduled and recovered automatically instead of needing someone to notice first.