When I was asked to repatriate an environment running on AWS back to on-premises, the first thing I did was resist the temptation to jump straight into the migration. Before moving any workload, I dedicated time to designing the destination infrastructure. It’s a phase that tends to be underestimated, but it is precisely what determines how smooth the final cutover will be.
The on-premises cluster consisted of 5 ESXi nodes, managed by a single vCenter. Not a massive environment, but appropriately sized to host both application workloads and the infrastructure components required for the project: NSX Manager, the Avi controller, and the HCX service mesh.
Initial Architectural Choices
The first decision was to clearly separate roles and responsibilities:
- Physical networking would be kept as simple as possible, delegating all logical complexity entirely to NSX.
- Application traffic balancing would be managed entirely by VMware Avi, without relying on external third-party solutions.
This clear separation kept everything organized right from the start and prevented mid-migration adjustments.
Deploying NSX
The first actual technical step was deploying the NSX stack. I installed NSX Manager within the cluster and configured Transport Nodes across all 5 nodes, applying a transport profile consistent with the existing physical vSwitch structure.
Once the Transport Nodes were ready, I built out the target network model:
- A Tier-0 Gateway for external routing.
- One or more Tier-1 Gateways to isolate different application contexts.
- The overlay segments to which VMs would connect once migration completed.
The great thing about working with NSX is that you can construct the entire logical network before any VMs even arrive. The segments existed, were fully functional, and were ready to receive traffic well before the first migration wave began.
Deploying VMware Avi
In parallel, I deployed the Avi controller as an OVA into the cluster and integrated it with vCenter as a cloud orchestrator. Avi automatically discovered the datacenter, network segments, and storage layout.
I then configured IPAM profiles for VIP address management and prepared templates for the Virtual Services I would use in the next phase. At this stage, there was no real traffic to balance yet, but having Avi fully integrated and operational made subsequent application configuration much simpler during the migration.
Key Takeaways from This Phase: Infrastructure preparation is often the least visible part of a migration project, but it serves as the foundation for everything else. Having NSX and Avi operational before turning on HCX ensured that the target network model was stable and testable, allowing every migrated VM to be placed directly into its final environment without intermediate hops.
In the next post, we will look in detail at the HCX deployment and the configuration of the service mesh between AWS and on-premises. link here
