For this edition of our Employee Spotlight series, I had a chance to chat with Arpit Garg, Principal Data Architect, to learn about his work with the team to architect the multi-cloud and multi-region strategy for FourKites, including the launch of the FourKites application/platform on Azure.
Arpit explained how this project was critical for customers — improving our existing application architecture by moving towards infrastructure as a code (IAC), app containerization (Kubernetes adoption), adopting additional security practices, CI/CD (continuous integration and continuous delivery/deployment) and moving towards cloud-agnostic solutions.
Arpit brings great technical expertise with him, especially around data platforms, data warehouses and business intelligence systems. He helped scale FourKites’ data ingestion and helped transform how we work with data by building a framework that allows dynamic transformation of incoming data to Fourkites canonical model. This solution reduced the time spent onboarding customers by 70% and removed the implementation team’s dependency on the development team. He also led the team in building a metadata-driven solution to automate the onboarding process of external data providers. Prior to this solution integration with external data provider API used to take anywhere between 2-3 weeks. Post the launch of the solution we were able to onboard these data providers within minutes (< 10 mins).
Here’s further insight into what Arpit learned during the Azure project and how the team has supported him along the way.
How did the need for the Azure project come into place?
We are a cloud-first company, meaning we’ve never had any on-premise solutions. As cloud solutions evolved, it was important to have a multi-cloud strategy in place.
What challenges did you encounter?
The first challenge was determining the strategy to migrate applications to Azure and whether we should use more vendor-neutral services. We had to make decisions around our common infrastructure items, like self-managed services versus managed (AWS/Azure/others) offerings.
Another problem we faced was how to handle the master data/entities in this multi-cloud situation — what apps or databases should be replicated across clouds and should the replication be two-way or one-way. Which tools or solutions should we use for database replication?
As we progressed with multi-cloud, we discovered additional challenges with how things are done in different clouds. For instance, something that is easy to do in one cloud might be different and difficult in the other.
Additionally, there were operational challenges – we had to build additional alerts and prepare for additional needs around site reliability engineering (SRE) and platform observability.
Though it was not an easy project with several challenges along the way, we came together as a team to identify solutions and make it a success.
How did your manager/teammates support this?
The entire team took up this challenging project and spent extra hours defining the strategy and ensuring the project’s successful delivery. It was also an opportunity for the team to learn something new – not many today have experience with both multi-cloud and multi-region solutions.
We engaged with a recommended partner to help us with our Azure journey. The partner planned a discovery phase with us to understand our current cloud architecture and then worked with us on building the cloud architecture for Azure and also helped us in implementing the architecture on Azure.
What are some of the learnings or takeaways from this project?
Time is of the essence. Multi-cloud projects are not easy and you will face challenges. Containerization and making solutions vendor-neutral are helpful. Also, if you adopt more third-party solutions, you lose the quick and easy integration that is supported within the cloud (if you go for similar offerings from cloud providers).
No matter what, you will have hard decisions to make. It is always good to involve all downstream teams like operations and support early on to get feedback on potential changes. Decisions you make today will have a major impact in the future.
Any closing thoughts?
I would like to thank FourKites for displaying confidence in me and giving me this opportunity to work on this large-scale and challenging project.