## From Monoliths to Macroservices: A Study in Scaling Systems and Culture. **Jordan Rinke** Hopeful future Sr. Director, Infrastructure Engineering

## My Career Story: From Hands-On to High-Level A multi-decade journey focused on one thing: **building and scaling platforms and building teams that build and scale platforms** and _having a bit of fun along the way_.

### The Foundation: Hands-On Engineering - **Sr Systems Admin -> Principal Engineer:** - Deep in the trenches with hyper-scale systems (250,000+ devices) and cutting edge tech stacks. - Developed core technical skills and practiced "DevOps" before it was a formal discipline.

### The Evolution: Leadership - **VP of Engineering & CTO:** - Led teams and organizations of up to 150 people. - Shifted focus to strategy, architecture, and building high-performing engineering cultures. - Good people make great teams, and great teams make award winning products.

### The Architect: Current Role - **Infrastructure Architect:** - Responsible for platform stability and scalability during intense, accelerated growth. - Currently leading the platform teams in tackling complex cross team challenges and architecting future-proof solutions.

## Career Highlights

### Infrastructure Architect/Director (dutchie) - Built the platform team from the ground up, and rebuilt the operations teams - Rebuilt the platform and migrated from Azure to AWS providing the first back to back holiday event without any outages or degredations - Currently processing similar volumes of traffic as DoorDash/Instacart etc with a fraction of the resources

### Principal Engineer (Walmart Labs) - Architected the only retail ecomm experience that was able to survive the PS5 launch event without any outages - Deployed 5k+ clusters, with GPUs and live video inference workloads that processed video frames in under 11ms

### CTO (iTel Networks) - Built the support team, NOC, and implementation teams - Grew the company from 10 to 100+ employees - Ran a mentorship program for highschool students that had a 100% long term career path rate - Learned I didn't want to be a CTO, or live in Canada

### DevOps & OpenStack Pioneer (Rackspace) - Deployed the first non-NASA OpenStack production cloud in 2010 - Multiple patents related to high density data center cooling - Did the first in air takeover of a consumer grade (Parrot) drone for Austin Drone Games - Facilitated the college hiring pipeline that lead to many long term hires

## Core Motivation - **Solving for Scale:** Designing and operating complex, large-scale heterogeneous environments in geographically distributed environments A large system elegantly orchestrated is like binary ballet, I absolutely love it

## Core Expertise - **Technical Leadership:** Leading teams (4 to 150) to implement new tech and re-architect systems and processes. - **Innovation:** History of contributing to cutting-edge tech (patents, early cloud work). - **Hybrid & Multi-Cloud:** Experience with Azure, AWS, GCP, and extensive on-prem

## Why I'm Excited About IonQ My career has been about building foundational infrastructure for or by leveraging transformative technologies. IonQ is both of those at the same time, a potential industry leap that will actually change what humanity is capable of.

### A Familiar Challenge, with a twist - You're building infrastructure scaling systems and processes for a new computing paradigm, which requires a leader comfortable with ambiguity who can work along side a team of experts as a player coach. - **I have done this before.** _Just not with networks with lasers_.

### A Mission I Believe In - I want to apply my multiple decades of experience to a mission that matters and is technically challenging. - Building the platform for the future of quantum computing is a **once-in-a-era opportunity**.

## Project "One Goal": Overview & Purpose **Overview:** A strategic project to improve stability, reliability, and scalability for e-commerce & POS.

### Purpose - **Enhance Stability:** Minimize blast radius and prevent cascading failures. - **Rapid Incident Response:** Isolate issues for faster recovery. - **Seamless Scaling:** Support horizontal scaling for increased volume and rapid growth. - **Enable CI/CD:** Facilitate safe, continuous deployment to increase feature and fix velocity. - **Improve Customer Trust:** Maintain high availability and performance.

## The Challenge: Three companies, three monoliths, three cultures Dutchie was a combination of three companies, all fast growth startups with unique stacks, tons of tech debt, and zero standardization.

### Technical Problems - **System Instability:** Multiple single points of failure (SPOFs). - **Slow Incident Response:** Difficult to identify or contain incidents. - **Limited Scalability:** Monolithic design hindered horizontal scaling. - **Risky Deployments:** Weekly releases for a complex monolith are high-risk events. - **Database Bottlenecks:** Legacy databases (MongoDB, MS SQL) designed for a different scale and access patterns.

### Cultural Problems - **Factions:** Each company thought the other two had trash code and tech. - **Different Problems:** Every team had slightly different sources for their stability issues. - **Startup Mindset:** Teams were focused on building features and getting product to market as fast as possible with no regard for safety.

## The Technical Solution: A Cell-Based Architecture (CBA) Divide the platform and application into multiple isolated **cells**, each an independent unit with its own resources and data domain.

## Cell Boundaries - **Resource Isolation:** Each cell has its own dedicated deployment, databases, and supporting infrastructure. - **Independent Operation:** Cells handle traffic and transactions without talking to other cells. - **Data Isolation:** Each cell has its own data domain completely covering a customer, and data is not shared between cells. - **Hub and Spoke:** Limited global services handle data that needs to cross cell boundaries.

## Cell Requirements - **Standardized Communication:** - **External:** REST APIs with OpenAPI contracts for public-facing interactions. - **Internal:** gRPC for efficient, low-latency inter-service communication and application contracts. - **Testability:** Published mocks and test data are included with each service, enabling reliable independent testing by consumers.

## Trading Complexity We were choosing to trade architectural complexity for operational complexity. - When orchestration complexity fails it is annoying. - When architectural complexity fails it is catastrophic.

## The Cultural Solution: A Goal - "One Goal" A singular bite size directive applicable to all teams and all technical decisions that forces critical thinking and alignment.

## All systems and software should be designed or redesigned in such a way that the blast radius of any one thing is only 1% of our customers, or 1% of our transaction volume, whichever is smaller.

## Supporting Content Resistance is usually due to a lack of clarity, so we needed to provide a lot of context and support. - **Presentations:** Explaining "Skyscraper to suburbs" architecture changes in accesible ways. - **POC:** POC CBA used to educate teams with areference implementation. - **Workshops:** Regular sessions to answer questions and provide support. - **Reinforcement:** Celebrating small wins to build momentum and a feeling of progress.

## Trade Offs 1% of volume is possible but it isn't always practical. What this really does is force conversations about trade offs and constraints. It makes you think harder about what is important and what is not. The real value was simply getting everyone talking about the same thing with the same language.

## Going Further: A Vision - "DOPE - Dutchie One Platform Experience" Once everyone is pulling in the same direction you can start to shape that path and stretch the boundaries of what is accepted as within reach.

## Stretch Goals become just Goals - **Serverless:** Transition all database and caching to AWS native serverless offerings. - **CI/CD:** Move to fully automated CI/CD pipelines with automated testing, deployment, and rollback. - **Noise Floor Zero:** All errors result in team pages, the acceptable noise floor is zero. - **Macroservices:** No new services in the monoliths, only new macroservices.

## Supporting Content Same stuff, different scope. - **Presentations:** Explaining individual services, approaches, and advantages - **Templates:** POC with templates for common infrastructure components and service bootstraps. - **Workshops:** Regular sessions to answer questions and provide support. - **Reinforcement:** Celebrating small wins early on to build momentum and a feeling of progress.

## Details: A Strategy - "Crazy Ivan" Two teams, two projects, going opposite directions, one result. Unification.

## Specific discrete decisions - **Replatform:** Ecomm uses POS, POS goes cloud native. - **Source of Truth:** All data is deduplicated and given a single source of truth. - **Services Data Hide:** Services hide their data behind a common interface, tightly coupled but contracted. - **Engine Swaps:** Individual teams with contracts swap to cloud native engines without affecting consumers.

## Supporting Content Same stuff, different scope. - **Presentations:** Explaining specific team directions and goals. - **Templates:** POC with templates for mocks, deployments, and communication. - **Workshops:** Regular sessions to answer questions and provide support. - **Reinforcement:** Celebrating small wins early on to build momentum and a feeling of progress.

## My Role & Leadership As the architect and leader it was my job to provide the vision, strategy, and support to the teams. But more critically get them to buy into the plan and take ownership of their domains.

### Technical Strategy & Execution - **Lead Technical Strategy:** Drove the cell-based architectural design. - **Guide Technical Leadership:** Provided the vision and strategy for the teams to follow working with their leads to ideate, own, and implement. - **Relentless Repetition:** Saying the same thing, in different ways, at every opportunity, for every audience.

### Team & Stakeholder Management - **Manage Cross-Team Collaboration:** Coordinated efforts across multiple charters. - **Managed Platform Teams:** Worked with the platform managers to define and implement the platform strategy. - **Communicate with Stakeholders:** Regularly reported progress and status up and down the chain. - **Grow Talent:** Identified safe opportunities for less senior engineers to take on more responsibility.

## Outcomes: Quantitative Results

## System Stability & Reliability - **90% reduction** in incidents causing significant customer impact. - Mean Time To Recovery (MTTR) **improved by 75%**, from 2 hours to 30 minutes. - Met and maintained **99.99% availability** for all cell-based services.

## Performance & Scalability - Achieved **400% increase** in transaction processing throughput with no added latency. - Cell architecture allows for **linear, horizontal scaling**, removing previous capacity limits.

## Development Velocity - **Increased deployment frequency by 8x**, from one weekly release to multiple daily deployments. - **Reduced change failure rate by 60%** due to smaller, automated, and isolated deployments.

## Outcomes: Qualitative Results

## Improved Engineering Culture - **Increased Autonomy:** Teams own their services end-to-end, leading to higher engagement. - **Proactive Mindset:** With a stable system and guardrails, teams focus on innovation and feature development. - **Blameless & Collaborative:** The "Test. Trust. Ship." motto and blameless post-mortems foster safety and a focus on collective improvement.

## Enhanced Business Agility & Confidence - **Stakeholder Trust:** Deploying features safely and quickly increased the business's confidence in engineering. - **Reduced Risk:** Product and business teams no longer feared deployments, viewing them as routine. - **Customer Satisfaction:** Higher reliability and performance led to improved customer sentiment higher sales.

## Lessons Learned

## The Importance of a balancing clarity and being prescriptive - **Lesson:** At first we were too broad and people didn't know what *they* were supposed to do. Then we were too prescriptive and people didn't know *why* they were doing it. - **Insight:** You have to constantly reinforce the why, have an outline of the what, and then help teams creat their own how.

## Celebrate Incremental Progress - **Lesson:** A multi-year timeline can feel daunting. We weren't good at celebrating small wins early on, which impacted morale and drive. - **Insight:** Celebrating incremental victories was crucial for building momentum and making the long-term vision feel achievable and worthwhile.

## Diverse Talent, Diverse Communication - **Lesson:** Dutchie has the largest divide in range of talent in a single title in a single company I have ever seen. - **Insight:** I had to repackage the same information for different audiences even in the same title and team, and, I had to do it often.

## How This Experience Aligns with IonQ

### Building Scalable, Mission-Critical Infrastructure - The "One Goal" project was about building a resilient foundation for the entire business in a growth phase but it was built on culture changes and a shared vision, through teams. I have direct experience helping platform teams get platform adoption and buy in.

### Leading Through Technical and Cultural Transformation - I led not just technical migrations but a cultural shift towards automation, ownership, and high-velocity development. - As IonQ scales, it will face similar challenges. I have the experience to guide this evolution and build a world-class engineering culture.

### Strategic, Long-Term Vision - I have a proven ability to manage complex, multi-year projects requiring deep technical expertise and sustained stakeholder alignment. - I am excited to bring this strategic leadership to IonQ to build the future of quantum computing infrastructure.

## Q&A **Thank you for your time.** I'm happy to answer any questions you may have. Also, you can ask [ChatGPT](https://chatgpt.com/g/g-68823abec168819192d042175e450bf9-preso), I created an agent loaded with this presentation and a bunch of context about myself.