Behavioral Questions

Situation: In our firm, we have well defined technical path and management path. However, there is no clear pathway for a techical engineer to become manager or choose management path. It is a catch 22 situation where one has to have people mgmt experience to become manager but one gets the experience only after becoming manager.

Opportunity: My team was growing very fast and within 5 years, we grew from 2- 3 people to over 50 team members. Hiring managers, training them, ensuring they align to our own culture was very time consuming process. I floated an idea with my superiors about why cant we come up with a process to train technical engineers who are interested in management path? My leadership was very supportive and they said they are very open to listen to my idea.

Action: As first step, I reached out to HR and collected the job roles/responsibilities and skills required for management and techncial career path. I also identified the kind of coachign and internal training sessions organized for managers and found equivalent trainings outside. We had many subscriptions such as Harward ManagerMentor and other technical libraries.

I created a new role called FPM (Future Project Manager), defined the roles and responsibilties requird for first line manager, identified trainings, timeline etc. and socialized with my leadership. They were very enthusiastic and even joked that I have come up with a structured approach for my succession planning. I then ran the idea with HR along with my leadership, and they OKayed this idea.

Result: I made a presentation on this to my team and invited their feedback. Some where skeptical because this is not formal role and feared what happens after an year or so. There were a couple of engineers very keen to explore this further. I made them poitn of contact to lead projects, point of contact for communciation and gave them ownership and responsibilties. At the end of 6 months, we could fast track them to managers and the idea got larger support across other teams. This got visibility all the way to senior leadership and the HR dept took over this framework and expanded the concept. I must confess that I could not have pulled this by myself. I received excellent support from my leadership and HR.

Example 2

Situation: The team I was managing was more into CPE or something called sustaining engineering. The main focus was to fix bugs and address customer issues and limited involvment in new product design. In all hands and other company wide activities, the product design teams used to garner all the limelight and my team developed a perception that they are second citizens.

Opportunity: There was serious motivational issue within the team. I wanted to ensure my team is proud of what they are doing, they are making huge difference and lack of visibility is no way limiting the importance of their work. I knew this well. I believed this. I wanted my team to know the same.

Action: I started socializing with product management teams and engineering management and asked for a slot in their monthly meetings/weekly team meetings. I brought my team members to this meeting and used it as a platform to explain customer problems and scenarios. It was interesting to know that many of the engineering teams had very limited visibility to real customer problems and pains.

Result: The customer scenarios, troubleshotting pains, simiplicity and usability issues and how the team handled these while facing wrath of angry customers made great story. There was excellent support and renewed respect for our team and we started getting lots of visibiltiy in leadership meetings. More importantly, team morale went up and they starte believing in their work and its impact. It was a very satisfying end result for me.

Example 3

Situation: Just a few months ago, while at Mphasis working on the "Bedrock Knowledge Base with Serverless Vector Search Implementation" project, we faced a perplexing challenge. We were aiming to build a highly scalable and cost-effective knowledge base search solution using Amazon Bedrock and OpenSearch Serverless. The initial architecture involved directly indexing the embeddings generated by the Bedrock Titan model into OpenSearch Serverless. However, we quickly realized that the indexing process was becoming a significant bottleneck and a major cost driver. OpenSearch Serverless, while scalable, was proving to be surprisingly expensive for frequent indexing operations, particularly with the high dimensionality of the Titan embeddings. Standard approaches like optimizing the indexing batch size or tuning OpenSearch Serverless configurations weren’t yielding satisfactory results. The project timeline and budget were at stake.

That’s when I started thinking "outside the box". I realized we were treating the problem solely as an OpenSearch indexing challenge. Instead, I started exploring alternative ways to reduce the frequency and volume of data being indexed.

Action: Here’s what I did:

• 1. Content Deduplication and Summarization: I analyzed the source documents and discovered a significant amount of redundancy and irrelevant content. Many documents contained overlapping information or boilerplate text. I implemented a pre-processing pipeline using NLP techniques to:
  • Deduplicate near-duplicate documents.
  • Summarize long documents to extract the key information. This

  significantly reduced the overall volume of data being indexed.

• 2. Selective Embedding Generation: Instead of generating embeddings for the entire document corpus, I explored generating embeddings only for the most relevant sentences or passages within each document. This required implementing a sentence scoring mechanism to identify the sentences that were most informative and representative of the document’s content. For example, we would give more importance to action verbs or key nouns.
• 3. Embedding Caching: For frequently accessed documents, I implemented an embedding caching layer using Amazon ElastiCache. This allowed us to avoid regenerating embeddings for documents that had already been processed.

Result: The result was dramatic. By combining these three techniques, we were able to reduce the indexing volume by over 70%, which translated into a significant reduction in OpenSearch Serverless costs. The indexing throughput also improved considerably, allowing us to meet the project timeline. Even better, the recall did not decrease as the right data was chosen to be used to create embeddings. We met all project milestones.

I was extremely happy with this outcome. It demonstrated the power of challenging conventional assumptions and thinking creatively to solve complex problems. It also highlighted the importance of understanding the underlying data and identifying opportunities to optimize the entire workflow, not just individual components.

Situation: During the implementation of a new data pipeline for a fraud detection model at NetApp, I was working closely with a senior data engineer. We had different ideas about how to handle real-time feature generation. The coworker, let’s call him Mark, proposed using a complex, micro-batching approach with Apache Spark Streaming to process incoming transactions. This was his area of expertise, and he was confident it would provide the necessary throughput and low latency.\n\nHowever, I believed that a simpler approach using Kafka Streams would be more efficient and maintainable for our needs. My concerns with Mark’s proposal were:

• 1. Complexity: Spark Streaming, while powerful, introduced significant complexity to the pipeline, increasing the risk of bugs and making it harder to debug and maintain in the long run.
• 2. Overkill: I felt that Spark Streaming was overkill for our specific requirements. Kafka Streams, being a lightweight stream processing library, would be sufficient for our expected transaction volume and latency targets.
• 3. Operational Overhead: Spark Streaming required a dedicated Spark cluster, adding to the operational overhead and cost of the infrastructure. Here’s how I expressed my opposition constructively:
  • 1. Active Listening and Understanding: I started by actively listening to Mark’s explanation of his proposal and trying to understand his rationale. I asked clarifying questions to ensure I fully grasped his approach and the benefits he envisioned.
  • 2. Respectful and Data-Driven Opposition: I didn’t dismiss his idea outright. Instead, I presented my concerns in a respectful and data-driven manner. I prepared a short presentation outlining the pros and cons of each approach, including: Performance benchmarks from similar use cases. Estimated cost and operational overhead for each approach.\n * A comparison of the complexity and maintainability of each approach.
  • 3. Focus on Project Goals: I emphasized that our primary goal was to build a reliable and efficient data pipeline for the fraud detection model, and we should choose the approach that best achieved that goal. I reframed the discussion from a personal preference to what solution best benefited the organization.
  • 4. Collaborative Problem Solving: I proposed that we conduct a small-scale A/B test to compare the performance of the two approaches using real-world transaction data. This would provide concrete evidence to support our decision. What happened: Mark was initially resistant to my concerns, as he was confident in his expertise with Spark Streaming. However, after reviewing my presentation and discussing the pros and cons of each approach, he agreed to conduct the A/B test.

  Result: The results of the test showed that Kafka Streams achieved comparable performance to Spark Streaming with significantly lower latency and resource consumption. The A/B test was not an adversarial process, but rather an investigation of the facts. \n\nBased on the test results, we collectively decided to proceed with Kafka Streams for the real-time feature generation pipeline. Mark, despite his initial preference, supported the decision and actively contributed to the successful implementation of the pipeline. This situation reinforced the importance of respectful communication, data-driven decision-making, and focusing on project goals when collaborating with coworkers. It also taught me that even when you strongly disagree with someone’s idea, it’s important to listen to their perspective and be willing to compromise if the evidence supports it.

Situation: If my manager provided negative feedback on my approach to a problem, my immediate reaction would be to listen attentively and resist the urge to become defensive. I view feedback as a valuable opportunity for growth and improvement, especially from someone with more experience and a broader perspective within the organization. Here’s a breakdown of my likely actions:

• 1. Active Listening and Clarification: I’d focus on fully understanding the feedback. I’d ask clarifying questions like:
  • “Can you give me specific examples of what I could have done differently?”
  • “What were the key concerns with the approach I took?”
  • “What alternative approaches would you have recommended?” It’s crucial to understand why my manager felt my approach was suboptimal, rather than just what they disliked.
• 2. Acknowledgement and Appreciation: I would explicitly thank my manager for providing the feedback. Acknowledging their effort and demonstrating my openness to improvement creates a positive environment for future conversations.
• 3. Self-Reflection and Analysis: After the conversation, I’d take time to reflect on the feedback. I’d analyze my approach to the problem, considering the points my manager raised. I’d ask myself:
  • “Did I miss any crucial information or context?”
  • “Were there biases or assumptions that influenced my decisions?”
  • “Could I have consulted with others or sought out different perspectives?”
• 4. Seek Further Input (If Needed): Depending on the nature of the feedback, I might seek input from other colleagues or mentors to gain additional perspectives on the situation. This helps to ensure a well-rounded understanding and prevents reliance solely on my own or my manager’s viewpoint.
• 5. Develop an Action Plan: Based on my reflection and any additional input, I’d develop a concrete action plan for how I would approach similar problems in the future. This might involve:
  • Researching alternative problem-solving techniques.
  • Improving my communication or collaboration skills.
  • Seeking out training or mentorship in specific areas.
  • Building a checklist of items for each problem.
• 6. Follow-Up and Implementation: I’d schedule a follow-up meeting with my manager to discuss my action plan. This demonstrates my commitment to improvement and allows them to provide further guidance and support. I would then actively implement the action plan and monitor my progress.
• 7. Seek Future Feedback: I’d proactively seek feedback from my manager on my subsequent projects to ensure I’m consistently improving and applying the lessons learned. It’s not a one-time event but more a consistent application of feedback.

  In essence, my approach to negative feedback is to view it as a constructive learning opportunity, actively seek to understand the rationale behind it, and develop a concrete plan to improve my skills and approach in the future.

Situation: At CBRE/Google, while optimizing the Kubeflow Pipelines deployment process for a set of complex, multi-stage machine learning models on Vertex AI, we were facing a recurring issue of slow pipeline execution times. The standard approach would have been to profile the pipeline steps, optimize individual components, and scale up resources, and we tried that initially. However, the profiling revealed that a significant bottleneck was the metadata synchronization across different pipeline components. Vertex AI Pipelines relies on a metadata store (backed by a database) to track artifacts and dependencies between pipeline steps. In our case, due to the high volume of data and complex dependencies, the metadata synchronization was becoming a major bottleneck. The “usual” solution of scaling up the metadata database instance was not viable due to cost constraints and potential scalability limitations of the underlying database architecture.

Action: The “unusual” solution was to use an eventually consistent caching mechanism. Instead of directly scaling the Vertex AI Metadata Service, which was cost-prohibitive and had limitations, I explored a novel approach: introducing an in-memory caching layer directly within the Kubeflow Pipeline itself. This caching layer would sit in front of the metadata store and serve as a fast, temporary storage for frequently accessed metadata. Here’s the unique approach:

• 1. Leveraged Pipeline Component Context: Kubeflow Pipelines provides a component context that allows sharing data and state between components. I used this context to create a distributed in-memory cache using Redis deployed as a separate microservice within the pipeline execution environment. Redis had to be installed to the image, and it was added as an initContainer to avoid network congestion. It also automatically scaled depending on load.
• 2. Implemented a Custom Metadata Cache Client: I developed a custom Python client that intercepted metadata read requests from the pipeline components. This client first checked if the requested metadata was present in the Redis cache. If so, it retrieved the metadata from the cache, bypassing the need to access the metadata store. If the metadata was not in the cache, the client retrieved it from the metadata store, stored it in the cache, and then returned it to the component.
• 3. Cache Invalidation Strategy: I implemented a cache invalidation strategy to ensure data consistency. When a pipeline component modified the metadata (e.g., by creating a new artifact), it would invalidate the corresponding entry in the Redis cache. This prevented stale metadata from being served from the cache. The pipeline code was changed so that if a particular object’s metadata changed, it would invalidate the cache. This required making changes to different portions of the pipeline.

Result: The outcome was significant: We achieved a 30-40% reduction in overall pipeline execution time, especially for pipelines with a high degree of data sharing and complex dependencies. This also resulted in substantial cost savings due to reduced resource consumption. More importantly, the solution was scalable without relying on scaling up the underlying Vertex AI infrastructure, proving it was significantly more efficient. The outcome, as a function of time and cost, was much lower than it would have been otherwise. \n\nYes, I was very satisfied with this solution. It was a challenging problem that required thinking outside the box and leveraging my deep understanding of Kubeflow Pipelines, Vertex AI, and distributed caching. It also demonstrated the value of optimizing not just individual components but also the underlying infrastructure that supports the pipeline execution. The approach itself proved transferable: we subsequently applied this caching strategy to other data-intensive pipelines, achieving similar performance gains.

At Google, while working on Kubeflow Pipelines for a large language model (LLM) serving application, I encountered a situation where the team was pushing for aggressive cost optimization by severely limiting the resources allocated to model monitoring and explainability components.

The rationale was that these components were adding significant latency and cost to the inference pipeline, and reducing their footprint would dramatically improve the overall cost-efficiency, a key metric for our OKRs. This initiative went against my deep-seated belief in responsible AI and the importance of model transparency, especially for high-stakes applications like LLMs.

I felt that drastically reducing monitoring would essentially fly blind, making us unable to reliably detect and address issues like data drift, bias amplification, or unexpected failure modes in production, potentially harming users. At an IC6 level, I recognized the strategic implications: short-term cost savings were potentially jeopardizing the long-term reliability and ethical considerations of our AI system.

Here’s how I addressed the situation, demonstrating influence and technical depth:

• Developed a Quantitative Model: I didn’t just express concerns; I built a detailed quantitative model to demonstrate the trade-offs. This model incorporated:
  • The cost of running different levels of monitoring and explainability, accounting for CPU, memory, and network bandwidth usage. I had to profile several model sizes for the latency impact.
  • The estimated cost of potentialmodel failures, including financial losses from service disruptions, legal liabilities from biased outcomes, and reputational damage (using industry benchmarks and internal risk assessments).
  • I used Bayesian methods to quantify the probability of different failure modes occurring given various levels of monitoring coverage. This model accounted for uncertainty and allowed for scenario analysis.
• Proposed Alternative Monitoring Strategies: I researched and proposed alternative, more efficient monitoring strategies that would minimize the overhead while still providing adequate coverage. This included:
  • Adaptive Monitoring: Implementing dynamic adjustments to the monitoring frequency and granularity based on real-time model performance and risk levels. For example, increasing monitoring intensity during periods of high data drift or model degradation.
  • Selective Explainability: Focusing explainability efforts on high-risk predictions or segments of the user population where bias was more likely to occur. This involved using techniques like Shapley values or LIME to generate explanations for only a subset of predictions.
  • Sampling Techniques: The model used importance sampling so that the underrepresented slices were monitored.
• Influence Through Data and Collaboration: I presented my model and alternative strategies to the engineering lead, product manager, and key stakeholders, emphasizing the long-term business implications of inadequate monitoring. I didn’t frame it as a technical disagreement but as a risk management issue.
• Championed MLOps Best Practices: Used the opportunity to advocate for the best practices in model deployment and MLOPs

Result: The outcome was that we reached a compromise. We implemented a more efficient monitoring strategy based on my recommendations, including adaptive monitoring and selective explainability. While we did reduce the overall cost of monitoring, we didn’t sacrifice essential coverage. The data-driven approach and the focus on risk management helped to shift the conversation from a purely cost-centric perspective to a more holistic view that considered the ethical and business implications of our AI systems. The company adopted importance sampling as a technique, reducing costs and providing a new strategy for the model. This experience reinforced the importance of advocating for responsible AI and using data-driven arguments to influence decision-making."

Situation: During my time at Google, I was tasked with leading the deployment of a critical performance improvement to the LLM serving infrastructure for a key product (I can’t name specifics due to NDA). The performance improvement involved integrating a new model caching mechanism. Thisinitiative was crucial to meet upcoming peak traffic demands during a major product launch. The deadline was extremely tight – we had just three weeks to design, implement, test, and deploy the solution. A delay would directly impact user experience and potentially derail the product launch.

Here’s how I managed my time and resources, focusing on what I would bring as an IC6:

• Rapid Prioritization and Scope Definition: I immediately convened a meeting with the key stakeholders (engineering lead, product manager, operations team) to clearly define the scope of the project and prioritize the most critical features. We used the MoSCoW prioritization method (Must have, Should have, Could have, Won’t have) to ruthlessly cut any non-essential tasks. We clearly defined what was absolutely essential for the product launch and deferred everything else.

• Strategic Task Delegation: I carefully assessed the skills and experience of each team member and delegated tasks accordingly. I assigned the most experienced engineers to the most critical and complex components of the caching mechanism. Clear and measureable success metrics were defined for each member.

• Parallel Development and Testing: To accelerate the development process, we adopted a parallel development and testing approach. We broke down the project into smaller, independent modules that could be developed and tested concurrently. I setup two teams, each with clear areas of ownership. Frequent code reviews and integration tests were conducted to ensure that the modules worked seamlessly together.

• Aggressive Risk Management: I proactively identified and assessed potential risks, such as unforeseen technical challenges or dependencies on other teams. We developed contingency plans to mitigate these risks, including:

  • Having backup architectures for critical components
  • Using automated testing so that tests were easily repeatable
  • Setting up dashboards to detect degradation.

• Continuous Monitoring and Communication: I established a clear communication plan to keep all stakeholders informed of the project’s progress. We held daily stand-up meetings to track progress, identify roadblocks, and make necessary adjustments. I also created a real-time dashboard to monitor key performance indicators (KPIs), such as code commit frequency, test pass rates, and bug resolution times. I also set up an asynchronous communication channel, and it was important everyone used it.

• Automation and Tooling: I leveraged automation and tooling wherever possible to streamline the development and deployment process. This included:\n * Automated build and deployment pipelines using Jenkins and Spinnaker.\n * Automated testing frameworks for unit, integration, and end-to-end testing.\n * Infrastructure-as-code tools like Terraform to provision and manage cloud resources.

• Ruthless time management: I used timeboxing techniques and the Pomodoro method to stay extremely focused to manage my day. The meetings were time boxed, with specific agendas and attendees had to come prepared with information.

  Despite the tight deadline and various technical challenges, we successfully deployed the caching mechanism on time and within budget. The deployment resulted in a significant improvement in the LLM serving performance, allowing us to handle the peak traffic demands during the product launch without any major issues. As a leader, this success highlighted my ability to effectively manage time and resources under pressure, prioritize tasks, delegate responsibilities, and communicate effectively with stakeholders. As an IC6 I was able to use my high-level understanding to make sure all the pieces were in place.

Situation:

At Mphasis, during the design phase of a new MLOps platform for a major financial institution, we were faced with a critical architectural decision: whether to build our own custom model registry from scratch, or to leverage a pre-existing managed model registry service offered by AWS (SageMaker Model Registry) or Azure (Azure Machine Learning Model Registry). Building a custom solution would give us maximum flexibility and control over the data model and security features. However, it would also require significant development effort and ongoing maintenance. Using a managed service would be faster and easier to implement, but it would limit our customization options and potentially expose us to vendor lock-in.\n\nThe difficulty arose because we lacked complete information about several key factors:\n\n1. Long-Term Requirements: The client’s long-term requirements for model governance, lineage tracking, and security were still evolving. We had some initial specifications, but there was a high degree of uncertainty about how those requirements would change over time.\n\n2. Future Model Types: We didn’t know what types of ML models the client would be using in the future. The current focus was on tabular data and classical ML algorithms, but there was a possibility that they would eventually adopt deep learning models or other more complex model architectures.\n\n3. Integration with Existing Systems: We had limited information about the client’s existing IT infrastructure and how the model registry would need to integrate with those systems.\n\n4. The technical team had different viewpoints. Some thought building it from scratch was more elegant, while others advocated managed solution for speed. \n\nFaced with this uncertainty, I made the difficult decision to proceed with a hybrid approach. Here’s how I reasoned through it:\n\n1. Phased Implementation: I proposed a phased implementation strategy. In the first phase, we would leverage a managed model registry service (SageMaker Model Registry) for the core functionality, such as model versioning, metadata storage, and access control. This would allow us to quickly deliver a working solution and get early feedback from the client. It solved 80% of the requirements.\n\n2. Customizable Extension Points: We designed the platform with well-defined extension points that would allow us to add custom features and integrations in the future. This would give us the flexibility to adapt to changing requirements without having to completely re-architect the platform.\n\n3. Open Source Commitment: Select some popular open-source frameworks to support the remaining 20%. The client was given the code and could build on it.\n\n4. Stakeholder Management: The hybrid proposal met everyone’s goals. The technical team would build on their strengths, while the end solution would meet all the key requirements.\n\nThis hybrid approach allowed us to balance the need for speed and flexibility. We were able to deliver a working solution quickly while also preserving the option to build more custom functionality in the future, as our understanding of the client’s requirements evolved. It was a difficult decision, but it ultimately proved to be the right one. The client was happy with the initial implementation, and we were able to adapt the platform to their changing needs over time. The initial data science team agreed to use it to publish their models.

Situation:

At Mphasis, I was leading the design of a new MLOps platform for a major financial institution. A critical decision point was choosing a model registry – build a custom one or use a managed service (SageMaker/Azure). We lacked complete information about the client’s long-term requirements for governance, future model types they’d use, and integration with their existing IT systems. The technical team also had strong, divergent opinions.\n\n**Task:** My task was to recommend an approach for model registry implementation that balanced the need for a functional platform within a reasonable timeline, while also accounting for future scalability, flexibility and cost. A key challenge was dealing with the uncertainties.

Action:

Given the incomplete information, I decided to proceed with a hybrid approach. My decision-making process involved:\n\n* Phased Implementation: To address uncertainty about the client’s evolving needs, I proposed a phased implementation. In the first phase, we would leverage a managed model registry service (SageMaker Model Registry) for core functionality like versioning, metadata, and access control. This gave us speed of delivery and valuable early client feedback.\n* Designed Customizable Extension Points: To ensure future flexibility, we designed the platform with well-defined extension points allowing us to add custom features and integrations later. This allowed adapting to changing needs without a complete re-architecture.\n* Selected Open Source Libraries: We also selected open-source frameworks to support the remaining 20%, providing the client with the code and the ability to extend as needed.\n* Stakeholder Management: To address divergent views within the team, I had discussions with members to come to an agreement on the best solution for all.\n\n**Result:** This hybrid approach balanced speed and flexibility. We delivered a working solution quickly, got client feedback and retained options for custom functionality as their needs evolved. The client was happy with the initial implementation and our ability to adapt over time. The data science team agreed to use the proposed solution to publish their models.

Situation: At Google, during the rollout of a new LLM-powered feature for Search, we encountered high latency in production. While my role was MLOps for the LLM and not direct model performance, this issue jeopardized the feature launch and revenue targets. Different teams investigated separately, without central coordination.\n\n**Task:** Although outside my core MLOps responsibility, I felt ownership because the infrastructure I helped build was supposed to enable efficient model serving. My task was to resolve the performance bottleneck, get the feature back on track for launch, and prevent future occurrences.\n\n**Action:** I stepped up to coordinate the troubleshooting, exceeding my defined role:\n\n* Volunteered Coordination: I offered to be the central point of contact, connecting the model serving, data retrieval, and networking teams.\n* Developed Unified Dashboard: I created a single-pane-of-glass dashboard aggregating key performance metrics from all components, using existing tooling for speed. This allowed quick detection of bottlenecks and anomalies.\n* Led Data-Driven Analysis: I facilitated troubleshooting sessions, promoting a data-driven approach. I encouraged data sharing and helped connect observations across teams, applying statistical testing to identify anomalies.\n* Identified Bottleneck: This collaboration revealed the bottleneck in data retrieval – the LLM made many small, uncached database requests, causing excessive latency.\n* Proposed & Implemented Solution: I worked with the data retrieval team to batch requests and improve caching. We deployed this to production.\n\n**Result:** The new feature performance increased significantly and met the launch and revenue targets. Latency was reduced by over 50%. By stepping outside my role, I ensured the feature’s success. I also documented the debugging process to use this in the future.

Situation: As the tech lead for a critical model migration at NetApp, our team was tasked with migrating a large-scale fraud detection model from an on-premise infrastructure to AWS SageMaker. We planned to leverage containerization and AWS native services (SageMaker, EKS) to improve scalability and reduce operational overhead. The initial plan was to perform a blue-green deployment, ensuring zero downtime and minimal disruption to the existing fraud detection system. After weeks of preparation and testing in a staging environment, we initiated the cutover during a scheduled maintenance window. Soon after cutting over, the new AWS-hosted system experienced a significant performance degradation - transaction processing latency increased by over 500ms compared to the old infrastructure, effectively crippling the fraud detection capabilities. We also got notified for data drifting impacting the false positive rate. \n\n**Task:** My primary task was to lead the team in quickly diagnosing and mitigating the performance degradation and any errors in the process. Reverting to the legacy on-premise system was an option, but this action could potentially result in substantial economic loss, and in a longer period affect the business’s credibility and reliability. Besides the time pressure, the analysis was also complicated due to the number of factors in the new system that had the potential to cause such a big error - networking, CPU throttling, different library versions etc.\n\n**Action:** I spearheaded a coordinated and systematic troubleshooting effort, and to effectively address these obstacles:\n\n1. Triaged and Assembled a Cross-Functional Team: I rapidly assembled a virtual war room with representatives from various teams, including data scientists, infrastructure engineers, and cloud architects, to ensure a comprehensive approach.\n\n2. Data-Driven Diagnosis: We established a robust monitoring framework to collect system metrics, performance data, and logs from various components in real-time. I had team members analyze the metrics of the old and new frameworks looking for areas of deviation. Dashboards were created to give us all a full picture. I also had the team double check that the old data had been migrated correctly.\n\n3. Focused Problem Isolation: Following the accumulation of system data, and observing the dashboard in real time, it was clear that most components were performing as expected with one exception: inter-container communication. While we had anticipated some performance reduction from the increased network communication, the observed data points were well over this. While continuing to validate inter-container communication, I engaged with a networking specialist to test the network throughput of the various containers. I realized that the containers for the data were only provisioned with limited bandwidth which explained the communication problems.\n\n4. Controlled rollback strategy: At this point there were two options: increase network throughput to the containers or revert. To avoid unknown unknowns, I decided to revert the containers, while creating a process to create the containers with a much larger network throughput. Because the data analysis was automated, a future deployment could catch a problem and redeploy the older system.\n\n5. Comprehensive post-mortem: After successfully reverting to the previous environment, I assembled the whole team to conduct a thorough post-mortem. Our purpose was to review the mistakes that resulted in the failure, and make an exhaustive plan that will reduce the chances of such happening again. \n\n**Result:** Within 4 hours, we identified the root cause as bandwidth throttling of container. I immediately reverted the deployment to the on-premise system with almost zero data loss, maintaining transaction integrity. Next, I used the findings to set up a detailed and automated pre-deployment system health check, including automatic network bandwidth testing, and capacity to quickly rollback. We successfully migrated the model 2 weeks later. We now have a repeatable and data driven model migration. I am now much more comfortable to deploy at scale because of this process. Even more, the system had a process to redeploy back to the older system, significantly reducing risk during deployments.

Situation: At Google, I was responsible for leading the efforts to enhance model reproducibility across multiple teams working on different aspects of NLP. At the time, each team had its own way of managing experiments, tracking model versions, and storing artifacts. This inconsistency made it incredibly challenging to reproduce results, compare models fairly, and collaborate effectively across the organization. Specifically, onboarding to an existing model required expertise and time to understand how it was built, and different dependencies made sharing between models difficult. Essentially, a lot of complexity was hidden into ad-hoc solutions. There was also a strong desire to use common building blocks and libraries to create consistent solutions.\n\n**Task:** My objective was to simplify the model development process, promote collaboration, and drastically improve the model reproducibility, which at the time was the biggest pain-point. I had to come up with a streamlined, unified process that all teams could adopt while catering to their diverse needs.\n\n**Action:** I took the following actions to simplify the complex process:\n\n1. Cross-Team Workshop and Requirements Gathering: I organized a series of workshops with representatives from all the different teams involved in model development. These workshops aimed to understand the different workflows, identify common pain points, and gather requirements for a unified model development process. To prepare I analyzed the codebase of the top 10 most used models and created a report of common functionalities. I found there was a lot of reuse of these functions, which gave strong data for the next action. \n2. Designed a Standardized Model Development Template: Based on the requirements gathered in the workshops, I designed a standardized model development template that provided a common structure for all model development projects. The template included:\n * A standardized directory structure for storing data, code, and artifacts.\n * A set of pre-defined environment setup scripts for managing dependencies. To achieve this all common libraries had to be containerized.\n * A common API for tracking experiments, managing model versions, and deploying models.\n\n3. Developed a Centralized Experiment Tracking System: I built a centralized experiment tracking system using MLflow that allowed all teams to track their experiments, log metrics, and store artifacts in a consistent manner. This system was integrated with the model development template, making it easy for teams to adopt.\n\n4. Developed shared Model Components: I took the list of top 10 model component functions and created a team to build those components. Then all teams were incentivized to use these existing blocks rather than creating their own. We also set standards so that the models could be plug and play, encouraging code sharing and reusability.\n\n5. Automated the Deployment Process: To simplify the model deployment process, I automated the deployment process using Kubeflow Pipelines. This allowed teams to deploy their models to production with a single command.\n\n6. Provided Training and Support: I provided training and support to all teams to help them adopt the new standardized model development process. This included creating documentation, providing one-on-one coaching, and hosting regular office hours.\n\n**Result:** By implementing these actions, I was able to significantly simplify the model development process across the organization. We achieved the following results:\n\n* Model reproducibility improved drastically. We could now easily reproduce results from previous experiments and compare models fairly.\n* Onboarding time decreased significantly, from about 2 weeks to around 1 hour. \n* Collaboration across teams improved. The standardized template and centralized experiment tracking system made it easier for teams to share code, collaborate on projects, and learn from each other.\n* Model deployment became much faster and easier. Teams could now deploy their models to production with a single command.\nI was extremely satisfied with this outcome. It demonstrated the power of standardization, automation, and collaboration in simplifying complex processes and improving overall efficiency.

Situation: At NetApp, we were facing a growing challenge with our existing automated root cause analysis (RCA) system for storage performance issues. The system relied on traditional machine learning techniques, primarily decision trees and rule-based engines, to analyze system logs and performance metrics. It was becoming increasingly difficult to maintain and scale. This was because the signatures of different failures would evolve over time, requiring ongoing manual analysis and rule tweaking. There was also an explosion of new models and data to be handled, stressing the on-prem infrastructure. The existing RCA system was struggling to keep pace with the growing complexity and volume of data. The system was simply not smart enough to catch novel system errors.\n\n**Task:** My objective was to develop a more scalable, robust, and adaptive RCA system that could automatically detect and diagnose storage performance issues, even those never seen before. This new system also had to be built on AWS to leverage cloud scalability. The new system needed to reduce the human involvement and be fully automated so that it did not lag in case of an outage.\n\n**Action:** I spearheaded the development of a novel RCA system based on a combination of unsupervised and supervised learning techniques, leveraging the power of deep learning. The innovative aspects of the solution were:\n\n1. Unsupervised Anomaly Detection using Autoencoders: We used autoencoders, a type of neural network, to learn the normal patterns of system behavior from historical data. This allowed us to detect anomalies in real-time by comparing the current system state to the learned normal patterns. I made a few architectural modifications to create better performance in the new system.\n\n2. Contrastive Loss for Fault Signature Extraction: The Autoencoder architecture was modified to create a contrastive loss function. The goal was to identify the key factors and features that distinguished between normal and anomalous states. The contrastive loss was minimized so that the key features could be readily accessed in the next stage.\n\n3. Knowledge Graph-Enhanced Diagnostics: Created a knowledge graph of the common systems and then enhanced that graph with the outputs of the supervised and unsupervised learning. Thus, the RCA would not just come up with the root cause of failure, but the next node of failure. The knowledge graph was based on a community project with some proprietary additions.\n\n4. Reinforcement Learning for Adaptive Alerting: Used a reinforcement learning (RL) agent to adaptively adjust the alerting thresholds based on the system’s current state and the historical frequency of false positives. This helped to reduce alert fatigue and improve the accuracy of the alerting system.\n\n5. Explainable AI (XAI) Techniques: Integrated XAI techniques, such as SHAP values, to provide explanations for the RCA results. This increased the transparency and trust in the system, allowing operators to understand why a particular issue was flagged as a potential problem.\n\n**Result:** The new RCA system significantly outperformed the previous system in terms of accuracy, scalability, and adaptability. We achieved the following results:\n\n* The false positive rate was reduced by 60%, significantly reducing alert fatigue.\n* The system could automatically detect and diagnose novel performance issues that the previous system missed.\n* The time to diagnose performance issues was reduced from hours to minutes.\n* The system was highly scalable and could handle the growing volume of data and complexity of the storage infrastructure.\n* The XAI features increased the transparency and trust in the system, making it easier for operators to understand and act on the RCA results.\n\nI was highly satisfied with this outcome. It demonstrated the power of combining different AI techniques to solve complex problems. The most exciting aspect was how these tools can all come together and produce a system far greater than it’s parts. It also highlighted the value of innovation and continuous improvement in MLOps.

Situation: At CBRE/Google, the standard practice for deploying and managing Kubeflow Pipelines on Vertex AI involved manual configuration of each pipeline and its associated resources (compute, memory, accelerators). While this approach was manageable for a small number of pipelines, it became a significant bottleneck as the number of pipelines grew and the team expanded. This was because the manual processes were slow, error-prone, and difficult to scale. Each pipeline deployment was a snowflake requiring substantial hand-holding and specialized knowledge. It was a serious impediment to faster iteration and overall velocity.\n\n**Task:** As a Sr. MLOps Engineer, my task was to identify opportunities to improve the efficiency and scalability of our MLOps practices. Realizing that manual configuration was a significant pain point, I challenged the status quo and proposed a new approach based on Infrastructure-as-Code (IaC) and automated pipeline deployment.\n\n**Action:** I took the following actions to challenge the status quo and implement a more automated and scalable approach:\n\n1. Developed a Terraform-based Infrastructure-as-Code Framework: I created a Terraform-based framework that allowed us to define and manage all of our Kubeflow Pipeline infrastructure resources in code. This included Vertex AI Pipelines, Cloud Storage buckets, service accounts, and network configurations. Terraform allowed us to reuse and extend existing code rather than rewrite from scratch. This framework was designed to be modular and extensible, allowing us to easily add support for new types of resources and configurations.\n2. Created a Standardized Pipeline Definition Format: I designed a standardized pipeline definition format using YAML that allowed us to define all aspects of a pipeline in a declarative manner. This included the pipeline’s inputs, outputs, steps, dependencies, and resource requirements. I made sure the YAML was easy to understand and follow, while supporting extensibility. The YAML would be validated against a standard schema to prevent deployments.\n3. Automated Pipeline Deployment using CI/CD: Integrated the Terraform framework and the standardized pipeline definition format into our CI/CD pipeline using Jenkins and ArgoCD. This automated the process of deploying and managing Kubeflow Pipelines, allowing us to deploy new pipelines or update existing pipelines with a single command. This CI/CD also automated testing of the pipeline, and validated the output before deploying. The CD tool also gave versioning capabilities.\n4. Promoted Adoption and Training: I actively promoted the new IaC and automated deployment approach across the team, providing training, documentation, and support to help others adopt the new practices. This involved hosting workshops, creating tutorials, and providing one-on-one coaching to help team members transition to the new way of working.\n\n**Result:** By challenging the status quo and implementing the IaC and automated deployment framework, we were able to achieve significant improvements in the efficiency and scalability of our MLOps practices. We achieved the following results:\n\n* The time to deploy a new Kubeflow Pipeline was reduced from days to hours.\n* The number of manual errors in pipeline deployments was reduced by 90%.\n* The scalability of our MLOps infrastructure was significantly improved. We could now easily manage hundreds of pipelines with a small team.\n* The consistency and reliability of our pipeline deployments was greatly improved. Pipelines were deployed into environments following a schema validated process.\n\nI was proud to have challenged the existing processes and driven a change that significantly improved our team’s efficiency and effectiveness. It demonstrated the value of constantly questioning the status quo and seeking out opportunities to improve our ways of working.

Situation: At NetApp, I led the team responsible for developing a new generation of predictive maintenance solutions for our storage systems. We had a wealth of telemetry data, and the initial impulse was to leverage the most cutting-edge deep learning techniques, specifically Graph Neural Networks (GNNs), to model the complex relationships between system components and predict failures with unprecedented accuracy. The draw to innovation had to be balanced with practical considerations, which the business had laid out. These included the explainability of the solution, cost of running a new solution, and how quickly we could put a solution to production. It was not acceptable if an elegant solution could not be delivered in 3 months.\n\n**Task:** My responsibility was to guide the team towards a solution that was both innovative and practical, meaning it had to demonstrate a significant improvement over existing methods while being feasible to implement, deploy, and maintain within our resource constraints. I also had to balance business value (e.g., improved accuracy, fewer false positives) with the engineering overhead required for implementing a cutting-edge solution. My goal was to minimize the risk while maintaining the upside of a better solution.\n\n**Action:** I approached this challenge by implementing the following strategies:\n\n1. Proof-of-Concept (POC) with Existing Methodologies: Before diving into GNNs, we first established a solid baseline using existing, well-understood techniques like gradient boosted trees and time series analysis. This allowed us to quantify the potential benefit of more complex approaches. More importantly, it allowed the team to explore the data and get familiar with all aspects of the predictive data.\n\n2. Benchmarking vs. Real World: I requested the team use actual customer data, anonymized to respect privacy. I used metrics that the customers had emphasized, and weighed all factors with those metrics. The team also was given constraints such as not requiring new hardware or software updates.\n\n3. Incremental Innovation with Careful Evaluation: I advocated for an incremental approach to innovation. We started by incorporating GNNs into specific modules of the RCA system, such as anomaly detection, where their strengths were most relevant, and we gradually expanded their use as we gained confidence in their performance and stability. I prioritized smaller changes so that if one of the features did not work, we could more readily remove it.\n\n4. Prioritized Explainability: Given the importance of trust and interpretability in predictive maintenance, I made explainability a key requirement. If the GNN solution was a black box, it would be difficult for engineers and operators to understand the predictions and take appropriate action. We explored techniques like attention mechanisms and feature importance analysis to make the GNNs more transparent.\n\n5. Defined Clear Success Metrics: I worked with the product team to define clear success metrics that balanced innovation with practicality. These metrics included:\n * Improvement in prediction accuracy (precision and recall).\n * Reduction in false positives.\n * Time to diagnose performance issues.\n * Deployment and maintenance cost.\n * Explainability of predictions.\n\n6. Fostered Collaboration: I encouraged close collaboration between the data scientists, engineers, and operations teams throughout the project. This ensured that the innovative solutions were not only technically sound but also practical to deploy and operate in the real world. This involved a daily meeting and feedback.\n\n**Result:** By carefully balancing innovation with practicality, we successfully developed a new predictive maintenance solution that outperformed the existing system while remaining feasible to implement and maintain. The final solution used a hybrid approach. While the GNN was the best technique, we used an ensemble of different methods. We also decided to incorporate SHAP values, which dramatically increased customer trust and use of our solution. By focusing on clear success metrics, explainability, and collaboration, we were able to deliver a solution that was both innovative and valuable to the business. The tool has now become a standard with all of our deployments and has been widely praised by our engineers and customers. As an IC6, it was important to deliver a world class solution, not just one that was innovative.

Situation: At NetApp, I led the team responsible for developing a new generation of predictive maintenance solutions for our storage systems. We had a wealth of telemetry data, and the initial impulse was to leverage the most cutting-edge deep learning techniques, specifically Graph Neural Networks (GNNs), to model the complex relationships between system components and predict failures with unprecedented accuracy. The draw to innovation had to be balanced with practical considerations, which the business had laid out. These included the explainability of the solution, cost of running a new solution, and how quickly we could put a solution to production. It was not acceptable if an elegant solution could not be delivered in 3 months.

Task: My responsibility was to guide the team towards a solution that was both innovative and practical, meaning it had to demonstrate a significant improvement over existing methods while being feasible to implement, deploy, and maintain within our resource constraints. I also had to balance business value (e.g., improved accuracy, fewer false positives) with the engineering overhead required for implementing a cutting-edge solution. My goal was to minimize the risk while maintaining the upside of a better solution.

Action: I approached this challenge by implementing the following strategies:

• 1. Proof-of-Concept (POC) with Existing Methodologies:

Before diving into GNNs, we first established a solid baseline using existing, well-understood techniques like gradient boosted trees and time series analysis. This allowed us to quantify the potential benefit of more complex approaches. More importantly, it allowed the team to explore the data and get familiar with all aspects of the predictive data.

• 2. Benchmarking vs. Real World: I requested the team use actual customer data, anonymized to respect privacy. I

used metrics that the customers had emphasized, and weighed all factors with those metrics. The team also was given constraints such as not requiring new hardware or software updates.

• 3. Incremental Innovation with Careful Evaluation: I advocated for an incremental approach to innovation. We started by incorporating GNNs into specific modules of the RCA system, such as anomaly detection, where their strengths were most relevant, and we gradually expanded their use as we gained confidence in their performance and stability. I prioritized smaller changes so that if one of the features did not work, we could more readily remove it.
• 4. Prioritized Explainability: Given the importance of

trust and interpretability in predictive maintenance, I made explainability a key requirement. If the GNN solution was a black box, it would be difficult for engineers and operators to understand the predictions and take appropriate action. We explored techniques like attention mechanisms and feature importance analysis to make the GNNs more transparent.

• 5. Defined Clear Success Metrics:

I worked with the product team to define clear success metrics that balanced innovation with practicality. These metrics included:\n * Improvement in prediction accuracy (precision and recall).\n * Reduction in false positives.\n * Time to diagnose performance issues.\n * Deployment and maintenance cost.\n * Explainability of predictions.

• 6. Fostered Collaboration: I encouraged close

collaboration between the data scientists, engineers, and operations teams throughout the project. This ensured that the innovative solutions were not only technically sound but also practical to deploy and operate in the real world. This involved a daily meeting and feedback.

Result: By carefully balancing innovation with practicality, we successfully developed a new predictive maintenance solution that outperformed the existing system while remaining feasible to implement and maintain. The final solution used a hybrid approach. While the GNN was the best technique, we used an ensemble of different methods. We also decided to incorporate SHAP values, which dramatically increased customer trust and use of our solution. By focusing on clear success metrics, explainability, and collaboration, we were able to deliver a solution that was both innovative and valuable to the business. The tool has now become a standard with all of our deployments and has been widely praised by our engineers and customers. As an IC6, it was important to deliver a world class solution, not just one that was innovative.