Adventures in DevOps

Share Episode We sit down with veteran Principal Engineer Justin Abrahms to discuss why having a clear engineering scoreboard is critical to winning the software delivery game. Drawing on his experience scaling deployment pipelines for payment engineers, platform engineering teams, and globocorps, Justin explains the simple truth: when you deliberately do the right things—embracing continuous delivery, small batch sizes, and tight feedback loops—you get the right outcomes. Yet despite these battle-tested practices existing for decades, our industry suffers from an "Eternal September" fueled by terrible onramps. We dissect the performative theater of modern engineering, taking aim at the severe lack of intentionality in code reviews. Too many peer reviews devolve into superficial nitpicking rather than educational, inquisitive assessments of long-term maintainability. Justin shares lessons learned from working alongside Trevor Foucher (author of The Art of Readable Code) at Google, demonstrating how…

Share Episode We welcome David Burns, Head of Open Source at BrowserStack and previously Mozilla, this week to help explain how can browser and mobile app testing be so bad still in 2026. "Even if we had to do it all over again, we probably still wouldn't have gotten it right", might just be the biggest learning. We review some of the core challenges that cause even the most experienced web engineers pain and how some companies are attempting to avoid them. We also review shares lessons from Mozilla’s 'War on Orange,' demonstrating why chasing a 100% green CI suite at scale is a fool's errand, and why engineering teams must instead manage delta tolerances and adhere to a strict testing pyramid. Additionally, we investigate the physical hardware nightmares of running 30,000 real phones in data centers; from battery swelling and custom Faraday cages to complying with Apple EULA requirements using racks of Mac Minis. And finally, we discuss WebMCP, Project Fugu, and why LLM browser agents still struggle against…

Share Episode From your feedback, we know that on-call issues, especially those that arise at the database layer are always the most challenging to solve. This episode builds on our Grafana Observability episode as we go deeper into the challenge of data ownership in data lakes. Apache Druid creator, previous Yahoo Distinguished Engineer, Fellow at Splunk, and currently the Chief Architect as Imply, Eric Tschetter joins us to confirm that there are unsolved problems in data platforms, and it doesn't get better when we throw agents into the mix. In the true DevOps mindset aligment, Eric reminds us that whoever is operating the system, knows what is going on, they know their service boundaries, and so it's their choice to make. And the sad reality is that at large enough companies, you will always have independent teams making independent decisions, it's never possible to fully align them, and realistically we get down to why it is even a mistake to try. 💡 Notable Links: ✨ Episode: Grafana — Observability ✨…

Share Episode Sauce Labs CTO Anoop Tripathi addresses the root causes of broken test suites and why teams frequently become numb to intermittent test failures. Anoop shares concrete lessons from his career, detailing how bad practices like embedding fixed sleep routines instead of signal-based waiting ruin test reliability. We discuss the necessity of designing products for testability from day one—such as using pair programming where one developer writes the implementation while the other writes the test suite, or building dedicated test interfaces directly into the application architecture. We also analyze the hidden business costs of poor quality, where engineering debt forces sales teams to spend their time managing upset clients instead of acquiring new revenue. We explore organizational anti-patterns—such as teams claiming to be 'Agile' with one-year sprint cycles, or shipping debug builds to production because release builds continuously crash, highlighting a pair-programming model where one developer…

Share Episode In this episode, we sit down with Nikhil Mungel, Head of AI R&D at Cribl, to discuss how LLMs have dramatically transformed the tech hiring landscape. The conversation opens with an honest look at the flood of AI-generated, low-quality resumes and remote interview cheating tactics, which have pushed many organizations toward in-person whiteboard interviews or closed-network referrals. We explore how software development has evolved from character-by-character coding into "judgment work," where an engineer's value relies on holistic decision-making, taste, and business alignment rather than raw syntax memorization. Nikhil emphasizes the importance of an "ownership mindset," urging developers to move beyond acting as mere specification-translators and instead evaluate technical trade-offs through a business-focused lens. We also pull back the curtain on modern technical interviews, highlighting why expressing strong convictions, asking intentional career questions, and proving human reasoning…

Share Episode Pushkar Gopalakrishna, Senior Staff Software Engineer at Snap, previously Cruise and AWS, joins to explore why engineering organizations pivot away from Terraform and HCL toward Kubernetes-native tools even when they might not be better. We unpack how developer friction, copy-pasted control structures, and misaligned organizational incentives create massive tech debt—often forcing SREs to manually update infrastructure repositories for compliance, resulting in broken pipelines and severe operational friction. We debate over the mechanics of Crossplane, detailing how its continuous reconciliation loop and Custom Resource Definitions (CRDs) allow teams to express cloud infrastructure as YAML alongside their application manifests at potentially the cost of async validation. Pushkar pulls back the curtain on how Cruise managed infrastructure at scale using a custom internal platform called 'Juno' to bootstrap GCP projects, repositories, and permissions, while leveraging Crossplane for…

Share Episode Co-Founder of Grafana Labs, Anthony Woods, joins to share his perspective on how open source solutions are thriving despite the deluge of pull requests being thrown at them through their open source repositories. Most importantly, he outlines how observability is no longer being done by users looking directly at dashboards. The data rarely makes to LLMs or automation, it rarely makes sense even to humans looking at them, without the context. The context is critical component, and having a model that was built on the semantic concepts relevant to your use cases. And of course we can't stay away from asking on the record the current state of security of open source repositories from a vendor side. Given how their was an inevitable incident with some of the Grafana open source repos, we dig in to figure out how they are dealing with the real world impacts of malware being spread throughout the ecosystem. 💡 Notable Links: Coinbase's ridiculous spend on observability Book: Crossing the Chasm Silicon…

Share Episode As it turns out, the entire artificial intelligence boom is essentially running on Wikipedia's free labor, but while knowledge is free, physical server infrastructure definitely is not. We sit down with Moriel Schottlender, Principal Systems Software Engineer at the Wikimedia Foundation, to dissect how public systems survive an endless onslaught of high-volume AI scrapers and aggressive crawlers. Because 65% of the resource-heavy requests originate from automated bots, we explore how Wikimedia navigates this traffic without blocking legitimate users. We skip the approaches of IP-banning which doesn't work in practice and discuss actual mature architectural strategies, by focusing on the users' needs. From structured database dumps and high-volume enterprise APIs to rate-limiting and CDN caching trade-offs. It's a mind-bogglingly complex ecosystem ­of open-source, a 25-year-old PHP monolith supporting over 900 distinct site instances across 300 languages and 11 unique projects. It's an immense…

Share Episode We are joined this week by Mark Hay, CTO and co-founder of TextQL and former lead of Text Classification Infrastructure at Meta, to uncover the hidden complexities behind massive-scale machine learning. Mark explains why the most crucial features for identifying abusive behavior, like drug dealers or scammers on Facebook and Instagram, rarely rely on the content itself but instead analyze the underlying behavioral graphs, such as abnormal friend requests or messaging patterns. Of course we review the adversarial nature of spam detection, where bad actors constantly evolve from simple regex evasion to embedding messages inside images or even utilizing pure symbolic communication, like comparing different sized cucumber emojis to evade text filters. That requires diving into the evolution of database querying and the rise of the semantic layer. Mark unpacks why relying on raw LLMs to write complex SQL is a recipe for hallucinations, and how implementing a "correct by construction" semantic layer…

Share Episode We sit down with Itacama CEO Pia Wiedermayer to discuss the absurdity of siloed QA, the disaster of AI-generated API tests, and why developers hate the word "quality." This time we are asking the age-old question: Who needs testers anyway? Pia and Warren discuss how to dismantle the toxic culture of isolated quality assurance. We explore how the ghosts of waterfall development still haunt modern teams, creating silos where developers blindly throw unverified code over the wall and expect a separate QA department to magically inject quality. Included is the inevitable discussion on the psychological safety of hiding behind narrow job titles and why refusing to take collective ownership of a product is a guaranteed recipe for architectural failure. Of course we can't avoid commenting on the terrifying reality of replacing human intuition with automated hype. Pia shares a case study of a scale-up that aggressively pivoted to "full steam AI development," intentionally excluding both their Product…

Share Episode We talk with Ragic CEO Jeff Kuo about Semantic Web origins, dodging DDoS attacks, and the absolute horror of a database that randomly deletes its own files. He revisits how a 25-year-old master's thesis on the Semantic Web evolved into a massive spreadsheet-driven database builder. It's the one better Airtable alternative. Rather than forcing non-technical users into complex two-layer SQL architectures, Ragic utilizes a highly flexible, graph-based data model. Achieving this performance meant abandoning traditional ORMs to build a custom graph indexing engine on top of Berkeley DB, a key-value store. This custom implementation came with brutal growing pains, including a terrifying bug that would randomly delete the wrong data files. To survive, Ragic's team shares with us just exactly how they had to hijack the internal implementation to avoid these sorts of problems. When we get down to it, we review how they dealt with critical DDoS against their cloud providers, how they performed a cloud…

Share Episode Developers spend more than 50% of their time reading code, making it the single largest expense in software engineering. Despite this massive cost, the industry rarely discusses or optimizes how we read code. So we've brought in Tudor Girba, CEO at Feenk to help us rethink, just how software engineering should be done. Instead of relying on manual reading and generic text editors, teams must shift toward building deterministic, contextual tools to directly extract information and answer questions about their systems. The suggested solution? Contextual and composable micro-tools writen by everyone focused on exposing just the right information at the right time. This creates the opportunity for structural interrogation of your solution. And how many tools should we? We'll if one example of tool is testing, and 50% or more of your code can be tests, imagine what percentage of your software should be actually production related! Most importantly, generic tools fall short, but where can we find how…

Share Episode Welcome back to another hopefully, relief from architectural existential dread. This week, we've pulled in Seth Eliot from Arpio, (Ar-Pi-O, RPO, get it?), to dive headfirst into the beautiful, deeply expensive illusion that migrating your legacy infrastructure to a major hyperscaler magically grants it instant immortality. It doesn't. We break down the shared responsibility model for resilience, which was conveniently cribbed straight from the security model, and analyze how the foundational promise of automated fault isolation boundaries routinely crumbles. From cloud providers sticking multiple "independent" availability zones inside the exact same physical building, to multi-AZ cascading anomalies, to regional power grid failures, it's clear your provider's abstractions aren't nearly as resilient as their marketing slides suggest. Discussed within is the "Thundering Herd" phenomenon, that can't be ignored even when the failover clusters are designed correctly. From cross-organization KMS…

Share Episode This week's adventure tackles the absolute absurdity of modern enterprise infrastructure, where a single company can easily find itself running multiple different CI/CD platforms due to unchecked mergers and acquisitions. We've brought in Chris Farris, AWS Security Hero and consults with companies via Securosis. And dig deep to find the security cracks and philosophize about the real world impacts of tech debt in the AI age. Management rarely prioritizes standardization, leaving security teams to defend a chaotic swamp of mixed cloud providers, GitHub repositories, and nostalgic on-prem Bitbucket instances. We define this accumulated technical debt not as some abstract concept, but as literal potholes on the infrastructure Autobahn—annoying speed bumps that permanently damage velocity and set organizations up for an inevitable disaster. We contrast this with the evolution from old-school sysadmins cutting their fingers on rack screws to modern engineers spinning up entire architectures with a…

Share Episode We grabbed Donald Nguyen, co-founder and CTO at Corvic, to discuss the absurd complexities of enterprise data and multimodal inference. We explore how organizations habitually hoard mountains of useless, "dead" data just out of the sheer fantascy that someone might ask for it later. We highlight the fundamental disconnect where data collectors using tools like Airbyte and Kafka speak a completely different language than the business consumers analyzing it in Excel. True scale isn't just about managing petabytes; it's the absolute nightmare of extracting subjective business meaning from flat PDFs and invoices. In the deep-end of vector embeddings, we're challenging translating data into a different semantic universe requires imposing a heavy business bias. Auditors and artists will view the exact same invoice completely differently, meaning your embedding model selection is incredibly subjective to the business context. The industry's desperate search for actual AI success stories beyond basic…

Share Episode Down to business with GitHub's Cassidy Williams, Senior Director of Developer Advocacy at GitHub, where we try to untangle the existential dread of modern software development. It includes the sheer absurdity of managing a platform that officially crossed the one billion commit mark in 2025. Currently absorbing a completely unreasonable 275 million commits per week, GitHub's technical debt is naturally showing its age under the weight of AI agents aggressively creating pull requests. And with company's own copilot advocating for more, we explore the daily reality of being the internet's punching bag during an outage, and how the "Tiny Wins" buy back developer affection by still shipping the critical features. Which of course is a small signal in the sea of the industry's collective identity crisis: vibe coding and the valley of AI-generated garbage. Discussed is one suggested solution of strongly typed languages which are skyrocketing in popularity because we desperately need rigid guardrails…

Share Episode Founder of Bespinian and long-time cloud solutions architect, Lena Fuhrimann, sits down with us to clarify the widespread confusion around serverless architecture. We discuss how serverless is often incorrectly equated solely with Function as a Service (FaaS), when it actually represents a broader spectrum on the abstraction ladder—including managed AI inference, container platforms, and databases. Lena shares her early career traps of building a fragmented landscape of sixty "nano-services" and explains why starting with a well-architected monolith and progressively breaking out microservices based on distinct resource or lifecycle requirements is a much saner approach. Then we shift to drivers behind cloud migrations, emphasizing that the primary financial benefit of serverless isn't necessarily shrinking the monthly cloud provider bill, but rather optimizing your most expensive resource: engineering time. By offloading mundane infrastructure patching to the cloud provider, teams can focus…

Share Episode We sit down with Ian Duncan, senior staff engineer on the stability team at Mercury, to discuss the delicate balance of choosing your tech stack and the implications. That means explore the concept of the novelty budget or frequently known as "Choose Boring Technology". It emphasizes why companies should carefully spend their innovation tokens on things that actually move the needle, rather than reinventing the wheel. Mercury leverages simple technology like Postgres and EC2 instances alongside high-innovation bets like Haskell and Nix to maintain stability. The conversation unpacks the hidden complexities of over-relying on standard tools, sharing a cautionary tale about using a Postgres table as a massive queuing system until it consumed all the database resources and caused login failures. To solve architectural scaling without descending into nanoservice madness, we jump to discussing monolithic build systems. By leveraging hermetically sealed, modular build targets, teams can achieve…

Share Episode We explore the past and AI-driven future of Infrastructure as Code with Cloud Posse's Erik Osterman, discussing various IaC traumas. Erik maintains the world's largest repository of open-source IaC modules. Looking back at the dark ages of infrastructure, from the early days of raw CloudFormation and Capistrano to the rise and fall of tools like Puppet and Chef, we discuss the organic, messy growth of cloud environments. Where organizations frequently scale a single AWS account into a tangled web rather than adopting a robust multi-account architecture guided by a proper framework. The conversation then shifts to the modern era of rapid integration of infrastructure development. While generating IaC with large language models can be incredibly fast, it introduces severe risks if left unchecked, and we explore how organizations can protect themselves by relying on Architectural Decision Records (ADRs) and predefined "skills". The hopeful goal of ensuring autonomous deployments are compliant,…

Share Episode We sit down with Jaikumar Ganesh, Head of Engineering at AnyScale, to explore the intricacies of heterogeneous compute. He unpacks the growing CPU/GPU divide, detailing how ML pipelines require precise orchestration — using CPUs for data reading and writing while leveraging expensive, massive-die GPUs for chunking and embedding. Warren brings the insight that, with AI agents rapidly changing how software is created, building is now a requirement of the business-focused team. And our guest shares how sales and marketing departments are increasingly using tools like Cursor and Claude to develop their own workflow automations. We discuss the challenges that this shift begs: what is engineering really doing for us? JK emphasizes that the core responsibility of the engineering organization is reliability. While anyone can generate code, running stable production software requires the deep "battle scars", robust observability, and meticulous release processes that only a dedicated engineering team…

Share Episode In this adventure, we sit down with Dan Wahlin, Principal of DevRel for JavaScript, AI, and Cloud at Microsoft, to explore the complexities of modern infrastructure. We examine how cloud platforms like Azure function as "building blocks". Which of course, can quickly become overwhelming without the right instruction manuals. To bridge this gap, one potential solution we discuss is the emerging reliance on AI "skills"—specialized markdown files. They can give coding agents the exact knowledge needed to deploy poorly documented complex open-source projects to container apps without requiring deep infrastructure expertise. And we are saying the silent part outloud, as we review how handing the keys over to autonomous agents introduces terrifying new attack vectors. It's the security nightmare of prompt injections and the careless execution of unvetted AI skills. Which is a blast from the past, and we reminisce how current downloading of random agent instructions to running untrusted executables…

Share Episode How much do you really know about the protocol that everything is built upon? This week, we go behind the scenes with Simone Carletti, a 13-year industry veteran and CTO at DNSimple, to explore the hidden complexities of DNS. We attempt to uncover why exactly DNS is often the last place developers check during an outage, drawing fascinating parallels between modern web framework abstractions and network-level opaqueness. Simone shares why his team relies on bare-metal machines instead of cloud providers to run their Erlang-based authoritative name servers, highlighting the critical need to control BGP routing. We trade incredible war stories, from Facebook locking themselves out of their own data centers due to a BGP error, to a massive 2014 DDoS attack that left DNSimple unable to access their own log aggregation service. The conversation also tackles the reality of implementing new standards like SVCB and HTTPS records, and why widespread DNSSEC adoption might require an industry-wide…

Share Episode We are joined by Daan Boerlage, CTO at Mavexa as we tackle the long-awaited arrival of IPv6 in cloud infrastructure. Here, we highlight how migrating to an IPv6-native setup eliminates public/private subnet complexity and expensive NAT gateways natively. As well as entirely sidestepping the nightmare of IP collisions during VPC peering. Beyond the financial savings of ditching IPv4 charges, we explore the technical superiority of IPv6. Daan breaks down just how mind-bogglingly large the address space is, and focuses on how it solves serverless IP exhaustion while systematically debunking the pervasive myth that NAT is a security feature. We also discuss how IPv6's end-to-end connectivity, paving the way for next-generation protocols like QUIC, HTTP/3, and WebTransport. The episode rounds out with a cathartic venting session about legacy architecture, detailing a grueling nine-year migration away from a central shared database that ironically culminated in a move to Salesforce. Almost by design,…

Share Episode In this episode, we examine how the software industry is fundamentally changing. We're joined by our expert guest, Matt Edmunds, a long-time UX director, principal designer, and Principal UX Consultant at Tiny Pixls. The episode kicks, analyzing how early AI implementation in Applicant Tracking Systems (ATS) created rigid hiring processes that actively filter out the varied candidates who actually bring necessary diversity to engineering teams. Of course we get to the world of "vibe coding", and revisit the poor LLM usage highlighted in the DORA 2025 report, exploring how professionals without traditional software engineering backgrounds are leveraging models to generate functional code. Matt details his hands-on experience using the latest models of Claude Opus and Gemini Pro, successfully building low-level C virtual audio driver in 30 minutes drive by personal needs. We discuss the inherent challenges of large context windows, and coin the term "guess-driven development". To combat these…

Share Episode We dive into the shifting landscape of developer relations and the new necessity of optimizing documentation for both humans and LLMs. Melinda Fekete joins from Unleash, and suggests transitioning to platform to help get this right by utilizing LLMs.txt files to cleanly expose content to AI models. The conversation then takes a look at the June GCP outage, which was triggered by a single IAM policy change. This illustrates that even with world-class CI/CD pipelines, deploying code using runtime controls such as feature flags is still risky. Feature flags can't even save GCP and other cloud providers, so what hope do the rest of us have. Finally, we discuss the practical implementation of these systems, advocating for "boring technology" like polling over streaming to ensure reliability, and conducting internal "breakathons" to test features before a full rollout. 💡 Notable Links: Diátaxis - Who is article this for? Fern - Docs Platform CloudFlare - Feature Flag causes outage AWS - Graceful…

Share Episode Dorota, CEO of Authress, returns to apply the US Supreme Court’s definition of obscenity to a scandalous topic: Engineering Productivity. In a world obsessed with AI-driven efficiency, Dorota and Warren argue that software development productivity has nothing to do with manufacturing "gizmos" and everything to do with feelings. They dismantle the factory-floor mentality that equates typing speed with value, suggesting instead that the most productive work often happens while staring out a train window or disassociating in the shower. The conversation takes a dark turn into the reality of performance reviews. If productivity is subjective, how do you decide who gets promoted? Dorota proposes the "Resentment Metric"—ignoring Jira tickets in favor of figuring out who the team has secret concerns fo. They also roast the "100% utilization" fallacy, noting that a fully utilized highway is just a parking lot, and the same logic applies to engineering teams that don't schedule downtime for actual…

Share Episode ⸺ Episode Sponsor: Rootly AI - https://dev0ps.fyi/rootlyai Paul Conroy, CTO at Square1, joins the show to prove that the best defense against malicious bots isn't always a firewall—sometimes, it’s creative data poisoning. Paul recounts a legendary story from the Irish property market where a well-funded competitor attempted to solve their "chicken and egg" problem by scraping his company's listings. Instead of waiting years for lawyers, Paul’s team fed the scrapers "Project Yellow Brick Road": fake listings that placed the British Prime Minister at 10 Downing Street in Dublin and the White House in County Cork. The result? The competitor’s site went viral for all the wrong reasons, forcing them to burn resources manually filtering junk until they eventually gave up and targeted someone else. We also dive into the high-stakes world of election coverage, where Paul had three weeks to build a "coalition builder" tool for a national election. The solution wasn't a complex microservice architecture,…

Share Episode "Those memes are not going to make themselves." Dorota, CEO of Authress, joins us to roast the 2025 DORA Report, which she argues has replaced hard data with an AI-generated narrative. From the confusing disconnect between feeling productive and actually shipping code to the grim reality of a 30% acceptance rate, Warren and Dorota break down why this year's report smells a lot like manure. We dissect the massive 142-page 2025 DORA Report. Dorota argues that the report, which is now rebranded as the "State of AI-Assisted Software Development", feels less like a scientific study of DevOps performance and more like a narrative written by an intern using an LLM prompt. The duo investigates the "stubborn results" where AI apparently makes everyone feel like a 10x developer, where the hard results tell a different story. AI actually increases software and product instability — failing to improve. The conversation gets spicy as they debate the "pit of failure" that is feature flags (often used as a…

Share Episode ⸺ Episode Sponsor: Incident.io - https://dev0ps.fyi/incidentio "My biggest legacy at Google is the amount of systems I broke." — Sam Goto joins the show with a name that strikes fear into engineering systems everywhere. As a Senior Staff Engineer on the Chrome team, Sam shares the hilarious reality of having the last name "Goto," which once took down Google's internal URL shortener for four hours simply because he plugged in a new computer. Sam gets us up to speed with Federated Credentials Management (FedCM), as we dive deep into why authentication has been built despite the browser rather than with it, and why it’s time to move identity from "user-land" to "kernel-land". This shift allows for critical UX improvements for logging in all users irrespective of what login providers you use, finally addressing the "NASCAR flag" problem of infinite login lists. Most importantly, he shares why you don't need to change your technology stack to get all the benefits of FedCM. Finally, Sam details the…

In this episode, we sit down with Barzan Mozafari, MIT alum and University of Michigan professor, to explore how AI-driven automation is revolutionizing cloud database optimization. Barzan shares his experiences in bridging academia and industry, and how his work in AI-powered database tuning is reshaping cloud infrastructure efficiency. We discuss the challenges of AI adoption in cloud computing, addressing concerns like implementation risks, security, and trust in autonomous agents. Barzan explains how machine learning models can optimize performance, reduce cloud costs, and automate database management, freeing engineers from tedious manual tuning. Additionally, we explore the future of AI in database systems, the evolving landscape of public and private datasets, and what the next decade holds for data-driven automation. Whether you're a DevOps engineer or a database architect, this episode is packed with insights into the intersection of AI and cloud technology. 🎯 Picks: Warren - L8 Conference 2024…

Will is absent this week due to an infrastructure issue, so Warren and Jillian take the reins in a fascinating discussion with Hikari Senju, founder of Omneky. Hikari shares how AI is revolutionizing ad technology by automatically generating creative assets tailored to different audiences and platforms. His insights reveal how businesses can scale their advertising while maintaining brand consistency and reducing content creation costs. The conversation explores the balance between automation and human oversight, the ethical implications of AI-driven marketing, and how the ad personalization technology is evolving. Hikari also dives into the infrastructure behind Omneky, discussing cloud-based AI deployment, model selection, and strategies for ensuring ad content stays fresh and relevant. If you've ever wondered how AI is reshaping the future of advertising. 🎯 Picks: Warren - Dune Hikari - Accelerando Jillian - Data Discovery with Dabble of DevOps AI

Share Episode ⸺ Episode Sponsor: Incident.io - https://dev0ps.fyi/incidentio Elise, VP and Head of UX at Unleash, joins us to talk all about UX. Self identifying as probably "The annoying lady in the room" and a career spanning nearly 30 years—starting before "UX" was even a job title — joins us to dismantle the idea that User Experience is just about moving pixels around. Here we debate the friction between engineering, sales, and the customer. We get to the bottom of whether or avoiding end-user interaction, understand, and research is a career-limiting move for staff+ engineers. Or should you avoid forcing a world-class developer to facilitate a call with a non-technical user if it makes them uncomfortable? Warren calls out the "Pit of Failure" often faced by teams as they seek to introduce feature flags. They can become a crutch, leading teams to push untested code into production simply because they can toggle it off—a scenario he calls the "pit of failure". And Elise dives into a great story recounting…

Share Episode ⸺ Episode Sponsor: Incident.io - https://dev0ps.fyi/incidentio Warren is joined by Olga Kundzich, Co-founder and CTO of Moderne, to discuss the reality of technical debt in modern software engineering. Olga reveals a shocking statistic: without maintenance, cloud-native applications often cease to function within just six months. And from our experience, that's actually optimistic. The rapid decay isn't always due to bad code choices, but rather the shifting sands of third-party dependencies, which make up 80 to 90% of cloud-native environments. We review the limitations of traditional Abstract Syntax Trees (ASTs) and the introduction of OpenRewrite's Lossless Semantic Trees (LSTs). Unlike standard tools, LSTs preserve formatting and style, allowing for automated, horizontal scaling of code maintenance across millions of lines of code. This fits perfectly in to the toolchain that is the LLMs and open source ecosystem. Olga explains how this technology enables enterprises to migrate…

Share Episode In a special solo flight, Warren welcomes Meagan Cojocar, General Manager at Pulumi and a self-proclaimed graduate of “PM school” at AWS. They dive into what it's like to own an entire product line and why giving up that startup hustle for the big leagues sometimes means you miss the direct signal from your users. The conversation goes deep on the paradox of open-source where direct feedback is gold, but dealing with license-shifting competitors can make you wary. From the notorious HashiCorp kerfuffle to the rise of OpenTofu, they explore how Pulumi maintains its commitment to the community amidst a wave of customer distrust. Meagan highlights the invaluable feedback loop provided by the community, allowing for direct interaction between users and the engineering team. This contrasts with the "telephone game" that can happen in proprietary product development. The conversation also addresses the recent industry shift and then immediate back-peddling from open-source licenses, discussing the…

Share Episode Microsoft's John Papa, Partner General Manager of Developer Relations for all things dev and code joins the show to talk developer relations...from his Mac. He reveals his small part in the birth of VS Code (back when its codename was Ticino) after he spent a year trying a new editor every month. The conversation dives deep into "Agentic AI," where John predicts developers will soon become "managers of agents". But is it all hype? John and Warren debate the risks of too much automation (no, AI should not auto-merge your PRs) and the terrifying story of a SaaS built with "zero handwritten code" that immediately got hacked because the founder was "not technical". The episode highlights John's jaw-dropping war stories from Disney, including a mission-critical hotel lock system (for 5,000+ rooms) that was running on a single MS Access database under a desk. It's a perfect, cringeworthy lesson in why "we don't have time to test" is the most expensive phrase in tech, and why we need a human in the…

Share Episode ⸺ Episode Sponsor: Attribute - https://dev0ps.fyi/attribute In the wake of one of the worst AWS incidents in history, we're joined by Lawrence Jones, Founding Engineer at Incident.io. The conversation focuses on the challenges of managing incidents in highly regulated environments like FinTech, where the penalties for downtime are harsh and require a high level of rigor and discipline in the response process. Lawrence details the company's evolution, from running a monolithic Go binary on Heroku to moving to a more secure, robust setup in GCP, prioritizing the use of native security primitives like GCP Secret Manager and Kubernetes to meet the obligations of their growing customer base. We spotlight exactly how a system can crawl GitHub pull requests, Slack channels, telemetry data, and past incident post-mortems to dynamically generate an ephemeral runbook for the current incident.Also discussed are the technical challenges of using RAG (Retrieval-Augmented Generation), noting that they rely…

Share Episode ⸺ Episode Sponsor: Attribute - https://dev0ps.fyi/attribute We're joined by 20 year industry veteran and DevOps advocate, Adam Korga, celebrating the release of his book IT Dictionary. In this episode we quickly get down to the inspiration behind postmortems as we review some cornerstone cases both in software and in general technology. Adam shares how he started in the industry, long before DevOps was a coined term, focused on making systems safer and avoiding mistakes like accidentally dropping a production database. we review the infamous incidents of accidental database deletion, by LLMs and human's alike. And of course we touch on the quintessential postmortems in civil engineering, flight, and survivorship bias from World War II through analyzing bullet holes on returning planes. 💡 Notable Links: Adam's book: IT Dictionary Knight Capital: the 45 minute nightmare Work Chronicles Comic: Will my architecture work for 1 Million users? 🎯 Picks: Warren - Cuitisan CANDL storage containers Adam -…

Share Episode ⸺ Episode Sponsor: Attribute - https://dev0ps.fyi/attribute Jenna Pederson, Staff Developer Relations at Pinecone, joins us to close the loop on Vector Databases. Demystifies how they power semantic search, their role in RAG, and also unexpected applications. Jenna takes us beyond the buzzword bingo, explaining how vector databases are the secret sauce behind semantic search. Sharing just how "red shirt" gets converted into a query that returns things semantically similar. It's all about turning your data into high-dimensional numerical meaning, which, as Jenna clarifies, is powered by some seriously clever math to find those "closest neighbors." The conversation inevitably veers into Retrieval-Augmented Generation (RAG). Jenna reveals how databases are the unsung heroes giving LLMs real brains (and up-to-date info) when they're prone to hallucinating or just don't know your company's secrets. They complete the connection from proprietary and generalist foundational models to business relevant…

Share Episode This episode we are joined by Andrew Moreland, co-founder of Chalk. Andrew explains how their company's core business model is to deploy their software directly into their customers' cloud environments. This decision was driven by the need to handle highly sensitive data, like PII and financial records, that customers don't want to hand over to a third-party startup. The conversation delves into the surprising and complex challenges of this approach, which include managing granular IAM permissions and dealing with hidden global policies that can block their application. Andrew and Warren also discuss the real-world network congestion issues that affect cross-cloud traffic, a problem they've encountered multiple times. Andrew shares Chalk's mature philosophy on software releases, where they prioritize backwards compatibility to prevent customer churn, which is a key learning from a competitor. Finally, the episode explores the advanced technical solutions Chalk has built, such as their unique…

Share Episode We welcome guest Ang Li and dive into the immense challenge of observability at scale, where some customers are generating petabytes of data per day. Ang explains that instead of building a database from scratch—a decision he says went "against all the instincts" of a founding engineer—Observe chose to build its platform on top of Snowflake, leveraging its separation of compute and storage on EC2 and S3. The discussion delves into the technical stack and architectural decisions, including the use of Kafka to absorb large bursts of incoming customer data and smooth it out for Snowflake's batch-based engine. Ang notes this choice was also strategic for avoiding tight coupling with a single cloud provider like AWS Kinesis, which would hinder future multi-cloud deployments on GCP or Azure. The discussion also covers their unique pricing model, which avoids surprising customers with high bills by providing a lower cost for data ingestion and then using a usage-based model for queries. This is…

In this episode of Adventures in DevOps, we dive into the world of FinOps, a concept that aims to apply the DevOps mindset to financial accountability. Yasmin Rajabi, Chief Strategy Officer at CloudBolt, joins us to demystify, as we acknowledge the critical challenge of bringing together financial accountability and engineering teams who often are not paying attention to the business. The discussion further explores the practicalities of FinOps in the context of cloud spending and Kubernetes. Yasmin highlights that a significant amount of waste in organizations comes from simply not turning off unused systems and not right-sizing resources. She explains how tools like Horizontal Pod Autoscaler (HPA) and Vertical Pod Autoscaler (VPA) can help, but also points out the complexities of optimizing across horizontal and vertical scaling behaviors. The conversation touches on "shame back reporting" as a way to provide visibility into costs for engineering teams, although the conversation emphasizes that providing…

Get ready for a lively debate on this episode of Adventures in DevOps. We're joined by Brian Pontarelli, founder of FusionAuth and CleanSpeak. Warren and Brian face off by diving into the controversial topic of multitenant versus single-tenant architecture. Expert co-host Aimee Knight joins to moderate the discussion. Ever wondered how someone becomes an "auth expert"? Warren spills the beans on his journey, explaining it's less about a direct path and more about figuring out what it means for yourself. Brian chimes in with his own "random chance" story, revealing how they fell into it after their forum-based product didn't pan out. Aimee confesses her "alarm bells" start ringing whenever multitenant architecture is mentioned, jokingly demanding "details" and admitting her preference for more separation when it comes to reliability. Brian makes a compelling case for his company's chosen path, explaining how their high-performance, downloadable single-tenant profanity filter, CleanSpeak, handles billions of…

Episode Sponsor: PagerDuty - Checkout the features in their official feature release: https://fnf.dev/4dYQ7gL This episode dives into a fundamental question facing the DevOps world: Did Kubernetes truly win the infrastructure race because it was the best technology, or were there other, perhaps less obvious, factors at play? Omer Hamerman joins Will and Warren to take a hard look at it. Despite the rise of serverless solutions promising to abstract away infrastructure management, Omer shares that Kubernetes has seen a surge in adoption, with potentially 70-75% of corporations now using or migrating to it. We explore the theory that human nature's preference for incremental "step changes" (Kaizen) over disruptive "giant leaps" (Kaikaku) might explain why a solution perceived by some as "worse" or more complex has gained such widespread traction. The discussion unpacks the undeniable strengths of Kubernetes, including its "thriving community", its remarkable extensibility through APIs, and how it inadvertently…

In this episode, Aimee Knight, an expert in Site Reliability Engineering (SRE) whose experience hails from Paramount and NPM, joins the podcast to discuss her journey into SRE, the challenges she faced, and the strategies she employed to succeed. Aimee shares her transition from a non-traditional background in JavaScript development to SRE, highlighting the importance of understanding both the programming and infrastructure sides of engineering. She also delves into the complexities of SRE at different scales, the role of playbooks in incident management, and the balance between speed and quality in software development. Aimee discusses the impact of AI and machine learning on SRE, emphasizing the need for responsible use of these tools. She touches on the importance of understanding business needs and how it affects decision-making in SRE roles. The conversation also covers the trade-offs in system design, the challenges of scaling applications, and the importance of resilience in distributed systems. Aimee…

In this episode, we delve into the concept of MCP (Machine Control Protocol) servers and their role in enabling agent interactions. Gil Feig, the co-founder and CTO of Merge, shares insights on how MCP servers facilitate efficient and secure integration between various services and APIs. The discussion covers the benefits and challenges of using MCP servers, including their stateful nature, security considerations, and the importance of understanding real-world use cases. Gil emphasizes the need for thorough testing and evaluation to ensure that MCP servers effectively meet user needs. Additionally, we explore the implications of MCP servers on data security, scaling, and the evolving landscape of API interactions. Warren chimes in with experiences integrating AI with Auth. Will stuns us with some nuclear fission history. And finally, we also touch on the balance between short-term innovation and long-term stability in technology, reflecting on how different generations approach problem-solving and knowledge…

Warren talks with Nathan Goulding, SVP of Engineering at Vultr, about what it actually takes to run a high-performance cloud platform. They cover everything from global game server latency and hybrid models to bare metal provisioning and the power/cooling constraints that come with modern GPU clusters. The discussion gets into real-world deployment challenges like scaling across 32 data centers, edge use cases that actually matter, and how to design systems for location-sensitive customers—whether that's due to regulation or performance. Additionally, there's talk about where the hyperscalers have overcomplicated pricing and where simplicity in a flatter pricing model and optimized defaults are better for everyone. There's a section on nuclear energy (yes, really), including SMRs, power procurement, and what it means to keep scaling compute with limited resources. If you're wondering whether your app actually needs high-performance compute or just better visibility into your costs, this is the episode.…

Inna Tokarev Sela, CEO and founder of Illumex, joins the crew to break down what it really means to make your data “AI-ready.” This isn't just about clean tables—it's about semantic fabric, business ontologies, and grounding agents in your company's context to prevent the dreaded LLM hallucination. We dive into how modern enterprises just cannot build a single source of truth, not matter how hard they try. All the while knowing that it's required to build effected agents utilizing the available knowledge graphs and. The conversation unpacks democratizing data access and avoiding analytics anarchy. Inna explains how automation and graph modeling are used to extract semantic meaning from disconnected data stores, and how to resolve conflicting definitions. And yes, Warren finally coughs up what's so wrong with most dashboards. Lastly, we quickly get to the core philosophical questions of agentic systems and AGI, including why intuition is the real differentiator between humans and machines. Plus: storage cost…

Sylvain Kalache, Head of Developer Relations at Rootly joins us to explore the new frontier of incident response powered by large language models. We dive into the evolution of DevRel and how we meet the new challenges impacting our systems. We explore Sylvain's origin story in self-healing systems, dating back to his SlideShare and LinkedIn days. From ingesting logs via Fluentd to building early ML-driven RCA tools, he shares a vision of self-healing infrastructure that targets root causes rather than just restarting boxes. Plus, we trace the historical arc of deterministic and non-deterministic tools. The conversation shifts toward real-world applications, where we're combining logs, metrics, transcripts, and postmortems to give SREs superpowers. We get tactical on integrating LLMs, why fine-tuning isn't always worth it, and how the Model Context Protocol (MCP) could be the USB of AI ops, but how it is still insecure. We wrap by facing the harsh reality of "incident vibing" in a world increasingly built by…

This week, Paul Marston from Ankr joins the crew to unpack the madness that is modern blockchain infrastructure. From his wild career transition out of financial services into 24/7 node ops for Web3, Paul shares the brutal truth about uptime expectations, decentralization challenges, and why hard forks are more like enterprise schema upgrades with a community twist. If you've ever wondered why managing a blockchain node is like owning a temperamental pet server, this one's for you. The team goes deep on the nitty-gritty of load balancing across dozens of chains, explaining why routing traffic to the “wrong” archive node could ruin your day—and how Ankr's custom load balancer is basically magic for JSON-RPC calls. Warren tosses out wild scenarios about encrypted data smuggling via blockchain, while Will confesses his angry typing habit (yes, it's back). The discussion gets even more fun with debates on innovation vs. rigor, Web2's forgotten best practices, and why testing in prod might not be such a dirty…

In this episode, Will and Warren welcome Adriana Villela — CNCF ambassador, Dynatrace advocate, and host of the Geeking Out podcast — for a wide-ranging conversation on observability in CI/CD pipelines. Adriana shares her journey from “On Call Me Maybe” to her own podcast, her work with OpenTelemetry, and why observability isn't just for SREs anymore. The crew digs into how telemetry should be integrated across the software development lifecycle — from development to QA to production — and what that really looks like in modern teams. Adriana drops knowledge on CI/CD failures, distributed traces, and even how to bring observability to other parts of the business like recruiting and onboarding. She also explains how she got involved in the OpenTelemetry end-user SIG and what's next for the observability movement. Things get persona as we trade war stories about SVN, terrible version control systems, reusable grocery bags, and the ethics of AI log parsers. Adriana closes with a powerful take: observability is a…