How to get multiple agents to play nice at scale

How to get multiple agents to play nice at scale - Stack Overflow

Stack Overflow Business Stack Internal: the knowledge intelligence layer that powers enterprise AI.Stack Data Licensing: decades of verified, technical knowledge to boost AI performance and trust.Stack Ads: engage developers where it matters — in their daily workflow.SPONSORED BY INTUITChase Roossin, group engineering manager, and Steven Kulesza, staff software engineer, from Intuit join the podcast to chat about what might be the hardest problem in engineering right now: getting multiple AI agents to work together in a complex system. They discuss how automated evals can make agent behaviors more predictable, agent swarms vs. one highly skilled agent, and how customer behavior shaped their technical architecture.Episode notesWant to work on complex engineering problems like these? Explore careers at Intuit.We’ve worked with Intuit on a few other great blogs and podcasts, including Best practices for building LLMs and How Intuit democratizes AI development across teams through reusability.Connect with Chase on LinkedIn.Connect with Steven on LinkedIn.Congrats to Lifejacket badge winner Sean for saving Creating the simplest HTML toggle button? with a great answer.TRANSCRIPT[Intro Music]Ryan Donovan: Hello everyone, and welcome to the Stack Overflow Podcast, a place to talk all things software and technology. I am your host, Ryan Donvan, and today we are talking about the complexities of multi-agent architectures, and today it's sponsored by the fine folks at Intuit. And my guests are Steven Kulesza, staff software engineer, and Chase Roosin, engineering manager at Intuit. So, welcome to the show.Chase Roosin: Thanks for having us, Ryan. We're excited.Steven Kulesza: Yeah, we really appreciate it.Ryan Donovan: Oh, my pleasure. So, before we get into the complexities of orchestrating multiple agents, tell us a little bit about yourselves. How did you get involved in software and technology?Chase Roosin: Yeah, I found at an early age I had this infatuation with building, and I always thought I was gonna be a mechanical engineer, but I rapidly found out that it's cheaper and quicker to write software than to go build real-world products. And early in middle school and high school, I started writing simple websites, and it just became a huge passion of mine, seeing the ability to impact, hundreds, thousands, millions of people so instantaneously. And I went off and got my computer science degree and then started my career at Intuit.Ryan Donovan: Steven, how about you?Steven Kulesza: Yeah, I think Chase and I kinda have a similar path there. I started super young too, probably like 12 [or] 11. And I was just really interested in building blogs. I was working on forums and stuff like that, and just managing them. And then, I was building games, so JavaScript games, and then got into building web scrapers.Ryan Donovan: Oh, there you go.Steven Kulesza: And I just went from there. And then, I got really into wanting to run startups and build startups. And I just saw engineering as this real-life magic where you, whatever you dream, you can build. And that just drove me throughout my career and as [an] individual in general, and just fell in love with it. Started from there and just took professional jobs after that.Ryan Donovan: So, you mentioned the magic. We're talking a little bit about some of the newest engineering magic, and we've talked with Intuit about AI, and the various platforms and programs you've been building all the way from Gen Os and stuff. How has your company's thinking evolved on AI on the enterprise?Chase Roosin: Yeah, no, that's a great question. You called it out. We've been heavily investing in AI for many years. This isn't something new to us. And the beautiful thing that Steven and I are fortunate enough to get to build upon is a lot of that prior investment. GenOS and other platform-specific products that the teams build centrally enable us to move with the velocity that we need to unlock some of these new experiences. Even though the technology is starting from scratch, we ourselves are not, and we get to build off those foundational layers, and that's how we've been able to evolve and iterate so quickly in this space that we know is changing by the minute. And yeah, it's been really great to see how our prior investments as a company have really paid dividends into our future work.Steven Kulesza: Yeah, I think with GenOS, with having security standards built into our calls and all that, and that's really powered a lot of our growth, because we don't have to worry about those things. Those are the foundational blocks that we get to build on top of, and with our velocity, just building these tools quickly. It's really excelled that.Ryan Donovan: It's a very composable AI system you're building, almost like having a design pattern library or something.Chase Roosin: Yeah, exactly. We have our central orchestrator, and then the operating system that's built around it, and then as Steven called out, all of these specific guardrails that we can just pull in, and it helps keep us nimble so we can go iterate with the customer. But when we need [to] dip back into the platform, we have those tools at our disposal. It's like you're going through a candy shop and you can pick and choose what pieces you need in order to be successful in that mission.Steven Kulesza: It's been a huge unlock at the enterprise scale too, having those safety nets and those Lego pieces that power product development and AI engineering that we don't really have to think about as much.Ryan Donovan: Now, basically the whole industry is moving to this kind of age-agentic AI paradigm, where it's each AI is a little piece, so maybe you're ahead of the game. But it seems like the issue people are running into is how do you coordinate that? How do you orchestrate multiple agents? How are you all thinking about that issue?Chase Roosin: I wanna maybe take us back to how these agents were originally released, and some of the customer problems we wanted to focus on. About a year and a half ago when we were releasing a lot of these agents, they were in bespoke parts of our product, and they did work in doing work on behalf of the customer, and guiding them through the product. But what we are trying to solve now is, how do we bring those on a level playing field to our customers, right? They spend 60 hours a month doing independent work across our products. How can we give them this one magical experience that ties all of these agents together? And that's what kick-started this work stream. And Steven can go into detail a lot about the orchestration. We wanted to be able to bring in our customers into this singular spot, where all the agents now are doing work on their behalf instead of them having to go and reach out independently to one of those agents.Steven Kulesza: Earlier on, as Chase was mentioning, we had all these individual agents. It came to the idea of we have all this distributed power across the platform, right? How do we have all those things coordinate and work together? So, it's almost like you wanna build an organization, and those are your employees, right? And how do we make them all work together? So, that kind of drove like the whole thought process of like, how do we go back to your main question of orchestrating these multiple agents? So, that was our starting point. We had all these distributed pieces, and when I think when NCP and all those protocols were first releasing, I was playing with that and, calling these different distributed agents, and using intent, and all that stuff through React Loops, and breaking apart user queries, get those answers as quickly as possible, and then synthesize on those, get those back to the user. So, that's how it first kicked off with that. And then, the evolution has gone on since there. And we can go into detail as much as you want.Ryan Donovan: I've been talking about this with folks, and it seems like AI agents are kind of speed running the microservices, service-oriented architecture paradigm. Are the problems that AI agents fac[ing] the same as the microservices, or are there sort of unique issues that come from agents?Steven Kulesza: Yeah. There's unique problems, as well as the distribution, too. So, microservices, especially at Intuit, where we have so many different teams, and so many different products and domains within QuickBooks itself, and within all of the other products and services at Intuit. Yeah, we face a lot of those problems. Some of the things that you wouldn't see in smaller agent systems or whatever it may be, we do face those distributed problems where similar to any other microservice. How are item potency, passing things back and forth, and yeah.Chase Roosin: And even, we're talking about agents for sure, and that's the output, but organizationally, how do you have teams divide, and conquer, and give them the agency and autonomy to build these independent experiences? And similar to the microservices, there's gonna be services that do relatively the same thing. You know what I mean? And to these models, that's a little bit more complex, right? They look the same at face value, and you're like, 'does it go to agent A? Does it go to agent B?' They have very similar inputs, or approaches, or things that it wants to accomplish. And some of those have been challenges or struggles that we've had to overcome that are outside of technological boundaries.Steven Kulesza: Dealing with that, we're super heavy on evaluations, so things are evaluation-driven. As we onboard agents, and we'll get into the skills and tools paradigm that we're growing into, all these distributed teams, they don't know who's working on what, how do we coordinate it all, and how do we make sure internet intelligence is giving the best answers possible? Evaluation. So, each team, as they onboard, has to give us their golden data sets. And we test that against the base layer into it intel