Ai harness

A harness is some code around an AI agent that makes it more effective. Why we've seen people build these specific use case harnesses is sometimes with a specific job, you just want to micromanage a little bit. You just want to be more prescriptive about how that job gets done. [music] I'm going to show you how it works and then we will talk about how I built it.

So the interface I built for my harness is a terminal UI. The harness core is run on clawed agent SDK and then it's connected to real tools. So it's connected [music] to Sentry Vcel and then it's connected to linear and GitHub in terms of getting [music] tasks done. I think we all have done good work.

But then now I've realized that these agents can help us solve very very specific problems by constraining that work. It's really like changed my mind about how work gets done. [music] Everybody's saying it's not the model, it's the harness. But you know what?

Not everybody is saying what is [music] a harness. In today's How I AI episode, I am going to demystify the idea of a harness, write my own harness, and show you how you can do the same, and explain to you why a custom harness makes sense and could be better than using claude code or codec alone. Let's get to it. This episode is brought to you by bolt.new.

New the AI app builder for people who have ideas and want to ship them. Most AI tools spit out code that looks great in a demo and falls apart the second [music] you try to do anything real with it. Or they lock you into their own platform with no real way out. Bolt is [music] different.

You describe what you want to build. A startup MVP, a landing page, an internal tool, a side project, and Bolt generates productionready code in minutes. Connect Stripe or any other MCP. Hook up your domain and deploy it live.

Founders are using Bolt to build businesses doing real [music] revenue. Product managers are shipping prototypes their teams actually use. Designers and marketers are launching campaigns without waiting in line. Anyone can build.

Engineering can ship. Everyone [music] wins. You just need an idea and a weekend. Check it out at bolt.new/howi aai.

Before I get into how to build a harness, let's talk about what a harness is. And I am going to make it as simple as I can for all of you. A harness is some code around an AI agent. Yes, you heard it here first.

A harness is just code around an AI agent that makes it more effective. Can that code have AI in it? Sure. Does that code have to have AI in it?

Not necessarily. What is the goal of a harness? to make the AI better. It is so simple and I feel like the way that people have been talking about this have made it such a mystery that I wanted to make it just very clear to you all.

It is just writing more code around your AI to make it more useful for a specific use case. So what are the parts of a harness? Well, a harness is going to have specific context. It's going to be able to take specific actions and it's going to have a goal of specific outcomes.

It's just as simple as that. And I want to talk about when it makes sense to build a harness and when it doesn't. And I think you'll want to build a harness when the same workflow needs the same setup and the same outcomes. And so it's kind of similar to when you would build an AI agent.

And in fact, harness agent sometimes you can interchange some of these concepts but really it's when there is a sort of combination of deterministic and non-deterministic workflow stepbystep process tools use cases you want your AI to follow up to do a specific job. Usually those jobs are like slightly more complex and this is why you've seen these coding harnesses come out like coding is a job to be done. It needs specific tools. that typically goes through kind of a standard workflow and so coding harnesses are very popular.

But you could also do things like managing production incidents where you need to go through a specific process, getting PRs ready for release, um handling support escalations, managing migrations, even non-technical use cases like doing research in a very specific way or consolidating docs in a very specific way. That's how you and why you would use a harness. So, how did I decide what kind of harness I would build? Well, I looked across my business at Chat Purity and I thought, what am I doing sort of repeatedly and consistently that I think AI could be good at that I think we could be doing better if we were more structured about the AI and how we used it.

And I thought that fixing bugs, you all, if you've listened to this podcast, look, I ship code, so I ship bugs. Fixing bugs is a very specific workflow where we've built some custom internal tools that I've been generally doing with cloud code or codecs, but I had us had this hypothesis that I could do a better job of triaging bugs if I built my own harness. And so I picked sentry debugging and sorry for the claude slop content here. um sentry debugging and debugging sentry issues.

Really figuring out the issue using some of our custom internal tools and then doing all the follow-up actions we do when we close bugs was like a good first harness. It had coding in it. It needed custom content and custom context. There were like specific outcomes I wanted to make sure that we followed like tracking everything in linear and writing follow-up docs that the rest of the engineering team could use.

And so we chose uh debugging our Century bugs. By we I mean me and Codex chose debugging Sentry as a good use case to demonstrate how to build a hardest. Now why wouldn't I just use an AI coding tool directly? Well, I have been using AI coding tools directly.

And I think the problem with using a generalurpose coding tool and why we've seen people build these specific use case harnesses is sometimes with a specific job you just want to micromanage a little bit. You just want to be more prescriptive about how that job gets done. And so if you can identify the right workflows you can actually be more efficient, more consistent and that have better outcomes if you build a harness. So for this specific use case, you know, with a direct AI tool like Cloud Code, um I would have to explain what I want the the agent to do.

So I have to say like, "Dear agent, please fix this bug. Here it is inside the link. Instead, this harness, I can literally just paste in the link and the agent already knows my intent, already knows what the job to be done." A second thing that I wasn't that worried about but is interesting when you build your harness is you can be really prescriptive about what tools it's allowed to do and what it's allowed to execute and not. So for example, if you wanted to build an investigate only harness, you could make sure that your harness, your code editor never actually wrote code.

It only explored and explained root cause. You can also repeat the same process over time if you encode it in a harness. And so if you want like a very precise step-by-step flow including outcomes. So for us, every time we fixed a set tree bug, we want it documented in linear.

We want a very specific report. We might even want to follow up with customers that it was impacted with. You could encode that in a skill, but then again, you have to babysit it. When we built this harness, we knew it would happen every time.

And then from a model perspective, you can do multimodel routing and all sorts of interesting things in ways that you couldn't with a general purpose AI model. So I'm going to show you how it works and then we will talk about how I built it. Okay. So the interface I built for my harness is a terminal UI again like quad code or codec, something you run in your eye in a UI.

And just so you know, your harness does not have to be a TUI. It doesn't have to be a CLI. It doesn't even have to have letters. It could be a web app.

I did it in a TUI. One, because I haven't built one in a while. I thought it would be fun. And two, I just want to show that building your own custom harness means you can build your own custom interface into these AI agents as well.

So the harness is the whole experience, including the human experience that makes it more useful and easier to use. And so um this TUI is pretty easy to invoke. I just run TUI. You can see it here.

It's kind of cute. It's been made cute. Um, I use this library called Ink, which helps you make cute to UIs. I don't think they would say cute, but I'm going to say cute.

And you can see here that this terminal UI really reflects the structure of the harness itself. So, you see all the runs um that it's done so far, errors, and how it's fixed things, and then sort of our harness process, which is it gathers evidence, it streams in activities, and then it builds some artifacts. And so I'm going to actually have it investigate this sentry error over here. It's one where our edit um operations are getting dropped sometime by the agents.

And that has now kicked off our specific harness. So what it's going to do is it's going to start this investigation run. It's going to kick off a clawed SDK session which is a fundamental part of how I built this. It's going to go ahead and start gathering evidence and coming up with a root cause hypothesis of what's causing this issue and how we might fix it.

Now, as you can see, I chose I investigate, not fix. So, the investigation should not touch and modify files. And again, this is something that I would have had to like prompt to the agent and say, I only want you to investigate. I do not want you to ship a fix.

but instead I can just click I paste in that Sentry issue and it's off to the races. This episode is brought to you by Customer IO. You're here because you'd rather use AI than talk about it. With Customer IO, you describe the campaign you want to build, and the AI agent creates it for you, the audience, the messages, and the timing.

You review it, make any changes you want, and launch. Instead of spending hours stitching together tools and workflows, you can focus on the work that actually drives growth. Every campaign is tied back to results so you can see what's working and what to do [music] next. More than 9,000 brands use customer.io to turn the data they already have into messages customers remember.

Visit customer.io/howi to try it today. customer io more impact from every message. While this is running, I'm going to just go and show you a little bit about how this works and how I have actually built it. Okay, so this is the high-level architecture of the app.

So the front end is a terminal UI or a C cli invocation of the harness we call a run. So it's running a task. Each task has a specific input. Usually that's a century issue.

And then there are specific flags I put on the harness that allow it to edit the source, modify um the inputs or even message customers only if I flag and approve it. So again, this is just a little bit more control over how the agent works. The harness core is run on claude agent SDK and so all the agentic planning is run through the claude agent SDK which has some of the primitives of claude code in including grapping files and writing files and all those sorts of things that we find useful. And then what's really interesting about this harness and you've seen in other harnesses like openclaw is it can create its own artifacts in its file store.

And so we have this artifact store. I will show it to you in a minute. And it basically saves all the evidence from these runs to the file system for the agent to use in the future. And then it's connected to real tools.

So it's connected to Sentry V for cell the cloud SDK. Um it's running Sonnet 46 because I think that's the right the right model for the job. And then it's connected to linear and GitHub in terms of getting tasks done. Now what's really interesting as well is you can prompt this in a custom way.

So instead of the general like you are claude code, make no mistakes, you are our, you know, sort of model genius, I'm saying specifically that you're working inside the chat engineering harness. It's chatur specific. It's not an open-ended coding system. We want to use these artifacts as the source of truth.

And here's the plan to attack a very specific problem. And what I want you to return is X, Y, and Z. And again, I don't have to copy and paste this. So I don't even have to put it in a skill where hopefully it will get invoked in the right way.

I've actually encoded this in a very specific step in the harness to make sure that the model falls it at every time. And so there's several of these types of custom prompts inside my harness. There is um the artifacts that get generated. There are tool policies around like what tools can be called and which ones can't.

And then um I have just decided again to use cloud sonet 46 um which is really I think the right model for this particular workflow. Okay, I want to talk a little bit about the code and how you generate this and then just like a peak behind the scenes. I actually ran dueling claude code and codec sessions and essentially said like help me build a harness. I think I want to use the claude agent SDK.

Here's what I would like it to do. and then like closed my eyes and tried to get it done. Honestly, it was not a oneshot. I don't know if it was my prompting or the models were being funky.

It was GPT 5.5 and Opus, but both of them really wanted to build something super deterministic. So, they like really resisted putting any AI in the harness and I had to really prompt it very very specifically to get what I want. So I would say if you were trying to do this, I would be very specific about the workflow. I would be very specific about the tools.

I would be very specific about where custom prompts make sense. And then I would suggest using an agent SDK either from Claude or from OpenAI to run most of it because without that prompting, I just did not get what I wanted out of these models. The second thing I will say funnily enough codeex did the best job of building the agent but it used claude agents SDK to actually implement the agent. So we are spanning across models and spanning across coding agents here.

But the actual harness itself is pretty simple. It's got sort of a highle index of how you get to the TUI and then it's got like I don't know eight files of specific things it can do. So it can hunt for bugs in Sentry. Um it has a Sentry adapter to effectively use the Sentry API in a very specific way.

So instead of using the MCP generally instead of like having your coding agent wander through all these traces, I'm just very precise about exactly what I think you need to pull from a bug report perspective, what's useful, what's not, and made that connector really opinionated. It's got similar a linear integration, a Versell integration, a GitHub integration. And so again, not like generally how you can use these tools, but specifically how you would use these tools when you are searching for a bug. And then after those tools and data sources are used, the bug is identified and triaged, then there is this artifact file here that outputs and spits out the specific artifacts I want to see after a bug run is done.

And that artifact bundle looks something like this. So, it's literally just uh the task run um which is all the messages, the reports, so what was the sentry issue? Here's a brief on what we discovered. Here's any logs that we think are relevant, what the claud worker ended up doing, and then the summary of the output.

And then we also output this beautiful HTML file um that I can show you that shows you what happened and how it all worked as well as a worker report. So, I will show you those outcomes as well. Just pulling up this code for you. Again, it's pretty straightforward.

It's giving me all the instructions on where to put my specific API keys. And then I can just run it in this very opinionated way. So, in addition to running the TUI, which lets me sort of like navigate through the UI and use this harness, something I might want to do as a human, it also has built these really easy command line tools where if I just quickly want to run this harness against specific issues with specific flags on tool use, I can definitely do that. And what's kind of interesting about this is yes, I built this harness and you can see here I built this like fun UI so that I could use it in a fun way and it makes for a better demo, but really this harness is a structured way to give agents the job of running these investigations on an on a simpler basis.

And so you can imagine while I designed the TUI for human actually giving a kind of all intelligent agent a specific harness to solve a specific problem with agents in that I think that's how you're going to get real leverage and really custom outcomes out of things like coding agents like cloud code. And so going through this process has really opened my mind to we've gotten so used to like the open chat field like if I just type in the agent will do good work and I think we all have done good work but then now I've realized that these agents can help us solve very very specific problems using other agents and by constraining that work we can actually get specific jobs done really efficiently and then use the general purpose agent to sort of orchestrate it. So, it's really changed my mind about how uh work gets done. As you can see here, again, it's just a couple files.

It's really not too much. the adapters to the data sources. Um, a couple workflows, in particular, this bug hunter workflow, which just goes through exactly how we want to hunt bugs, including how we want to put together summaries of bug reports, and then some files here in terms of running the TUI or the CLI. And then, as I said, we have this artifacts folder that gets updated every time a run happens where I can click in and actually see exactly what happened out of a run.

So, let's go and see if this run happened well and what I can find out. So, now I have the full context. Here's the investigation brief and I can go look for it. So, this is bug hunter C7.

Let's see if I can find this one. Here it is. Here's the investigation brief on that edit document. Operations dropped.

I have confirmed evidence. So, it's saying yes, there was definitely a sentry warning. It's impacted 150 users. It's still happening hourly.

Um, it's a warning, so it's not an actual error. And the versel logs were unavailable and so we weren't able to use that data. And then it found likely root causes. So, invalid original range or overlapping original range.

And so it's identified a couple potential root causes as well as a blind spot in this particular function. It's told me exactly where in the product surface um the issue is and then how I would actually verify this by fetching a raw sentry event to see if the issues that they've identified are correct. it's identifying should it um issue a linear issue and it says yes we should definitely make a linear issue to fix this and so this should get assigned to somebody and then it doesn't recommend turning on patch mode and actually fixing this. So again, this is like a very specific outcome.

I wanted I wanted to say like what's all the evidence priority rank the root causes make a suggestion on the next step if we need to verify this more tell me if I need to assign it to somebody in linear and then tell me if you can fix it and they're saying no I don't think I can fix it yet I need a little bit more information and all of that is built because I have done this like very specific workflow and encoded that in what we're calling a harness which is just code around an agent. So, how would you build your own harness? I feel like hopefully you're still with me. Not too much of that went over your head.

Just to reiterate, I just identified a specific workflow. I determined what the run against the task would look like. I made very opinionated calls to tools or data sources. So, I didn't just say like use an MCP, although that could be part of your harness, that what I did is I made adapters that made the calls to these external APIs and tools very specific.

I thought about what the structured artifacts out of that workflow might be. I decided what rules and permissions I wanted to give this harness and which ones I didn't. I decided whether I wanted to use cloud code or codecs or a model router to actually run these things. And then I built a surface to interact with this agent.

So I built a TUI so I could actually look and work with this harness in a way. It could be a TUI. It could be a CLI. It could be a web app.

But I built some way to interact with this. So this is what you need to do. Identify a workflow. Uh really write it down on you know proverbial paper HTML or markdown.

figure out what sources of data you want and then plug it all into cloud code or into codec as I did and have it build your own harness and then test it against real data. So that's it. I just I really hope that you walk away from this realizing that these mystery terms like harness are not that mysterious. A harness is simply putting some structure around how AI works.

Yes, cursor is like a really complex harness. Yes, codec and cloud code are very complex coding harnesses, but at the end of the day, they are code that wraps these AI agents and these AI calls to make them more efficient at doing a very specific job. And so whether you're doing that in a very prescriptive way like I just showed where I want to show you how I triage sentry bugs, do the investigation and pass it on to the team or you're doing it a broad way like these general purpose coding agents that just have access to tools and context and methods that make the coding workflow better. That's all a harness is.

You can think of harnesses that you can build. You can build them in the terminal. You can build them for CLIs. You can even build them as web apps.

I'm starting to hypothesize that a wrapper is just a harness and that is going to upgrade everything that I've vibe coded over the last three years. This has been totally a learning experience for me here on how I AI. This is my very first harness that I've built live on the show. I hope it's useful for you.

And if you're interested in me building other things and demystifying AI terms, let me know in the comments. Thanks for joining How I AI. AI. AI.

Thanks so much for watching. If you enjoyed this show, please like and subscribe here on YouTube, or even better, leave us a comment with your thoughts. You can also find this podcast on Apple Podcasts, Spotify, or your favorite podcast app. Please consider leaving us a rating and review, which will help others find the show.

You can see all our episodes and learn more about the show at howiaipod.com. See you next time.

Agent harness is just one of those terms that is so confusing to understand because of how broad and also specific the term harnessing actually is. A common rhetoric out there is that harness refers to an environment for the agent, but that still doesn't really help us understand what it is and what it isn't. How exactly is harness engineering different from prompt engineering and context engineering? Welcome to Kayla Bright's Code where every second counts.

Quick shout out to Cursor, more on them later. To put it simply, harness engineering actually existed before the term harness was coined around early 2026. Shortly after the release of ChatGPT in 2022, we were dealing mostly in the context window of 4,000 tokens. And this small context window really limited our ability to do anything substantial with it.

And we could build an agent around this, but what we found is that simple prompting ChatGPT to get what we want to do just wasn't good enough. So, the question started to emerge on how to recycle this small memory space to effectively do more with less. So, we quickly expanded from prompt to context engineering by using various techniques like tool calling, MCP, and rag for the purpose of managing the context window more efficiently. Tool calling allowed us to explore the repository and read only specific files that are relevant to the task at hand and creating actions externally.

MCP allowed vendor-specific features to be added on top of the model. And finally, rag allowed custom databases to be connected for an on-demand data to be available at any time. All these gave birth to a new era of agents, mostly in coding agents. Cursor, WinServe, Cline, Rue, and Aider are all examples of early players that adopted tool calling for context engineering.

And they were really good tools that got the job done. And while all of this work was undergoing, the underlying models actually evolved and context window started to grow as well, which meant that coding agents started to be able to do longer duration tasks. And that's exactly what we were seeing. People started to ask these coding agents to work on features and bug fixes at a bigger and bigger scope.

And suddenly, context engineering that autonomously loaded proper context and take necessary actions gave these coding agents more and more ability to work on more complex tasks. But even this had its own limits, because as the duration of tasks got longer and we asked the coding agent some incredibly long tasks like clone an entire website, a simple prompt engineering will give you a very sketchy website because it can only respond in one shot. And even with context engineering, the result you got was not that great given the huge scope of the task. Not because context engineering was necessarily bad, but because you have symptoms like this where the website will be partially finished and some buttons would just not work and features aren't really tested all the way through.

And one major issue with context engineering was that we typically use context summarization to essentially keep shrinking down the context as it was topping up. So, if the task that we gave to an agent took, let's say, 12 hours, as the context window started to fill up, it would summarize its context to shrink it and continue working on them without having to run out of context window. So, effectively, the agent was bound by its own ability to properly summarize its previous work. And that's why you see tasks that are either half completed or not even attempted at all.

If context started to fill up mid-task, it would summarize and assume the task is already finished in some instances or oversimplify the task and assume some features are completed and verified when it really wasn't. So, as much as this elastic way of self-managing the context window gave the appearance of ability to work on longer-range tasks, it really wasn't all that effective. Now, this narrative I'm putting out here is an overview of what happened in the past, Meaning, people had been experimenting with different ways to get around this problem by implementing sub agents for hierarchical context management or even swarms of agents where you deploy multiple agents with their own context window. So, we were already converging towards a point of harnessing the underlying agents.

And as you can see, having a better orchestration layer and having a better execution environment and better context management are all ingredients that we needed to master for harnessing the agent. And this is when the concept of harnessing an agent or agent harness started to emerge and officially coined the term in early 2026. And while you can certainly make the argument that harnessing is a buzzword, it does capture the essence of something transformative that was happening in the AI industry. So, the question is, how is harnessing an agent really different from what we've seen before?

But first, a quick word from Cursor. I'm always trying to build on my websites, but not only do I have multiple devices that I have to keep track of, but I also want to keep working on them on my browser or my phone without having to set up the entire project on my devices. I use Cursor for that reason. For example, I can see that the models tab on my website is already behind in information since OpenAI released newer models since.

So, I can just spin up Cursor locally to keep this up to date. And while the agent is working on that, I can concurrently fix different features at the same time, spawning multiple agents with their own context as needed. Pretty cool, but I want to raise the bar. With cloud agents, I can actually have this entire thing run on cloud instead of my desktop, which means I can just close Cursor and the job will continue on without my machine and create a pull request once it's done.

Pretty cool, but I want to raise the bar again. I can integrate with Slack to send my feature requests to Cursor and it'll also run the cloud agent to get the job done and ping me once it's done with a PR. Pretty cool, but now I want to raise the bar again. Now I want to take this website and somewhat run it autonomously because I don't want to manually check for new information, I can add automation in the cloud to check daily for new model releases, and Cursor now keeps my website up-to-date autonomously.

One of the most critical changes that happened with the rise of harness engineering was the idea of loops. By stepping away one layer above context engineering and essentially looping the agents in a loop where at each iteration they have a fresh clean set of context but under a strict rule of how the agent should start and finish its task, we started to see an incredible result by putting the agent under this very environment. One of the primary example is Raw, which took over the internet given how effective it was, but more importantly, just how simple the architecture was underneath. One clarification to be made here is that harness engineering doesn't necessarily deprecate context engineering and it certainly doesn't deprecate prompt engineering.

If you peek below open-source coding agents like Klein, you see that their system prompt is still largely driven by a well-written prompt. So, prompt engineering is still used but a much smaller component in comparison to the system as whole. So, prompt engineering reminds the coding agents who the agent is and gives them the persona of a coding agents. In the layer above that is context management and context engineering.

So, harness engineering effectively leverages both prompt and context engineering. It's a shift away from relying on these two approaches but a paradigm change on the environment that puts the agent into series of steps where typically you start by generating a large requirement file and then looping each task and selecting only one task to be completed from the document. And it tests and documents each steps and this loop continues on iteration after iteration until the entire step is finished. And at each iteration they're given a fresh set of prompt and fresh set of context.

You see this kind of architecture mirrored in Raw's documentation as well. Where first starts with creating a production requirement document, which gets outlined into a JSON file, and it goes into a loop implementing feature after feature until completion. And you can see just how simple this entire architecture is when you look at how small the repository really is. Same thing for Anthropic's simple demonstration of harnessing when we look at their repository.

Similar story here, lightweight and simple environment. In fact, many coding agents now have already adopted this harnessing layer directly inside the application. Although each of them implemented their own way of harnessing their agents. That's why you're seeing so many companies talking about harnessing layer these days because of how effective it really is.

Hi everyone. My name is Audit, pronounced Audit like a tax audit. Um and I'm here to talk to you about what if the harness mattered more than the model? A little bit about me first.

I'm a staff engineer at Etsy. I'm also Etsy's IC initiative lead for agent of commerce. I wrote Grokking Algorithms, which is an illustrated book on algorithms. Uh and I also do illustrated posts on ducktyped.org.

Um Um Um and I want to talk to you about what if the harness mattered more than the model? By which I mean what if we should be paying more attention to building expertise in harnesses? Um Um Um and my controversial take here is I hear a lot of people say, "Oh, the models are so good that you can just keep the harness simple. Just give the model a few tools and it'll do the rest." And I've been hearing this so often that it has now uh started becoming the wisdom of the tech industry.

Uh Uh Uh to which my answer is, "Yes, but that's moving in the wrong direction because that is making us reliant on fancy proprietary models that can't be run run run locally. So, what if we instead talked about the other direction? What if we focused on open-source models that can be run locally? What if we focused on building a harness that is so good that we can get the performance of a cutting-edge model through a local open-source model?

So, what if the harness mattered more than the model? What if our focus should be on the harness? Um how much does a harness matter? Right?

This is something I have been researching a lot in my spare time. Um Um Um there's a really interesting paper that talks about Harness Bench, which is a benchmark for harnesses, which is something we haven't seen a lot of yet. Um Um Um it's got 106 tasks. Um it's testing these different models in different harnesses.

Uh but the point is that it's running the same setup. It's doing the same evaluation, same model, but testing different harnesses. Um and the results are pretty interesting. So, scores range from 52.4% to 76.2%.

So, more than a 20-point difference, and only the harness changed. Um and the really interesting thing is that for weaker models, the harness matters more. Um if the harness matters more than the model, that's great news. Uh because if the model matters more, then we're dependent on a handful of companies who are able to build and train these models.

Uh but if the if a good harness can compensate, and can make a weaker model perform better, then you can build our own harnesses, and that's something that any of us can do, do, do, and we don't have to depend on paid models. Um so, I have been spending a lot of time and doing a lot of research specifically into what would it take to build a really good harness. Um Um Um and and and I realized early on that none of the existing tools or frameworks of were able to do what I wanted. And I realized that what was actually needed was a language-level solution.

So, this is my other controversial take in this talk is I think building a good harness actually requires language-level support. Um and this is kind of where my research has gone in the last 6 months and I've spent a a a good part of that 6 months building a new language called Agency, which is a language for building agents. Um Um Um and just to talk about the terminology there real quick because the terms agent and harness aren't super crisp in the industry yet. They're not super well defined.

Um but in this talk I'll kind of use them interchangeably. But when I talk about the agent, I really mean the harness and the model. So if you take Claude code for example, Claude code is an agent. It uses Opus, which is the model, and then everything else is the harness.

Um Um Um but you know, when we talk about Agency, it's really for building agents, but the thing it's providing is the harness. Um Um Um Agency is really good. Uh you know, it's still new. Building a language is a lot of work and it's only 6 months in.

Uh but I love this quote by Alan Kay. Simple things should be simple, complex things should be possible. And that's something I love about Agency is that I think you know, when people build agents, there are a lot of primitives, a lot of things that every agent will need to be able to do and those things should be simple to do. Um and there's a lot of complex things that we want to be able to do with agents and it should be possible to do those things in the framework of your choice and with Agency it is possible.

Um Um Um So now we have a clear goal, which is try to build the best possible harness. And we have a great tool to do it, which is agency. Uh so, the rest of this talk I'm going to talk about building a coding agent in agency. And we'll see the same agent evolve over seven different examples.

We'll see the same model, the same task, but the harness will improve each time, and we'll see the harness improve the agent's performance each time. Uh Uh Uh So, you know, like I said, I've been doing a ton of research on this. My goals with this talk are to give you like the basics of building a harness. When we talk about building a good harness, what does that mean?

You know, you're starting from zero, what are the basics? Um Um Um I'll cover a good bit about agent safety because, you know, when we talk about building agents, what you really want them to be able to do is take action on our behalf. Um But the big problem with that is how do you get how do you give agent capabilities so they can take actions, but but but not so much capability that they do something unsafe. So, when we talk about agent quality and capability, agent safety goes hand in hand with that.

Um At the end, I'll do an introduction to some of the more advanced concepts like sub-agents and self-optimization. I think that's where, you know, more interesting things come in, and we really start to talk about like what are all the possibilities, you know, because when you talk about building a good harness, there are so many things to explore. And so, I hope you stick with me till the end there because that's where we get into some really interesting stuff. Um so this is the code we're going to ask our coding agent to edit.

This is a code to calculate the median of an ordered list of numbers. Um Um Um and if it's an odd length list, you just take the middle element. If it's an even length list, you need to take the middle two elements and take the mean of those two elements. But right now this function doesn't do that.

So there is a bug in this function. There's a test that shows the failure um and we're going to ask our agent to fix this code. So let's go ahead and start. Let's just dive in.

Um this is the first example. This is agency code now. And you'll notice that agency looks a lot like TypeScript. It's heavily inspired with by TypeScript with some Python syntax thrown in.

Um so the entry point is this node main. I have a simple prompt. The Python function median in demo/median.py has a bug. Please fix the bug.

And I just passed that prompt into the LLM function. I get the result and I print the result. Um Um Um And of course this isn't going to work because it can't read or write to this file. So let's run this example real quick.

Okay. So of course as expected it says you know, in order to help me fix the bug I'll need to see the existing code. Um Um Um so this is just the model. Very little harness and of course it can't do anything.

So, the most obvious improvement to any harness is give it some tools. So, So, So, real quick, let's talk about how tools work in the agency cuz it's very simple. Um I'll define a function. In this case, just for an example, I'm going to define a greet function that greets a user.

And then I'll make an LLM call that says, "Use your greet tool to greet audit." And then pass in that tool. Um Um Um and every function can automatically be used as a tool in agency. So, there is nothing else to do. Uh agency creates a JSON schema out of this function and sends it to the LLM call.

The doc string here will get used as a tool description. So, let's go ahead and run the second example. So, NPX agency Oops, actually. NPX agency example since actually 01 cuz we're going to talk about tools first.

So, howdy audit? And how can we know that it ran that tool and then just generate an example, right? So, I'll just also show real quick. Um Um Um agency writes out logs and has this great log viewer built in.

So, I'll just expand this last run and you can see here it's making this tool call to the greet function with these parameters. The tool responds by returning howdy audit, and the assistant just passes that straight back to us. Um So, So, So, that's tools. Now, let's look at using the tools in Agency.

So, again, this is you know, the next step of our agent, which is give it tools. Um Um Um here is an example where we're giving the agent read and write tools. Again, read and write are just functions that are built into Agency, and because every function can be used as a tool, I can just pass this straight to the LLM. Uh except giving the agent the ability to read and write arbitrary files on our file system is really unsafe.

So, by default Agency won't allow you to do this. Agency has a human-in-the-loop feature that would require some sort of approval. We're not providing it anywhere, so this code will crash and print an error. So, let me show you what that looks like.

Great. So, interrupt standard read was not handled. Your interrupt message, "Are you sure you want to read this file?" You can see the file name it's trying to read, and it's basically telling us that we didn't approve this interrupt. And it's giving us a pointer to the guide, right?

Tells us how to use handlers. Um Um Um so so so we started with just a model, we tried tools, and the next step is to make the tools safe. So, we're going to do exactly what that message asked us to do and use a handler. So, now, here's the same code that you know, passing through that read and write uh those read and write functions as tools.

Um but now it's wrapped in a handle block. Uh and it has a handler function down here. So, now when those tools call or raise an interrupt, this code is going to execute. So, it'll print the information about that interrupt.

Um, Um, Um, it'll ask the user for input using the built-in input function. And if the user approves, the tool call is approved. And I know I'm saying interrupt, and I'm not really defining what that means. Um, and I can get to that a little bit later, but it's essentially a way to pause execution at some point.

Um, and ask for human input. And so, interrupts are a great feature in agency, and all the functions in the agency standard library that mutate code or mutate something, or have some sort of destructive action, or read sensitive data data, all all all throw an interrupt first. So, So, So, here's the safe version of our agent where the tools will raise an interrupt by default, and now this code is going to ask the user, do you approve? Let's run this example.

Okay, are you sure you want to read this file? I'll just say yes. You can see it's reading. Now it's trying to write to that file.

So, you know, that's an example of >> [snorts] >> us making the agent safer by asking for user input. And this is better, and it is safe, but it's also very slow. So, the next thing we want to do is make the agent autonomous, while still keeping it safe, and that's the tricky part, right? How do you give it it it just enough capability so that you don't have to manually approve every single action without giving it without giving it so much capability that it's like able to do something destructive with your file system.

Um so agency has a really cool feature to handle this. It that's called partial function application. >> [clears throat] >> Um it's a concept borrowed from functional programming. It's a really fancy term for a pretty simple concept.

So, here's a signature of the read tool, right? Takes a file name and a directory. Um and when you read a file, it's going to look inside that directory for that file name. And with partial function application, what we do is we take that read function and call the partial on it.

And give it a directory. And what we're doing here is we're locking the directory argument to demo. Um Um Um the LLM isn't going to be able to change that argument. It's not even going to know that that argument exists.

When we pass in the read tool now, it's just going to see one parameter, which is file name. It's going to pass in a file name. And it will read that file from this directory. So, we've locked the directory parameter.

Now, the agent can only read files from and write files to this directory. Uh Uh Uh partial function application is a really great way to constrain the capabilities of your agent. Now, no human input is needed, but it's still safe. So, let's see that in So, again, run NPX agency examples.

PFA is partial function application. So, here it's thinking and you can see it read the file. Didn't need to ask us for permission. And now it said, you know, to fix this issue, it's kind of giving us a new code.

But it's not updating the code. It asks, "Would you like me to update the file with this corrected function?" But it didn't do that itself. So, you know, this version of this agent is the best one we've seen so far because it does give us the correct solution without any human input needed, but it hasn't actually fixed the code. So, the next step is to get it to actually fix the code.

Um and that's where the feedback loop comes in. Um so, there's a very popular agent pattern called react. And react stands for reason and act. And essentially, you're asking the agent to work in this loop.

Where you reason, act, observe the consequences of your action, and then decide what to do next. So, in this case, reason, read the two files, act, run the tests, observe, if the tests fail, read the error, and then reason again. So, look at that test failure and think about what you want to do next. And just do this in a loop until the tests pass.

So, this is a really key pattern to making the agent better. And so, now let's see the same agent. The only thing we're changing is the prompt and giving it the ability to run these tests and now the agent should work better. In this case, I'm going to also print all the tool calls just so you can see what's happening.

Um Well, should maybe All right, hang on. It's always there. There we go. Okay.

Now, let's see it in action. Um Um Um So, now you can see it's calling the tools. It's reading those two files, like we said. It's running the tests.

The tests fail. It's writing to the file. It's running the test again and confirming success. So, this is a really important stage in the harness development because now now now we finally have an agent that successfully figures out the task, checks that the code is failing, modifies the code, and checks that the code now passes.

Um Um Um So, let's talk about how we have improved the harness so far. So, we started with just the model, couldn't do anything. Then we added tools, but the agent wasn't safe. Then we added handlers, which added safety, but needed human input.

Then we added partial function application, which was safety without requiring human input. And finally, the last one that we just looked at was a feedback loop, which added reasoning. So, the agent made the change and confirmed that the change worked. So, I have two more examples.

This one we're going to go in a slightly different direction. And this is where we're getting into some of the more advanced concepts. So, this one is sub-agents. Um everything so far we've done has made the same task better.

But with sub-agents, we're asking, "How do you make the agent do more things?" Um Um Um and let me show you what this code looks like real quick. Okay. So, this program, I'll start by just showing you the prompt we're giving it because it's slightly different. So, fix the failing test in test median.py.

Then research Jensen's inequality for medians with the Wikipedia agent and explain it to me in the network form. So, So, So, here is a case where we're actually asking the agent to do a little more, right? And this one is interesting because this is our main agent, but for tools, we're not giving it the read and write functions. Instead, we're giving it two sub-agents as tools.

And And And I really like this pattern in agency because I've seen a lot of frameworks make sub-agents their own concept and make them kind of hard to grasp. Um in agency, a sub-agent is just another function. And you just call it, use it like a tool just like you would anywhere else. And that consistency has a lot of benefits.

Makes it really easy to write and reason about. So, here's the coding agent sub agent. Um Um Um same one as before. You know, there's a system message.

This sub agent has its own LLM call, gets its own set of tools. >> [snorts] >> Similarly, you have the Wikipedia agent that has its own tool, which is the search tool. Uh which is a tool built into this agency standard library that lets you search Wikipedia. Uh so, agency has a big standard library that lets you do all kinds of things, including this Wikipedia search tool.

I'm also using the callback feature to print the tool call information. And now, let's see this example in action. And this is an example where we're actually asking the agent to do a little bit more. And we're giving it two sub agents to accomplish its task.

So, here you can see it's calling those two sub agents. Uh it's calling them in parallel, so the Wikipedia agent is doing the search, the coding agent is modifying the file, and and and that fixed the error, and now here's a limerick. So, sub agents are cool because they let you add new capabilities without bloating context. Um Um Um and this is a good pattern because it allows your agent when it has a lot of different tools at its disposal, it allows your agent to do the right thing more consistently.

Often when agents get confused and fail, it's because they have too much context blow, but also because they have too many unrelated concepts in their context. And they have too many tools that are unrelated. And so they have a hard time picking the right tool to use. In this case, we make that problem a lot cleaner.

For the top-level agent, we just say, "Pick the sub-agent you want to call." And then the sub-agents have those tools that are kind of grouped in a way that makes sense. Um Um Um So now, let's look at the last example, which is self-optimization. Uh Agency has a Japa optimizer built in. It actually has a few optimizers built in, but the Japa one uh is a really interesting one.

And And And the way the optimization feature works is you mark any variables that you want the optimizer to optimize. You can just use this optimize modifier. And then you run the optimizer and give it a goal. So, So, So, you know, in this case, you can see that the two initial prompts I have are very basic.

I'm giving it hardly any information. And then the goal I give it is, "Fix the bug in median.py using TDD." And And And the reason this is such a great step for our harness is we're no longer doing trial and error. We're no longer just trying stuff to see what works. Instead, now we have a way to systematically measure and improve performance.

So, I'm just going to go ahead and run this code. Um I'm actually going to also reset. Here, I'm going to modify this code. Cool.

So now I won't show the full run here because the optimizer takes a while, but you can see first it's like checking what prompts it has that it needs to optimize. It's going to run the agent to establish a baseline. So the baseline objective is 0.2 and it's going to try to improve upon that. So in this case I did run the optimizer earlier just so I could show you what a full run looked like.

Um Um Um and so here is one where you can see it achieved the objective in the first iteration, so it was actually quite quick in this case. Um and you can see that it's rewritten this prompt. Um So the reason this self-optimization is so great is because we're not guessing and checking. We're systematically measuring and improving, which is a big leap forward.

So this is the ladder we just went through for improving the harness. Um Um Um you know, first step was nothing, just a model can't even read the file. Then we added tools, but in a way that was unsafe and not allowed in agency. Next we added handlers, which added safety, but made things slower because they required human input.

Then we added partial function application, which was safe and fast. Then we added reasoning, which gave us higher confidence in the output of the agent. Uh and then we switched tracks to subagents, added more capabilities, and finally we did self-optimization. So we got measured improvement and not guess and check.

So, So, So, you know, I hope this gives you an idea of of of how you can build harnesses and then slowly improve those harnesses. And I hope, you know, you try doing that and come along on this, you know, experimentation with me to see if we can build a better harness. Um Um Um because what of the harness matters more than the model. You know, what if we can build a harness that's good enough that we can now start using local models and still get the performance we need for the tasks we want to do.

So, what language level support does Agency provide? You know, we used Agency for all these examples. How did it help? Um simple syntax for defining tools.

Again, you know, the simple thing should be simple. Every agent is going to need tools. Let's make it easy to use them. Tools for safety, very important for agents.

So, interrupts handlers, PFA. True pause and resume execution. I didn't cover this a ton cuz I didn't want to make this talk too long. Um but one of the really killer features of Agency is that when you raise that interrupt, it's actually going to pause execution and return control back to the user.

But in a way where you can resume execution later. Uh and as far as I know, very few languages allow you to do something like this. And so, typically most frameworks you'll use, if they have a human in the loop feature, it has a lot of restrictions. There's a lot of things you need to do to work around that.

Agency is completely different. If you wanted, you could, you know, raise interrupts inside of a for loop, inside of a tool call, inside of a sub-agent. Um, and it would correctly pause execution, go back to the user, and then resume execution at that exact point inside that subagent, inside that tool call, at that exact point in that for loop. Um, Um, Um, which is really cool.

You can serialize the execution, you can come back to it a week later, and you don't need to change anything about all the code we have been looking at. It just works, which is really cool. Uh, it also has these built-in optimizers, which are great, you know, which kind of get you to a more analytical, mathematical way of doing the improvements. So, if you want to try it out, please do.

You can PM install All you need is the package and a API key, and you're set. Um, Um, Um, main takeaways, you know, an agent is a model plus a harness for this talk, at least. Uh, better harness equals better performance, especially for weaker models. And then, how do you build a better harness?

Give it tools, make it safe, make it autonomous, make it reason, give it subagents, and have it self-optimize. Thank you so much. Again, my name is Aditya Parikh, check out agency agencylang.com, uh, and then please follow me on Blue Sky, if you wish. Thank you.

Take care.

All software will be self-evolving software very soon. This is a new paper from a team out of Stanford, MIT, and Crafton, and it's called Meta Harness. End-to-end optimization of model harnesses. If you've been following the AI industry at all, if you've watched this channel at all recently, I have been talking about agentic harnesses a ton.

What is a harness? It is basically the traditional code that is wrapped around a model, like Claude, like GPT-4, like Gemini, that tells it how to operate, that allows it to store memories, search through text, write code, execute code, and so much more. It is what makes things like Claude code, Cursor, Factory, and so many other agentic harnesses so special. When you type in a prompt in Cursor, it can run for hours on end, and that's all because of the harness.

I'm going to explain all of this in detail, but now imagine if the harness itself can actually self-improve. Typically, they're written by hand, by a human. They're run and they're evolved over time, but they're not self-evolved. And let's pause there for a moment and talk about Andrej Karpathy's auto research project, which absolutely blew up in the last couple weeks.

Already at 61,000 stars. Basically, what it does is it allows a model like Claude to propose a series of experiments to run overnight, self-improving its ability to train a GPT-2 level model. If all of this sounds very foreign, just think of it like this. It's AI training itself.

It is looking at what worked, proposing new experiments on how to do those things better, and then continuously doing that all night long. And since this project came out, there have been a number of projects that are very similar-looking to essentially self-evolve, self-improve different pieces of software. And now, we actually have the harness itself that is able to self-improve. So cool.

Let's get into the details. So, here's the paper. They already published the paper, and they open-sourced the code, so you can use it right now. I'm going to drop all of these links in the description below.

All right, let me give you the gist of what we're going to talk about. The performance of large language model systems depends not only on model weights, but also on their harness. The harness being the code that determines what information to store, retrieve, and present to the model. The harness is just as important as the model weights.

Now, if this is confusing, let me just explain quickly. The model itself, when you hear GPT-5.4, when you hear Opus-4.6, we're just talking about the model weights. When the Frontier Labs do these huge pre-training runs, they're basically baking what eventually becomes the weights of the model. And the model weights are really, really good at predicting the next token or the next word in a series of words.

That is really all they're doing. But they become very special when you wrap them in a harness. When you say, "Hey, next time you're predicting the next word, maybe you should also look at the memory over here. Or maybe you should write some code and then execute it over there.

And I'll give you an environment for that. And you can save files over here." There's so much you can do with an agentic harness. And the models themselves, the weights, are incredibly powerful already. They are incredibly intelligent.

I've been saying this for a while. The models are good enough to reach AGI today. All we need to do is build out the harness. I used to call it scaffolding, but now it's called a harness suddenly, to make them really good.

So, then they introduce meta harness, which is an outer loop, so around the agentic harness system that searches over harness code for LLM applications. All right, let's get into the details of meta harness now. Changing the harness around a fixed large language model, so just the weights, can produce a 6x performance gap on the same benchmark. Harnesses are everything.

The models are so good at this point, but that's just raw intelligence. That's like the engine of a car, but the engine of a car is not enough to get you from point A to point B. You need seats, and you need a steering wheel, and you need a way to deliver all of that power that's coming out of the engine to the tires, and the tires to the road. So, the harness, the code that determines what to store, retrieve, and show to the model often matters as much as the model itself.

Yes, this is called harness engineering, and the best frontier labs are making their own harnesses, and there's a bunch of other companies that make their own harnesses that are fantastic. And I'll talk about a few of them today, but you've probably have heard about most of them. But, despite its importance, harness engineering remains largely manual. And you know what else needs to be self-evolving?

Your skills. And that's why I'm excited to tell you about the sponsor of today's video, HubSpot. I know a lot of you are trying to keep up with how fast AI is moving, and you want to actually learn about these tools, not just read about them. Claude CoWork just came out and lets you automate huge portions of your workload.

And so, I suggest you check out the Claude CoWork stack, 12 prompts to replace a week of work. This is the guide to understanding AI agents and how they actually work in practice. It covers the different ways to use Claude CoWork, and specific walkthroughs about how to get the most out of it. You'll understand how to give your Claude Co-work agent access to hundreds of files and to let it do deep analysis, pattern recognition, run autonomous tasks in the background, and all without having to babysit it.

You will get real deliverables from Claude Co-work like spreadsheets, presentations, and reports instead of just chat responses. I personally found the section about background processing and real file creation to be particularly helpful because background processing is what AI agents look like in 2026. So, this ebook was made by HubSpot. They're giving it to you for free.

Shout out to HubSpot once more. They've been a fantastic partner. So, go download this ebook. It is a great resource, plus they're great sponsor.

It helps the channel. Now, back to the video. And so, the authors of the paper asked, "Can harness engineering be automated?" And it turns out it can. So, check this out.

A natural starting point is recent work on text optimization. When they say starting point, they mean, "How can we experiment on a harness made for a kind of narrow use case?" And this is text optimization. Harness engineering also involves iteratively improving text and code artifacts using feedback from prior attempts. So, the actual harness itself is executing that learning pattern, but the harness is not doing it itself if that makes sense.

Let's keep going. So, these methods are poorly matched to harness engineering because they typically operate with short horizon or heavily compressed feedback. Some condition only on the current candidate, others rely primarily on scalar scores. So, just think, you know, zero to one scores.

But, a zero to one score is not really good for what they're describing here. It doesn't really work. The problem is harnesses are incredibly complex. They include so so different factors, so many different success signals that boiling them all down to a single score like 0.4, 0.8 makes it very difficult for us to understand how to improve or what parts of the harness we should improve based on that score.

So, harnesses act over long horizons. A single choice about what to store, when to retrieve it, or how to present it can affect behavior many reasoning steps later. Compressed feedback, so that means okay, let's figure out what happens, let's summarize it using a model and then feed that summary into this experimentation loop that we're trying to do, it doesn't really work. Compressed feedback often removes the information needed to trace downstream failures to earlier harness decisions.

Across the tasks studied by several representative text optimizers, the available context per optimization step ranges from only 100 to 30,000 tokens. Basically, they're trying to boil down everything that comes in a harness, which can be millions and millions and millions of tokens down to a few tens of thousands of tokens to try to figure out what happened. And obviously, you lose a lot of quality. You lose a lot of the signal doing that.

So, it's not a good approach. And here's an important concept we're going to go over a few times during this paper. Work on retrieval and memory augmented language models, meaning we have this large amount of memory, this large amount of data, what should I choose to give the model? That's all it means, suggest that useful context should often be accessed adaptively rather than monolithically packed into a single prompt.

And there are multiple reasons for that. But first, what does that mean? That means let the model decide what it needs. Don't try to figure out what it needs.

Don't try to pack everything into a single prompt up front and give it to the model. Just say, "Hey, here's is you have available. You choose." And it seems more often than not, that is the right approach. All right, so how does Meta harness actually work?

Well, it has a proposer and it is a coding agent, a language model based system that can invoke developer tools and modify code, basically a harness around a model. And the choice of using a coding agent versus a raw LLM was deliberate and makes a ton of sense. Let me show you. The choice of coding agent rather than raw LLM matters because the amount of experience quickly exceeds context limits.

Meaning of all the code necessary to write a harness, as I said earlier, you can't fit it all into the context window. So you have to give the model and the harness around the model the ability to choose which parts of the harness it's going to read and understand as part of its experiments. The proposer must decide what to inspect and validate edits through direct interaction with the code base. So this is very similar.

If you're doing any kind of agentic coding right now, you're probably familiar with how to visualize what I'm talking about right now. So for example, this is Cursor. I said, "Review the entire code base for code issues, user confusion, and so on." And what it did is it read the entire code base, but not all at once. It looked at the different files in the code base, it looked at the different methods, decided where it should gather more information, and that's what you're seeing here.

And that's exactly what the coding agent in Meta harness is doing. The proposer. And now we get the full history of all the harnesses. So imagine this, you have a harness, you test it, you make changes to it, and then you test it again.

And now you have two versions of the harness. And now imagine you do that 10 different times. And so if every harness is a million tokens and you do that 10 times, you now have to reason over 10 million tokens, which is not possible based on current LLM context windows. So, instead, we give it everything and we give it the tools to know how to look and find what it needs.

And it works exactly like that. So, for previous candidate harnesses, the file system stores the source code, evaluation scores, and execution traces, which the proposer retrieves via standard operations such as grep and cat, rather than ingesting them as a single prompt. So, it is very meta, as the name suggests. So, they ran a bunch of tests.

We evaluate Meta Harness on text classification, great, on retrieval augmented math reasoning, which I'll explain when we get there, and on terminal bench two, which is the benchmark testing how these harnesses operate terminals. All right, but Meta Harness didn't come out of nowhere. They built on top of a lot of ideas and concepts that are floating around the AI industry right now, and let me point those out. So, here's the related work.

So, first, external memory and adaptive access. Retrieval augmented generation, interleaved retrieval and reasoning, memory-based agents, or recursive language models are mechanisms for adaptive access to external context. And they leveraged a lot of these concepts in the Meta Harness. Executable code search.

Meta Harness searches over domain-specific harnesses, including prompt construction, retrieval, and state update strategies that reset between tasks. It's outer loop is deliberately minimal, which I really like. Minimal is good. Instead of relying on a fixed scaffold, an archive of prior discoveries, or a persistent memory mechanism, it gives the proposer unrestricted file system access to prior experience.

Again, let the model choose what it needs. It also gains inspiration from other systems that improve itself. One of the ones that I want to point out is Alpha Evolve by Google, which figured out ways to optimize Google's own systems architecture and actually figured out a faster way to do matrix multiplication, which was the first time we figured out a faster way in like 50 years. So, this self-improving loop is incredibly powerful.

So, how does it actually work? At a high level, it repeatedly proposes, evaluates, and logs new harnesses. Very similar to how auto research by Carpathy works, trying to figure out a faster way to build a GPT-2 model. The goal is simple, find the harness that makes the underlying model perform best on the target task distribution.

So, here's the loop. Meta harness uses a single coding agent proposer with access to a growing file system that serves as its feedback channel. Here, the coding agent is a language-based model system that can invoke developer tools and modify code. Meta harness delegates diagnosis and proposal to the coding agent itself.

It decides which prior artifact to inspect, which failure modes to address, and whether to make a local edit or a more substantial rewrite. Why did they do this? Why are they letting another harness try to figure out how to improve the harness itself? It is just so meta, and it's so recursive, and that's exactly why.

As the core models get better, the LLMs themselves, as the harnesses get better, as the meta harness gets better, these multiple layers of self-improving, recursively self-improving, I should say, factors stack on top of each other, the success rate and the velocity to success increases dramatically very quickly. So, each evaluated harness contributes a directory containing its source code, scores, and execution traces. So, that's stuff like prompts, tool calls, model outputs, and state updates. The proposer is free to inspect any prior harness, so it is not just told only inspect the best performing ones.

It can inspect any of them, even the low performing ones, because you can avoid local maxima that way and while it's proposing new ones. So, you give it a fixed number of iterations and perform a final test set evaluation on the Pareto frontier. This simplicity is deliberate. Listen to this.

By leaving diagnosis and edit decisions to the proposer, rather than hard coding such heuristics, meta harness can improve automatically as coding agents become more capable. So, the target for improvement can improve the improver, which then improves back the target for improvement. So, now meta harness reads everything. It's looking at the code, the prompts, the tool calls, the output, the terminal output, everything, and it's determining what should we change?

Should we make a little change? Should we make a big change? Then it puts forth a new harness, and then they test against that harness, and they continuously do that over time. And interestingly enough, I'm a big fan of Opus 4.6, and it seems like the authors of this paper are as well.

So, in our experiments, the proposer, they chose Claude Code with Opus 4.6. Now, personally, I use Cursor with Opus 4.6, but a lot of people I know are using Claude Code. As I mentioned earlier, they have three experiments that they're running. One on text classification, one on math reasoning, and one on agentic coding.

And they compete against two different things. One, human-designed strategies. So, if a human wrote the harness themselves, that would count there. And program search methods.

These methods search over candidate harnesses using feedback and reward signals, but are designed for smaller scale settings than harness engineering. So, basically kind of self-improving code, but not meant for this level of self-improving code. All right, so let's look at some of the initial results. So, first we have the online text classification benchmark.

And what that is meant to do is take text and exactly what it sounds like, classify it. And it's not just the model, the raw LLM that does that, it is also the harness around it. So, we have zero shot, so that literally just means put the text in the LLM and say, "How do you classify it?" And it takes zero context and does okay. All right, so for USPTO, which is the patent trade office, I believe, it gets a 12 on the symptoms to disease, which is a medical text benchmark, we have 63.2, still really good.

And then the law benchmark, a seven, which compared to the other results is very bad. We have few shot with eight examples, few shot with 32 examples, and few shot with all examples. And you can see they get a little bit better over time. So, the more examples you give it, the better it does.

We all knew that. This is early prompt engineering learnings. Next, we compare against MCE and ACE, which is basically kind of the next evolution of prompt engineering. MCE is meta context engineering and it allows the model to build and curate a library of natural language skills to help it with its task.

And then ACE, which is agentic context engineering, which basically allows the model to reflect on what it's learned over time. So, as we see for the USPTO, ACE actually did best. Although the meta harness down here did pretty well at 14. For the S2D, the meta harness did really well, going four points above the second place of MCE.

And then for law, it absolutely dominated. The highest score being a 29 with few shot and ACE, and the Meta Harness coming in at 45. A massive massive improvement. And the average overall, it is the best.

It scored a 48. Second place, 40.9 with ACE. Not only was it better, but it also used a far fewer tokens. Look at this.

This is the context usage. So, zero shot, few shot, few shot all the way up. ACE and MCE 28.5 and 50.8 on their context. 11.4 for the Meta Harness.

So, it is far cheaper and just as good if not better on most cases. So, Meta Harness matches the best prior text optimizers, Open Evolve, TTT, Discover with 10x fewer full evaluations, and its final accuracy surpasses theirs by more than 10 points. So, these are literal optimizers that were built for text classification, and it still beat it. And the nice thing about Meta Harness is it doesn't go in with any preconceived notions.

It just says, "Okay, let's start testing and see what works." A human doesn't have to curate the experiments or decide what to change in future experiments. It's all automated. And so, it's really just limited by how many tokens can we throw at it and how fast are those tokens. And so, let's look at one more thing cuz this is super interesting.

The Meta Harness median score against the other best in class harnesses, state-of-the-art harnesses for this task, it gets a 50 as compared to a 39.1. Now, the best score for all of the other ones is 45.6. The median score for the Meta Harness is higher than the best score from all the others. And of course, the best score for Meta Harness, 56.7.

So, you can see this is not just luck. The Meta Harness, the auto research, the self-improving way of doing things is the best. So, then they thought, "Okay, are we overfitting? Are we making the harness that is just so good at this one task and maybe not so good at most other tasks?" Well, as it turns out, no.

They actually took the harness that they built based on those three tasks and generalized it out to nine other data sets that the harness had never seen. And more often than not, it won. And on average, it was the highest score by three points, 73.1 versus the second highest score with Ace 70.2. And it was relatively inexpensive, definitely more expensive than few-shot, but less expensive than Ace.

And then they thought, "What if we applied this to mathematical reasoning? What if we put our meta harness up against the IMO, International Math Olympiads?" That is, mid last year, all of the frontier labs were competing in the IMO, showing that their models with a ton of harness around them can perform incredibly well in that competition. That is, the frontier of math, the hardest math problems given as a competition. And here's what they found.

The discovered meta harness retrieval strategy improves reasoning on these IMO level problems across all five held out models with a 4.7 point average gain over no retriever. But how does that even make sense? Why would retrieving past memory or past context help with a new math problem? Well, they explain.

There is a good reason to expect retrieval to help mathematical reasoning in principle because solutions often share reusable proof patterns. So, that's it. They said, "Well, what worked for the last one? Maybe we can use part of that proof for our current problem.

So, very interesting. All right, last and my favorite, Terminal Bench. This is the benchmark that tests a model or a harness's ability to interact with a terminal across long-horizon difficult tasks to be done via terminal. So, Terminal Bench 2 evaluates LLM agents on 89 challenging tasks that require long-horizon fully autonomous execution under complex dependencies and substantial knowledge.

They use this benchmark as a discovery problem. The goal to discover a harness configuration that improves performance on a hard publicly contested benchmark. So, look at all of these. You've probably heard of some of them.

Claude Code, obviously. Droid, one of my favorite. And so, what are the results? We have Claude Opus 4.6 and Claude Haiku 4.5.

These are the two core base models that they tested against. And remember, this is against the Terminal Bench benchmark. So, Meta Harness on Opus 4.6, this is the frontier model on the market today. It scored a 76.4, higher than every other benchmark that has been handwritten with the exception of Forge Code.

So, remember, Meta Harness wasn't really written at all. It was just evolved from the Meta Harness framework. Then, for Haiku 4.5, it did better than all of them, 37.6 versus second place Goose, 35.5. So, just letting the model figure out how to build its own harness seems to be the way.

And this seems to all tie back into something called the bitter lesson. I know it's not a direct analogy, but it feels close to me. And so, the bitter lesson shows that handwritten heuristics by humans, basically patterns described by humans, never beat just end-to-end neural networks. The AI itself learning those heuristics on the fly.

And so, the most prominent example of the bitter lesson is Tesla's full self-driving. For a long time, it used a combination of neural nets and handwritten code. If you see a stop sign, you better stop. But, over time, as the neural net got bigger and bigger, what they figured out and they swapped it all out for end-to-end neural nets one day is that those neural nets will figure out those heuristics itself.

And that's always the best way. AI figuring out what to do will always beat humans telling it what to do. Remember that. That's going to pop up in our lives a lot over the coming years.

And so, right now, the harnesses are written by humans, but should they be? Obviously not. If you're writing any code, if you're writing any automations yourself, why can't we extrapolate this to say all code should be self-improving? All code needs its own meta harness.

Why not? We've already seen a lot of examples. We've seen the founder of Shopify improve a code library significantly that hadn't been touched in years, and it just let Karpathy's auto research just iterate on it and figure out how to improve it, and it had massive gains from just letting AI write it and rewrite it and rewrite it. And so, a lot of the token usage in the future is going to come from just letting the tokens burn and iterating as much as we can on these problems.

So, I found this paper to be absolutely fascinating. And I really think self-evolving software, self-improving software, is going to be such a big presence in artificial intelligence in the coming years. We're already seeing it. We're starting to see the frontier labs put out models that were trained by previous models.

And we're going to start to see harnesses being built by previous harnesses. And then we're going to see all software just being built by previous software. And it is so awesome to watch. It is so fascinating.

And it is hard to keep up at times. But if you follow me here, I will help you keep up as best I can. If you enjoyed this video, please consider giving a like and subscribe and I'll see you in the next one.

All right, let's talk about AI agent harness in this video and why this is a core topic of why people feel some of the people feel like Opus 4.6 or 4.5, for example, is a great model or if it's like a really bad model or, you know, any of these models, right? And why basically all of us have some preferences. For example, some of you probably like cursor a lot more than Claude code, than Codex. You might have different preferences about GitHub Copilot and so on and so forth.

And I'll also tell you why and how these are fundamentally different things. So, the core idea is that how can a model like Opus 4.6, let's take example of this, how can this model actually be good as well as bad at once, right? This is a question. And this actually happens.

It's not like a philosophical question or something that is just that just has a perspective meaning. This actually can happen and the core reason behind that is harness. And harness is a new sort of thing that people have now started to say a lot more this year. I did not used to hear this a lot in the last year, but harness is the new thing now.

Let's actually break down what this AI harness is and how you can develop your own AI harness and why you should probably in the first place. So, see, an AI model like Opus 4.6, if you look at this model, this is just a raw intelligence in its raw form, right? If you look at Opus 4.6 API, for example, the API that Anthropic gives you, that is just an API, right? That is not harness.

That is not doing anything special, any magic by default, like if you're not using any tools or anything. That's just pure API in the first place. Now, what happens is that LLMs, which is this thing over here, a large language model, a large language model is generally not very useful if you're just chatting with it, right? What you want to do is you want to give it tools.

What you also want to do is you want to make sure that the context inside the LLM is not exceeding because if you give it too many tools, for example, let's say if you are giving it the ability to browse a browser, if you are giving it the ability to make edits to your code, you know, run an arbitrary bash commands. At some point, you will run out of tools, not because obviously you can just keep on adding more and more tools, but because the context would be too large, right? So, you want context management as well. For example, Opus 4.6 has a maximum context length of 1 million, and that is also like, you know, it's something that you would probably not want to do because the performance start to get impacted at this point, right?

So, you want to probably keep it under 500k, 300k, so on and so forth. So, context management is another task that, you know, you might want to do. Third is of course safety features. For example, you know, I don't know if you have seen that, but in Claude code, there are a bunch of modes which are possible, right?

First is like if you just start Claude directly, it'll just start like that. Then you have Claude dangerously skip permissions. You know, this is sort of a mode where Claude would not ask you anything, and then there's a Claude auto mode as well, which basically takes its own guess on, you know, whether it's what it's doing is right or not. Now, those sort of things also, you want to probably incorporate in your, um, you know, whatever you're doing, whatever real-world task you're doing.

So, we're not getting to harness right now, but this is something that makes sense, right? That you might want to have. For example, it's completely fine for you to run DB queries because, you know, you have given only read-only DB access, right? So, for somebody who's creating a custom harness, let's say, they can just give it a tool which has DB query access without any permission, but for somebody else, they might want a safety feature where the harness or AI actually confirms or asks before it runs the runs the query that is there, right?

So, these are like two, three things I can think of right out of my head which requires you to build additional software on top of just a large language model, right? Because remember that a large language model is nothing but a simple API call. You give it some tokens, which are just some numbers, and you get some out output, which is again like just some numbers, right? That's all what's happening.

In order to make it useful as work, as meaningful work, you need to give it real-world access, and that is how it this is done. Now, this whole thing that I just told you about, this is nothing but an AI harness, right? That is what we call as a harness. If you look at Claude Code, for example, Claude Code is a harness, right?

It can run multiple models, it can run Sonnet, Opus, Haiku. You can even hack around and run different models with it. Claude Code is a harness. Codex is also a harness, right?

Codex uses OpenAI's GPT 5.5. That is why you would have seen that people always refer to two things. They refer to the model that they're using, like Opus and, you know, GPT, for example, and they refer to the harness that they're using it on, as well. So, if you use Open Opus on Claude Code, and if you use GPT on Codex, you are basically in the official territory, because both the model and the harness has been created by the same company.

And same thing over here, as well. But, then there are a lot of external harnesses, as well, which exist for different different use cases. For example, one of the one of an interesting use case which is possible with these LLMs is code testing, as well as verification. Right?

Whether your code is actually working right or not, right? So, this probably requires a completely different type of harness in the first place, right? What you are doing with your regular editors, like GitHub Copilot or Claude Code or even Codex, might be much better if you're using a custom harness, and that is where today's sponsor, Test Sprite, also comes into play. So, Test Sprite is exactly that.

It's an AI testing agent that handles not only just the testing part of your code base, but also the verification whether the tests that are written in the first place are right or not. Now, Test Sprite sort of is a unique harness, because it can sit in your existing harness, as well, right? So, it can modify the tweak the harness which you are currently using, and it can integrate itself inside it. It's super simple to get started with Test Sprite, and it's super powerful, as well.

Once you create a free account and log in, you're going to see that on the left you have a bunch of these links, where you can get started either with MCP test or web test. So, just to give you a quick example of how web test work, which we can experience directly inside the browser itself, you can create tests which are either checking your backend APIs or your frontend applications. You specify either the website URL, in case of frontend, some context on what do you want to test in the first place, the logins it might need, and finally the test files associated with that. Look at this.

I gave it my website's address sometime back. This is my previous startup, which has now been acquired, and it created a bunch of these test cases automatically, and it also ran them. And you can actually see what the test actually did. For example, over here, it verified that the pricing page is working fine or not in the first place.

So, all of these tests are actually auto-generated, but you can also generate them on your own wish. For example, if you want to test a specific part of your application or a specific feature, you can ask the AI inside Test Right, and it'll generate that for you. You also get detailed test reports for every single test that it's running, and this can be integrated with your GitHub repository as well. So, do check out Test Right.

It's free to start with, and also offers first month free of cost if you sign up using my link in the description. Now, one interesting part of this discussion, probably you know from the Test Right discussion, I can also add verification over here as one of the things, right? So, your agent harness is also supposed to verify what it's doing, whether that's actually the task is actually done or not in the first place. So, what we are seeing over here effectively is that we are approaching a place where Claude code, Code X, a bunch of these harnesses are more or less trying to optimize for the model that they are for, right?

In a way. You can technically run Opus inside Code X, and you can run GPT inside Claude code. It'll be very weird. I'm sure like there are some people who are doing this in the world because it's a weird place.

But the idea is that Opus, or you know, Anthropic is T-Mobile Anthropic is actually optimizing Opus for Claude code or vice versa. I highly doubt like they are doing anything magical in training that leads it to have a better experience inside Cloud Code. But anyway, you get the idea. But at the same time, you will see that there are people who are able to show that Opus is actually much better inside their own harness.

Why? Because they're able to provide a much more specific use case or a specific execution environment for the model instead of Cloud Code. What you have to remember is that look at these harnesses like Codex, Cloud Code, you know, even like GitHub Copilot, let's say. And a bunch of more, right?

There are a lot more over here. For example, Antigravity also just happened, right? They just released their CLI recently. The idea is that they are building it for general public, people who are building or trying to build software at least, right?

Now, these could be actual developers, these could be new dev new people who are just trying to get into coding, they could be product managers, they could be like XYZ people, any of them, right? The reason I said in the starting of the video that you can probably develop your own harness and maybe you should is imagine let's say you are developing a company harness, right? Let's say your company name could be anything. I'm just making it up, let's say Google for now.

Let's say you have a Google AI as an internal harness, which is not Cloud Code, which is not GitHub Copilot, Antigravity, any of this. What this is is just day-to-day operations that you're doing inside Google, inside your company, and it simplifies them, right? A very quick, very simple example is for example, let's say this has a direct prod DB access at all times, right? Because of the environment variable that is configured, and this has basically no security for SQL execution.

Right? Because you want minimal um interruptions for this harness that you have, but this has strict security for any bash command, right? So, any bash command that it is executing on your system has to be approved by this. So, you can create this sort of harness, which is custom for your use case.

I'm giving you a very a super bad example, but you would actually be able to create a much better contextualized example based on where you are working or what product or what project you're are on. The idea is that this harness is not magic, right? All of these are basically front ends for your model. So, all of them are actually also introspectable, and I assume that at some point within the next few months, all of them would actually be the same in a way.

Like, they'll be basically same in terms of features or things or stuff that they are able to do. You know, if you look at Claude Code, for example, you will see that they have sort of like an agent team where somebody's going, dispatching, and you know, doing a bunch of work. Same for Codex. They They'll create like a list of, you know, if you There was a recent thing that, you know, Codex came up with goal command, and then Claude Code copied that, right?

All of this is part of Harness, right? Because what What is this goal command? This is nothing but just the AI generating a list of tasks over here, and then this main AI, which is there, this is supposed to just keep on, you know, doing this in a loop. So, it'll dispatch other AIs to do this work, and then it'll also have a verification system, right?

So, that is This is nothing but Harness. Simple code that is written that just keeps working with this Opus model over here or GPT model over here, right? There is basically nothing stopping you from writing your own Harness inside your own company, which would probably be much, much better for you than using Claude Code or Codex internally for specific tasks, right? I'm not talking about general coding task.

Probably Claude Code and Codex are better for that, but for specific tasks, there is nothing stopping you from creating your own Harness. Now, an interesting take that I have about these Harnesses generally is that all of this actually will go down the drain the moment we solve or any of the frontier labs, rather I should say, would solve this context window thing, right? Because if you think about it, technically speaking, two things, not just one, sorry. Context window is one, plus even more intelligence.

Now, I would probably say that models like Opus and GPT are intelligent enough if they are less than, let say 50,000 tokens, right? You can basically ask them to do any sort of sophisticated task or have a like a actual conversation and they will be smart enough to do that, right? So, if we solve this, let's say if there is virtually no context window for a model like Opus 4.7 or GPT 5.5, like 100 million or so on, and they are retaining that intelligence, right? They are not losing out on information.

They are able to keep their focus on attention on what details are important which things to track, which things to not track. I feel that most of the existing, uh, you know, general-purpose harnesses at least that are there, they would actually go out of business, right? Why do you need a Claude code, Codex, different different harnesses when basically you can just have one single AI, like let's say let's say the ultimate harness. This is plugged in with like let's say Opus 10, which is like the ultimate model, and this has access to everything, right?

Let's say it has 10 million off tools available. And this is like it's not losing out on context and it's not degrading its intelligence, right? The moment this happens, that's it. Even the security, I would argue, because the the model is intelligent enough, I assume that, you don't even need the security, right?

The model would just not do something stupid that requires your human intervention. So, the moment this context window thing is solved, like we have 100 million context window models with models which are not losing out on their intelligence as the context window scales up, I feel that this whole business of creating harnesses would go down the drain. It'll still be interesting to have custom use cases harnesses because of like one of the examples that I just gave you, right? I mean, but even in that case you can argue that you can just create probably, you know, a rules file or, you know, a agents.md file, whatever, which is actually, uh, you know, it's going inside their system prompt and they're not forgetting about it at all, right?

But yeah, that's pretty much it for this tutorial. Hopefully, you took something interesting on how these AI agent and harnesses work. It's a very interesting topic and something that you know is thrown out a lot as a buzzword. It's not as complex as it seems.

It's actually super simple and I suggest you I recommend you create your own harness as well at some point. Just try to do it and you'll be surprised how far you can go with intelligent models now. So that's all for this video. Hopefully you liked it.

If you did, make sure you leave a like and subscribe to the channel. I'm going to see you in the next video very soon. If you're still watching, make sure you leave a comment I watched till the end below to tell me that you were still here and let me know what do you think about the video. >> [music]

Welcome back to the AI Daily Brief. Today, we are digging into a topic that first, you might have heard this term floating around a little bit, but second, even if you haven't, if you are among the subset of the audience that has been dabbling with Claude code or Codex or even using open claw, you have been living in and doing this thing whether you realize it or not. I'm talking about harness engineering. And you might notice that there is kind of a lineage of engineerings that we focus on that have changed over the years in AI.

In 2023 and 2024, we talked a lot about prompt engineering, the art and the science of finding the right ways to prompt the model to get the results that you wanted. There was so much in prompt engineering that people spent so much time on. Think about the things that everyone used to recommend like getting the model to adopt a persona or later on the whole idea of JSON engineering where people hyper-structured their prompts in the way that an engineer might. Now, last year in 2025, we started to talk a lot more about context engineering.

The idea of context engineering was that it turned out that what mattered for AI performance was not just the way you spoke to the model, but what set of information or context that model had access to. Take the example of asking ChatGPT to help you create a marketing campaign. One part of getting good results, sure, might be what you prompted for and how you ask it, but obviously it's kind of intuitive that if ChatGPT had access to information about the performance of all your past marketing campaigns, it might be able to be more informed in how it helped you. So, context engineering was all about the way that we brought together different context and gave AI access to it.

Now, interestingly, context engineering actually kind of has had divergent meanings for different people. For engineers and developers, context engineering has often been about designing the systems that surround AI and agents in order to better interact with and use context, dealing with problems like persistence and memory and state. in a way, this is kind of a part of what we'll talk about with harness engineering. For lay people, for non-technical users, context engineering has been much more about what's the best way to give AI access to the information it needs to help me do its job.

Now, it's important to note that while prompt engineering might have decreased a little bit in its importance scale, context engineering is still very much alive and important. In fact, I did that entire episode about a week ago about how to build a personal context portfolio so that you could transport your personal context from LLM to LLM or agent to agent without having to repeat yourself every time. But the term de jure right now is harness engineering, which is effectively about everything you put around a model, the systems, the tooling, the access that help it do what it's meant to do. And when one starts to look around, you kind of start to see the harness engineering conversation popping up everywhere.

At the beginning of April, Cursor launched its newest version, Cursor 3. In their announcement post, they wrote, "Software development is changing and so is Cursor. In the last year, we moved from manually editing files to working with agents that write most of our code. How we create software will continue to evolve as we enter the third era of software development where fleets of agents work autonomously to ship improvements.

We're building towards this future, but there is a lot of work left to make it happen. Engineers are still micromanaging individual agents, trying to keep track of different conversations and jumping between multiple terminals, tools, and windows. We're introducing Cursor 3, a unified workspace for building software with agents. The new Cursor interface brings clarity to the work agents produce, pulling you up to a higher level of abstraction with the ability to dig deeper when you want.

It's faster, cleaner, and more powerful with a multi-repo layout, seamless handoff between local and cloud agents, and the option to switch back to the Cursor IDE at any time." So, all of the features that they then go on to announce, having all of your agents in one place, the ability to run many agents in parallel, new UX for handoff between local and cloud, all of this is the instantiation of harness engineering into a product. Even more recently, we got Claude managed agents. In their announcement post, they said explicitly, "It pairs an agent harness tuned for performance with production infrastructure." And in the accompanying blog post, they basically say, "This is kind of all about harnesses." The post was called "Scaling Managed Agents, Decoupling the Brain from the Hands." Now, of course, in this metaphor, the brain is the model and the hands are the harness. "Harnesses," they write, "encode assumptions that go stale as models improved.

Managed agents, then, is built around interfaces that stay stable as harnesses change." Now, we'll maybe come back later to some of the specifics of that new product, but again, the point here is that harness engineering is kind of everywhere. At the beginning of March, Latent Space dropped a post called "Is Harness Engineering Real?" And to provide another analogy, their team references back to when they worked in finance. It doesn't say for sure, but I assume this is Shawn/Swix writing because this was part of his experience set, but whoever it was wrote, "A common debate in my finance days was about the value of the human versus the value of the seat. If a trader made 3 million in profits, how much of it was because of her skills and how much was because of the position, institution, and brand she is in?

And any generally competent human could have made the same results." They continue, "The same debate is currently raging in harness engineering, the system subset of agent engineering, and the main job of agent labs." Agent labs, by the way, are how the Latent Space team refers to everyone like Cursor, Cognition, et cetera. "The central tension," they continue, "is between big model and big harness. An AI framework founder you all know once confided in me at an OpenAI event, 'I'm not sure these guys even want me to exist.'" To define harness, they write, "In every engineering discipline, a harness is the same thing, the layer that connects, protects, and orchestrates components without doing the work itself." They continue, "Talking with the big model guys, you really see it. Every podcast with Boris Cherny and Cat Wu, the creators of Claude code, emphasize how minimal the harness of Claude code is, meaning their job is mostly letting the model express its full power in the way that only the model maker knows best." In one interview, Boris said, "I would like to say there's nothing that secret in the sauce.

Generally, our approach is all the secret sauce, it's all in the model. And this is the thinnest possible wrapper over the model. We literally could not build anything more minimal." Cat added, "It is very much the simplest thing I think by design." Noam Brown from OpenAI seems to agree. They quote him as saying, "Before the reasoning models emerged, there was like all of this work that went into engineering agentic systems that made a lot of calls to GPT-4O or these non-reasoning models to get reasoning behavior.

And then it turns out we just created reasoning models and you don't need this complex behavior. In fact, in many ways it makes it worse. You just give the reasoning model the same question without any sort of scaffolding and it just does it. And so, people are building scaffolding on top of the reasoning models right now, but I think in many ways these scaffolds will just be replaced by the reasoning models and models in general becoming more capable.

On the other side," says Latent Space, "are the big harness guys." Jerry Liu from Llama Index wrote a post on this on X that he titled "The Model Harness Is Everything." He added a picture that sums up his point as saying, "Agent reasoning is exponentially improving, but models are blank slates. The biggest barrier to AI value is the user's own ability to context and workflow engineer the models. The more complex the business process, the more complex the prompt that users need to define." Now, where Latent Space comes out is that while they might have some bias towards the big model thesis, actually referencing the bitter lesson that we talked about in episodes a couple of weeks ago, they also acknowledge that harness engineering has real value. So, let's dive a little deeper into what harness engineering actually is.

And for part of our guide, we're going to use a post from humanlayer.dev from the middle of March called "Skill Issue, Harness Engineering for Coding Agents." Author Kyle writes, "We spent the last year watching coding agents fail in every conceivable way, ignoring instructions, executing dangerous commands unprompted, and going in circles on the simplest of tasks. Every time the instinct was the same, we just need better models, GPT-6 will fix it, we just need better instruction following, it'll work when niche library I'm using is in the training data. But over the course of dozens of projects and hundreds of agent sessions, we kept arriving at the same conclusion. It's not a model problem, it's a configuration problem.

Yes, models will get smarter and yes, some existing failure modes will disappear. And then because they are smarter, we will give them new problems which are bigger and harder and they will continue to fail in unexpected ways. Unexpected failure modes are a fundamental problem for non-deterministic systems. So, instead of praying for GPT-6 for Codex Ultra High Extended to save us all, what if we focused instead on answering the question, how do we get the most out of today's models?" And the next point that Kyle makes is the one that I was saying before, which is that most of us who have been dabbling in these systems, be it open claw or Claude code or Codex, have been doing harness engineering whether we realize it or not.

He continues, "There are lots of ways to get better performance out of your coding agent. If you use coding agents for moderately hard tasks, you've probably configured your coding agent a bit. Have you used skills, MCP servers, sub-agents, memory, agents.md files? A coding agent equals AI models plus a harness.

These are all technically separate concepts, but they are all part of the coding agent's configuration surface. Basically, what does the model use to interact with its environment? Harness engineering," they write, "describes the practice of leveraging these configuration points to customize and improve your coding agent's output quality and reliability." They continue by arguing that harness engineering is the subset of context engineering which primarily involves leveraging harness configuration points to carefully manage the context window of coding agents. It answers, "How do we give our coding agents new capabilities?

How do we teach it things about our code base that aren't in the training data? How do we increase task success rates beyond magic prompts?" And one of the things that they point out is that harnesses aren't just one thing. To some extent, harnesses work backwards from what models can't do natively to create some component to solve for that. In another post from Viv from LangChain called "The Anatomy of an Agent Harness," Viv added a chart that showed the desired agent behavior versus what the agent adds.

For example, the simple one that's a part of every Claude code session, if the desired agent behavior is to write and execute code, the harness adds bash and code execution. If the desired agent behavior is safe execution and default tooling, the harness adds sandboxed environments and tooling. If the desired agent behavior is remembering and accessing new knowledge, the harness is going to need to provide memory files, web search, and MCPs. And importantly, when you've heard about all of these techniques like Karpathy's auto research or the Ralph Wigam loops, those are harness additions to get to the desired agent behavior of completing long horizon work.

They also point out that this is something that the big labs are talking about quite a bit now, too. Back in February, OpenAI dropped a post called harness engineering leveraging Codex in an agent-first world. The place that they start from in this post is the goal of building and shipping an internal beta of a software product with zero lines of manually written code. That has been the context through which they have had to figure out what needed to be part of the harness that they were designing.

One of the big experiments that they found was effectively that in this new approach to engineering, they had to uncover new ways of giving in the agent progressively more context. This is this idea which you might have heard me talk about before called progressive disclosure, which is a key part of the way that agent skills have been designed where skills that provide context effectively unfold with the agent being able to access the minimum amount of information to know if it needs to go deeper into that skill without having to crowd out its context window with all sorts of unnecessary information. The key part of the story, though, is in some of the last lines in the post. They conclude, "Our most difficult challenge is now center on designing environments, feedback loops, and control systems that help agents accomplish our goal building and maintaining complex reliable software at scale." That is a very different proposition than just making a model better.

Now, as we get more discourse about harness engineering, we also get more maps and guides for what it actually means. In an Anthropic Labs post, for example, they describe the key components of harness as a three-layer architecture. The information layer, which determines what information an agent can see and what capability it can invoke, i.e., memory and context management and tools and skills. An execution layer that determines how work is decomposed, how agents collaborate, and how to recover in the case of failure, which is basically orchestration and coordination and infrastructure and guardrails.

And a feedback layer which determines how the system can improve over time, whether the results of the execution are verified, and whether each failure is recorded and transformed. So, that's evaluation and verification and tracing and observability. And there is increasing evidence out there of the power of harnesses. Blitzcy, who has over the last year been a frequent sponsor and collaborator on the show, recently released a 66.5% performance on SWE-bench Pro.

That is much higher than, for example, GPT-5.4's 57.7. Now, effectively, Blitzcy's whole thesis could be reframed as being that the harness layer, the agent scaffolding, the orchestration, and the context infrastructure wrapped around the foundation models can unlock bigger performance gains than the models themselves. One of the key things that they found when auditing their performance versus GPT-5.4 is that in many cases GPT-5.4's failures weren't catastrophic. It got close on every problem, but missed intricate details and corner cases.

When Blitzcy succeeded on those same tasks, it succeeded because its knowledge graph gave its agents deep code base context that a raw model doing a single pass couldn't match. LangChain has also recently been writing about how they've been improving agent performance with harness engineering as well. Nicolas Charrier thinks that there is some amount of consensus around the power of the harness that is starting to happen. The AI entrepreneur recently wrote a post on X called The Great Convergence, and it plumbs through some similar themes as my episode every AI product is becoming every other AI product, but puts it in the context of the harness.

Nicolas writes, "Over the last year, a strange thing has happened in tech. Very different companies have started moving towards the same product shape, and it feels like everyone is building the same thing." Linear announced last week that they're building coding agents. OpenAI is deprecating Sora and focusing entirely on Codex. Anthropic is obviously all in on Claude Code and Co-work.

Notion is building agents for work. So are Google, Microsoft, Meta, lovable Retool, and many others. What changed, he writes, "is not just that models got better, although that's a major part of it working." He argues that the important shift is the invention of what he calls the general harness. The simple harness architecture diagram that he describes, a user input hits context engineering, which moves to the model, which calls in tools, which access context engineering in a loop until the task result comes out on the other side.

Nicolas writes, "Claude Code was a massive breakthrough. Although initially invented for coding use cases, it turns out that a smart looping agent generalizes incredibly well towards any computer-based task if you give it the right tools. So, this new technique emerges and turns out to be a general problem-solving machine. It also scales on a very unique dimension.

It can keep running for a long time. It takes the shape of a model harness, a goal, and a set of tools. It runs in a loop calling tools until it stops and produces a result." He points out that so many of the new agents that we're seeing all come back to just a harness looping agent architecture with the right tools and context management. Ultimately, he predicts that by the end of 2026, many software companies will look like they are selling the same thing.

He writes that it's not because the industry lost imagination, but because the architecture and economics are pushing everyone towards the same destination, self-improving software systems that can take a goal, use tools, and produce business outcomes. The harness, he writes, explains the convergence. The self-improvement explains the acceleration. Once agents can be monitored, evaluated, orchestrated, and improved by changing their own code and context, the companies that own more of the loop will improve faster, and their progress will compound.

The winners, he says, will not just have better models. They will have distribution, trusted workflow positioning, proprietary context, and the shortest path from observation to improvement. Which brings us back to Anthropic's managed agents. Because what they show is that we're now at the point where we're not only recognizing harness engineering, we're starting to build towards inevitable changes in what harness engineering is.

Remember, the subtitle from their accompanying blog post is, "Harnesses encode assumptions that go stale as models improve." Managed agents is built around interfaces that stay stable as harnesses change. Here's how they set it up. A running topic on the engineering blog is how to build effective agents and design harnesses for long-running work. A common thread across this work is that harnesses encode assumptions about what Claude can't do on its own.

This then goes back to that idea from the LangChain blog that the harness is about adding things that address a certain desired agent behavior that aren't in the model natively. Continuing, Anthropic writes, "However, those assumptions need to be frequently questioned because they can go stale as models improve. As just one example, in prior work we found that Claude Sonnet 4.5 would wrap up tasks prematurely as it sensed its context limit approaching, a behavior sometimes called context anxiety. We addressed this by adding context resets to the harness, but when we used the same harness on Claude Opus 4.5, we found that the behavior was gone.

The resets had become dead weight. We expect harnesses to continue evolving, so we built managed agents, a hosted service in the Claude platform that runs long horizon agents on your behalf through a small set of interfaces meant to outlast any particular implementation, including the ones we run today." Basically, they say building managed agents meant solving an old problem in computing of how to design a system for programs as yet unthought of. Another way to put it is that Anthropic is building a meta harness, a system that's deliberately unopinionated about what any specific harness should look like because they expect harnesses to keep changing as models improve. Going back to that brains versus hands metaphor, effectively, Anthropic separated the agent loop, the brain, from the execution environment, the hands or sandbox, and even separated it from the event log, which is the session.

Each can fail or be replaced independently. And I think that this has big implications for the debate that we started on of big harness versus big model in the way that Latent Space framed it. It sort of a little bit obliterates that debate. Anthropic is saying, effectively, "Yes, harness engineering matters.

In fact, it matters so much that we're building infrastructure to make harnesses disposable. The whole point is that any given harness is temporary. The discipline is permanent, the specific implementation is not." But let's talk about why any of us should care. First of all, if you use Claude Code, Cursor, Codex, or Open Claw, which is by far the most successful open-source harness we've had so far, you are already doing harness engineering whether you call it that or not.

Every time you write an agents.md file, structure your repo so the agent can navigate it, anything like that, you're building an outer harness. Prigida Bakula actually distinguishes between the user harness and the outer harness. Basically, there's the inner harness, which is built by the builders of the coding agent, i.e., Anthropic or OpenAI, and the outer harness, which is built by you, the user. The outer harness is what's going to determine whether the agent produces good work around your specific code base or your specific goals.

But I would argue that that is not the only reason to care about harness engineering. If you're an enterprise leader, the mental model matters because it reframes AI adoption from pick the best model to pick the best environment for agents to work in. It is a technical capstone on the larger truth which all enterprises are realizing that AI success is not about dropping in a tool and hoping it works, but about designing a new system in which the capability set that AI has and that AI enables among your people can thrive. One could torture the harness engineering frame of reference and, frankly, extrapolate it out to the entire organizational design need.

Simply put, the model and the tools are necessary but insufficient. The environment you put them in is going to determine the output quality. And lastly, if you're just a consumer watching this, hopefully understanding harness engineering a little bit better will go some way to explain why every product seems to be turning into every other product. When we understand that the core loop of models calling tools in a loop until they are done, actually turns out to be general purpose, you understand why Linear is building coding agents and why Notion is building work agents and why they're all heading towards the same place.

The harness enables them to use the models to accomplish whatever goal they were set out to do with a process that a pattern that starts to look really familiar. Anyways guys, hopefully this was a valuable primer in a term that you're going to be hearing a lot more of. But for now, that's going to do it for today's AI daily brief. Appreciate you listening or watching as always and until next time, peace.

Updating the docs and making sure they work is not the most fun part of an engineer's job. Recently, an engineer and I had to update our docs to work with a new CLI for Agent Core, but we used an agent to make this so much faster. We built the harness around the agent by allowing it to access an AWS account, run CLI commands written, and report back any errors. Agent harnesses define the control layer between a model and the real world, but what is an agent harness?

It's an infrastructure environment that engineers wrap around a model so it can do more than just be a plain chatbot. Code agents such as Cloud Code, Curie CLI, and Codex use their harnesses to take your prompt, pass it to the model, and then give the model tools to execute tasks. That's why if you ask a coding agent to write tests for this function, it won't just generate code for you to copy-paste. Instead, it'll have access to bash in a sandbox, create a test file in your repo, paste the generated output, and then run to see if it works.

The finding challenge for developers this year is how to build an agent harness that can stand the test of time if a new and improved model is released. While the model acts as the brain, the harness provides the hands, the infrastructure, and the memory that allow an agent to do real work. As models get smarter, our harnesses should get simpler. Over-engineering with rigid guardrails can really choke the system and create noise for the model.

By offloading memory to the file system and using standardized interfaces like MCP, we allow the model to be as smart and capable as it can be. That's the philosophy behind Trans agents. It's an open-source framework we built around model-driven architecture, which means you never hardcode the sequencing. You define the tools, write a prompt, and then the model figures out what to call and when.

Let me walk you through what building a harness looks like with Trans. This is a real use case for the Trans engineering team. An agent built with the Trans TypeScript that monitors OpenAI's change log and decides whether an API update matters for the SDK, and if it does, files a GitHub issue. Trans works with Amazon Bedrock by default, but it's provider-agnostic.

You can swap in any supported model provider in one line. The first stage, we have a tool. It needs to have a name, a description, an input schema, and a callback. The description is what the model reads to decide when to use it.

The schema defines what inputs the model can pass. The callback is your code that runs when the model calls it. We pass the tool to an agent and invoke it. Model reads the prompt, decides it needs to fetch change log, calls the callback, and uses the result to respond.

That's tool use. One cycle, think, act, respond. Now the agent needs to make decisions. We're going to add two more tools.

One classifies a change log entry, one searches GitHub for duplicates. We have three tools all in one agent. The model fetches the change log, classifies each entry, and only searches GitHub for the ones marked repo change needed. There's no routing logic we have to build.

The model reads the description and sequences the calls itself. Descriptions are how you steer the agent. If you write a vague description, you're going to get unpredictable behavior. Precise descriptions will lead to reliable tool selection.

Notice the safeguard and classify entry. A new model slug alone is not repo change needed. That's a constraint we wrote plain English. The model reads it and respects at runtime.

No middleware needed. We're going to also add two life cycle hooks, before and after each tool call. Every decision the model makes gets logged. With these, you'd see model fetch a change log, classify each entry, and then only search GitHub for the ones it flagged.

It skips the model launch because it applied the safeguard. If it classified something wrong, you tighten description. Debugging is just editing strings. Now the agent can act.

One more tool, creating GitHub issue using a structured template. Here's what the end result looks like, where we found an issue with the change log, title follows a convention, body has a summary, usage sketch, and references. There's a template. The guardrail is in the description.

Classify first, dedupe first, and then create. The template is enforced through the input schema. The model has to provide every field. That structure mirrors the issue we just looked at.

All four tools running in the same agent. Model fetches, classifies, skips what doesn't matter, dedupes, and files an issue all from descriptions and the system prompt. There you have it, a Trans agent improving Trans. That's what a model-driven framework gives you.

You're not locked into a particular [music] model's strengths. You easily adjust the harness anytime. You don't need the best model to build a great agent. You need a model-driven harness that gets better every time the model does.

[music] >> Our next speaker is here to speak about Harness Engineering. How to build software when humans steer and agents [music] execute. Please join me in welcoming to the stage member of technical staff at OpenAI Ryan Lepopolo. >> [applause] [cheering] >> Good morning, London.

>> [applause] >> I'm super excited to be here today. I'm Ryan Lepopolo and for the last 9 months, I have had the privilege of building software exclusively with agents. Uh I am a token billionaire and I believe that in order for us to get into our AGI future, we want everybody to be token billionaires to use the models to do the full job. And what that means is to lean into the idea that the models are capable of being a full software engineer.

And I've lived that experience by banning my team from even touching their editors to have to work through the models in order to get the job done. And today I'm going to talk to you a little bit about what it means to lean into that and operationalize the way you work, the code spaces you live in, and the processes on your teams in order to get the agents to do the full job. I believe I'm preaching to the choir here when I say that the way we build software has changed. In the last 6 months, we have seen coding agents take over the world and capability has continually advanced at a super fast pace to have these models and the harnesses within which they live take more complex actions, do more complicated work with higher reliability over longer time horizons.

And the place we've gotten to here is that implementation is no longer the scarce resource of what it means to do the job of software engineering. Code is free. We have an abundance of code to solve the problems that we come across in our day-to-day as we run our teams, build software, and solve user problems. Hiring the hands on the keyboards as part of our teams is only constrained by GPU capacity and token budgets.

And each engineer today in this room has access to five 50 or 5,000 engineers worth of capacity 24/7 every day of the year. The only thing that needs to happen or roles is to figure out how to productively deploy these resources into our code and into our teams to make use of this new capacity. And in this world skill sets are shifting more towards systems thinking, system design, and delegation in order to make use of this abundant capacity to produce code to solve problems. There are three reasons that this happened, all of which happened in late 2025.

For me, the magic moment was GPT 5.2, which when it came out was able to do the full job of a software engineer. The models at this point are good enough where they're isomorphic to you and I in terms of the ability to produce code at high quality that solve real user problems in real code bases. Code is free. And I know this is maybe a scary thing to hear because code carries maintenance burden, but it's free to produce, free to refactor, and it is not a thing to get hung up on anymore.

We think of code as burden because it it's a synchronous attention drain on the human engineers on our team, but the models are incredibly patient. They are infinitely parallel, so the ability to produce, maintain, refactor, and delete code is no longer a forcing function on figuring out how to allocate resources on your engineering teams. So, sort of be AGI pilled here is to believe that the models are capable of producing every line of code we could ever possibly need, figuring out when to delete them, figuring out when to refactor them, or make them more reliable. And it's your role as software engineers to figure out how to unblock your team of agents and humans driving those agents from being able to drive them over long horizon work to do the full job.

The idea here is that every one of you is a staff engineer. You have as many team members as you can possibly drive concurrently and have tokens to support, and you need to look 1 day, 1 week, 6 months into the future to figure out what structures you need to put in place to productively harness this infinite capacity to produce code. The scarce resources in this world that we see today are three things: human time, human and model attention, and model context window. And in the world where human time and attention is scarce, the role is to think about where that time is going, figure out ways to productively automate it, and move that synchronous human time into higher leverage activities.

In a world where human time is scarce, and human time is required to produce code, we have a stack rank. Things are either P0s or P2s. Those P3s will never get done. However, in a world where code is free and infinitely abundant, all those P3s get kicked off immediately.

Maybe 4X in parallel, we pick one that solves the problem, and in it goes. I've had the privilege of building a ton of agents internally at OpenAI to improve the productivity of my co-workers. And when code is free, all these internal tools can have good localization and internationalization from day one. I can make tools that my colleagues in London, Dublin, Paris, Brussels, Zurich, and Munich are able to experience in their native languages without really having to trade against any of my other team's capacity in order to make high-quality tools.

We should be working with the assumption that the best parts of software engineering that we all know live and breathe are available in any product that we could ever build all the time. Humans no need no longer need to concern themselves with implementation. The important thing is not the code, but the prompt and the guardrails that got you there. This is why leaving breadcrumbs, documentation, ADRs, persona-oriented documentation around what a good job looks like, all the historical logs of tickets and code reviews, this is the process that got you and your teams to the code and products that you have today, and this is what is need needs to happen in order to get your agents there as well.

Your job is to build systems, software, and structures that enable your team to be successful. And to do that, we need to make them legible to those agents that are driving the implementation. That means structuring them in a way that's native to the agents, writing them in a way that is respecting of scarce context, which is this other scarce resource here, and figuring out ways to make the tokens that are required to do the job easy to predict. That means making things the same as much as possible, so we can limit the amount of attention the model needs to activate in order to do the job.

Large-scale refactoring in this world is free. So, making things the same is something that you are all able to do. There's never going to be a migration that hangs open for 6 months now that you can't get the last parts of the code base to do because you can just fire off 15 agents to drive that work to completion. This is what it means to have a migration, right?

We can finish them now. Come on. That's good. That's good.

Clap. >> [applause] >> There's sort of this like meta epistemological question here about like what it means to do a good job. And doing good job as a software engineer is hard. It requires us years of being in the industry to fully internalize what it means to write high-quality, maintainable, reliable code that our teammates are able to build on top of that is going to accrue leverage to the code base.

To do a single patch well probably requires 500 little decisions along the way around the underspecified non-functional requirements that go into producing good code. The agents, the models during their training, have seen trillions of lines of code that make every possible choice of those non-functional requirements that you could ever imagine. So, it's our job to specify those non-functional requirements, to write them down in a way that the agents can see this is what it is to do a good acceptable job that's going to produce a merged patch. And if the agents aren't doing that, it's our job to figure out ways to refine and restrict their output such that the code they write is acceptable.

You can just simply say do not produce slop. Don't accept slop, you won't get slop in your code base. But to do that requires taking short-term velocity hits in order to back up or double-click into a task to figure out what it is the agents are struggling with in your environment. Put the guardrails in place so they stop making those mistakes.

And then figure out ways to step back and spend your time on higher leverage activities once you solve some of the blockers in the short term. When I think about empowering my team in this way, everyone is an expert in what it is they bring. I have a diverse full-stack team that is experts in front-end architecture, back-end scalability, being product-minded, and each one of those different personas fleshes out the skillset of my team by bringing a different understanding, a different set of solves for those non-functional requirements. Getting teammates to write those down actually means that every engineer driving agents gets the best of every single person on my team.

I don't need to block on low-signal code review in order to learn what it means to write a good QA plan. To have one engineer on my team document that in a durable way means every agent trajectory is going to get a good QA plan, and we can do this once in a high-leverage way that we're able to stack on top of. So, So, So, how can we get the agents to do a good job? What are some of the tools and techniques we have in order to essentially prompt inject our agents and continually remind them of what it means to make those specific choices that we expect around those non-functional requirements?

And there's a bunch of ways we can do this. We can write good agents.md5 files. However, with auto-compaction, which is a thing that has continued to improve, GPT-5.4 and Codex is fantastic at auto compaction. I essentially never have to write \{{}slash} new anymore.

I've got some pictures on my Twitter of me strapping my laptop into the back of my car so I can continue to do running inference while I'm commuting to and from work. And in this world, you have to kind of build for that expectation that context will get paged out over time. We need to be continually refreshing context as the agent goes about doing a task. And the ways we can do that are by having reviewer agents look at the code along the way through the lens of what it means to be successful, right?

We have security and reliability review agents in our code base that are continually running as part of every push in CI that look at those documentations and the proposed patch and do simple things like say, are there timeouts and retries on this bit of network code? Has the code that has been introduced have a secure interface that is impossible to misuse? I'm sure everyone here has been paged at some point for network code that failed in production causing an outage that could have been remediated by a retry and a timeout. And I know I'm guilty of putting that retry and timeout in, merging the bug fix, and otherwise ignoring that.

I am not a reliable reviewer or author of code with respect to this non-functional requirement. However, taking the time to write some docs, write a lint that is bespoke to my code base that is going to look at every time I call fetch to make sure that there's a retry and a timeout wrapped around it means I've durably solved this problem and I'm able to do it cuz I lean on this axiom that code is free, that the agents are able to do a good job, that I can completely migrate the code base to solve this problem durably once and for all. And in order to kind of operate in this way, we need to step back and look at the durable classes of failures that the agents and the humans in the code base are making time after time, figure out why we're spending time on it, devise a solution to systematically eliminate this class of misbehavior, and then continue to observe, refine, and make additional choices on those non-functional requirements. One really neat trick I use here is that you can write tests about the source code as well that are separate from lints, right?

If we know that context is limited, we can write a test that limits the fact that files are no longer than 350 lines. We're adapting our code base to the harness, to the models, to do a little bit of engineering to be context efficient, and squeeze more juice out of the model capability that we have today. The other things we can think about are providing good error messages that give actual remediation steps to the model and to humans for how to proceed next. It's not enough to say we've got a lint failure because we're awaiting in a loop, or that we have an unknown at this deep part of the code base, and why is the model writing a function called is record?

What we need to do is provide a prompt via a lint or a test failure that says, "No, no, no, you shouldn't have an unknown here at all because we parse don't validate at the edge, and you certainly have a type here which was derived from Zod, load-bearing infrastructure for our AI future." You can just prompt things. Everything I've talked about here today is a prompt. You can do this without touching the model weights at all. Kind of uh funny digression here is it seems like each advancement we've had in the complexity of the way we write code to interact with these models comes from both increasing capability in the models and increasingly niche ways for injecting prompts into those models.

Prompts, I'm sure you're aware, are prompts, powers prompts, rules files prompts, skills prompts. These lint error messages that I am talking about prompts, review agents that inject comments onto the PR that we require the agent to address before it is able to propose it for merge, prompts. You're going to find lots of ways to insert prompts into your code, and one way you can do that is by embedding agent SDKs into your tests. They're going to review the code base for acceptability using prompts that get embedded into the code.

And if I find myself spending a ton of time writing prompts, we can actually shell out to the agent for that as well. Uh I've pointed Codex at all of the prompting cookbooks we have on the OpenAI developer guide and told him to synthesize a skill out of them for how to write prompts, which means when I find a need to write prompts in order to improve my agent performance locally in the code, I use the skill to write prompts that I wrote with the agent looking at the prompts to write the prompts. >> [laughter] >> All the leverage that you're encoding in the into your repository, your team, and the agents in this way stacks incredibly well. To kind of pull back to this idea that a single product-minded engineer on my team was able to give us a big lift.

They know what it means to write a good QA plan. To write a good QA plan though, you have to document all the features that you have, the critical user journeys, and how users engage with your applications, web apps, APIs, and services. Once you write those down on how to write a good QA plan with the expectation that all user-facing work has a QA plan, now a review agent is able to assert expectations around what it means to prove that you have effectively written the feature. A QA plan indicates what media should be attached to the PR for the humans and agents to know that you've done a good job, which has the consequence of me trusting the output more, needing to shoulder surf the agent less, and removing myself from the loop even more to delegate more and more of the work to agents.

And all of this is just making sure the agents have the tools and tokens and context to do the full job, to remove myself from the need as a synchronous driver. The models crave tokens. We can operationalize our code base to give them tokens to drive them forward using sub-agents and all these other techniques to refine the agent output. I'm excited to let you all know today in the way you all do that you can just go build things.

Do not hesitate to remove yourselves from the loop by getting the agents to do the full job because they can. Thank you. >> Very excited to bring on our guest. We've got Ryan Leppolo today.

He just gave the keynote. Um, very exciting speaker. The man is full send hyper engineering at OpenAI. So, uh, a little bit of background.

We did a Latent Space episode with him. We shipped it the other day. The the story, he wrote this great article called Harness Engineering and we're like, "Wow, this is pure gold." We have him on the podcast. He's a token billionaire spending over a billion output tokens a day.

That's like over a thousand dollars. So, you know, man is really living it. Uh, we want to keep this exciting. Ask good questions.

Ask interesting stuff. Ask things that people can learn from, but you know, let's all come right on TO THE STAGE. >> [applause] >> HI FOLKS, HOW'S IT GOING? EXCITED TO BE HERE.

UH London has been fantastic and excited to kind of walk through what it is that we do and how we work here. >> I think you got to come on this camera is just here, so >> got blinded by the QR code, so we're good. >> [laughter] >> Okay, so background, we have about an hour. Um scan this QR code, you should get Slido.

Slido will let you ask questions. If you see interesting stuff, you can thumbs them up and we'll try to get through them. Unfortunately, the first one I can't super do, but let's just kick it off. Ryan, can you show us your actual working setup with [laughter] no laptop?

>> Uh yeah, here. Beach >> [laughter] >> margarita linear, right? >> Um I'll say watch watch the podcast we put out. We go through some of the work, but if you want to talk about it, I guess without actually showing us, what's your what's your workflow like?

What's your setup? How do you how do you approach a task? >> Sure. So uh uh uh the way me and my team work is to start with tickets, right?

We have chunks of work that we want to do, features we want to add to our apps, reliability work that we want to do. Uh we give that ticket to an agent along with a couple of skills that enable it to manipulate our app. Uh we want the entry point to the development process to be Codex, not uh an environment which we build around it. So we kind of do things um outside in, right?

Like Codex is the entry point, the same way you would be and we give it tools, we give it instructions on how to cook. So rather than like creating a shell that our app and Codex gets spawned into, we have a skill that teaches Codex how to launch the app. That teaches Codex how to spin up that local observability stack to give it logging and telemetry. We give it a skill that enables it to uh boot up Chrome dev tools and attach to the application with a, you know, local CLI that will connect via some daemon that we have.

So, the whole way we have set up the repository and all of the local dev tools is for Codex to invoke them first. Um that means we have kind of like a bunch of little mini harnesses within the code base that make it really easy for us to slot in additional guardrails. Uh you know, a big package of custom ESLint rules which get wired into every pnpm package in the workspace. We have another sort of local dev harness that allows us to add sort of like higher-level wholesome tests that assert the structure of the code itself rather than like either the syntax or the behavior of the code.

Things like, you know, package privacy, dependency edges between different layers of our stack, these sorts of things. Uh making sure that, you know, across multiple files zod schemas are deduplicated, that there's a single canonical implementation of like our async helpers, uh these sorts of things. Because, you know, the way we have seen the agents work is to sometimes optimize for local coherence of a package rather than using like our shared utilities and things like that. So, having observed that behavior, we kind of have built a bunch of little pseudo linter, source code verification things that shake out some of that bad behavior so the humans don't get distracted paying attention to that in reviews, stuff like that.

But, uh the setup optimizes for the agent to do the job and for the humans to not have to keep track of the high churn in the code base. Um Um Um we kind of centralize our leverage around five to 10 skills. Uh we don't go super wide on skills preferring to make the existing skills better because at least I I that the infrastructure within the repository, all the local developer tools, change super frequently, uh and I don't really have the bandwidth to keep track of this. So, we hide all that complexity beneath the skills that the human has to invoke and let the agent just kind of figure it out.

Like, one one kind of neat thing here is um when we move from using uh Chrome DevTools protocol directly to having this like daemon thing, like, I didn't know that had happened for like 3 weeks. Uh I was like totally fine because Codex was able to do the thing, uh you know, with the documentation and things that we had in place. >> And part of this, you can get more detail in your article. So, some background, you wrote a great piece called Harness Engineering.

There's a whole section in there on how you thought about skills, thousands of skills versus simplifying it to just quite a few. But, okay, uh continuing on, how do you stop yourself from over-engineering harnesses? And a little bit of a similar follow-up is, do you often build small tools for yourself if ever? Uh do you do you build custom tools?

>> Yeah, so I think this is kind of gesturing in the direction of the bitter lesson here, right? Which is, how do I make sure the work that I do isn't like completely obsoleted by an increase in model capability? And the way I have thought about that is doing sort of the bare minimum amount of context management to kind of pull in requirements uh for the agent to do an acceptable job over the course of its work. And context is a thing that I don't think will ever be obsoleted, right?

Like, the the models must be told like the requirements of the task, which guardrails to pay attention to, these sorts of things. So, a good harness is really operationalized around giving the model text at the right time so it can look at the work it has done and the information around what a good job looks like. And, you know, fundamentally, the models are trained to follow instructions. All the harness should do is surface instructions to the model at the right time.

So, we do want to minimize that, too, right? You don't want to front-load all those instructions because then you kind of like overwhelm the agent. But, all of these sort of requirements around what a good job do need to be paid attention to over the entire course of a PR, right? So, figuring out ways to either defer or just-in-time surface those instructions is kind of what uh a good harness should do, right?

If you know that uh you want your React components, right, to be decomposed so that they make good snapshot tests for individual more stateless pieces, right? You don't need to load that up front. Instead, you should kind of let the agent cook and prototype and experiment with the UI you want to build. And then at lint or test time, say, "Okay, you've done the work.

In order to finish it, you have to break this apart so that your components are small and as stateless as possible and have local dependencies on hooks instead of prompt drilling or whatever it is you want uh the code to look like. And then the agent will say, "Oh, this is a new instruction for me. Let me take the patch as written, modify it to make sure that it adheres to the instructions, and then up it goes to GitHub." And this sort of thing is not going to be obsoleted by increases in model capability. It's really just about getting that right text, that right context to the agent at the right time.

>> Can we talk about an example of a good harness? So, a lot of people are asking about the Codex model, the Codex harness. How does that compare to other harnesses? So, Cloud Code, Open Code, uh how do you guys take these decisions into play?

You don't work directly on Codex, but if there's you can if there's stuff you can speak about about the Codex harness, what you guys see as you architect it out. >> Yeah, so one thing that I think is super powerful is this notion that the labs are not just post-training the models, but post-training the models in the context of the harness in which they are primarily deployed in, right? Like the apply patch tool or like the specific quoting semantics of how to invoke the bash tool or like in the loop for the post training process for the harnesses from the labs, which means like there is leverage to be had by depending on these sort of like first party harnesses directly. At least this is what I believe.

Uh and as such kind of being able to direct through them via things like the SDK or manipulating the Codex app server directly means you kind of get to ride the wave of all that leverage in post training and instead focus on the parts that you care about, which is like what correct code looks like. Um I kind of have high confidence that things like Claude code and Codex will continue to get better. Uh that is the responsibility of like the teams working on these coding agents. So in my role where I don't really want to focus on the coding harness at all is figuring out ways to plug into them in ways that um um um kind of like steer the agent.

That means my job can sort of like move up to thinking about differences in model behavior between releases rather than deeply understanding the nuts and bolts of the harness. Instead, I can think about what it means to you know, drive the behavior that I want based on the observed behavior rather than like the inner mechanics of the thing. >> It's a perfect follow-up to the next question, which is uh do you have any recommendations for collaboration platform? So when you're in the software development life cycle, is there any platform that you use for agents, engineers, developers all to collaborate on working on anything?

Any tips? Any tools? >> Yeah, so in this world it has largely been just markdown files in the repository and GitHub. They have been the primary sort of hub and spoke sort of thing.

If you think about collaborating on a document, like you open Google Docs, you write something, you ask for feedback, people comment, you apply suggestions, these sorts of things. This is kind of like a little clean room environment just for this work artifact that you're producing. Like a PR kind of has a similar purpose. So, we kind of treat that as a big hub and spoke broadcast domain where all of the agents and humans collaborate together.

Uh and because we optimize for throughput, we don't block on any sort of like contribution to that. Like folks can either review or not, agents can either review or not. The implementation agent can acknowledge, defer, or reject any feedback that it gets. Uh really allowing each participant in the production of diffs to kind of make their own judgments around what it means to deliver, receive, respond to feedback.

Uh and this has a nice property of like not putting the model in a box in a bunch of places. We want them to use their good reasoning sort of thing. So, being super prescriptive around like every bit of feedback must be addressed can kind of have this like catastrophic failure mode of your coding agent being bullied by all of the reviewers. When really we want to bias toward code being accepted, not perfect, not drowning in minutia, and these sorts of things.

>> How should people get started with using coding agents? People that have been using a lot of doing a lot of manually written code, how how do they start to transition? What should they offload? How do they kind of come over that barrier of okay, I'm still checking every PR, I'm copy-pasting from Codex.

How should like the average engineer start to use these tools? >> I think there's two ways to approach this problem. One is to start using the coding agents to improve your confidence in the code itself as it is written today, right? I think we would all agree that like more tests is probably a good thing, right?

To assert that our programs are well specified and behave correctly as our users interact with them is a good thing. Uh and the agents are super good at looking at the existing code with some context around how it is meant to be used and writing tests that assert that behavior. So, kind of using this to improve your confidence in the quality of the code will also increase the agent's ability to successfully navigate it, which means you don't have to worry as much around doing super detailed review of the agent output. The other way to think about this is to look at how you are spending your time.

Is it, you know, staring at your editor writing code? Is it waiting for tests to run? Is it waiting for human review feedback? Is CI slow?

And you're like waiting on that. Maybe you have a ton of flaky tests. And using the agents to incrementally automate the parts where you are spending your time. Cuz ultimately, the high-leverage parts of our jobs is to define the work that must be done, prioritize and schedule that work, and then effectively empower folks on our team to do that work.

Uh and the more and more we can delegate and move into sort of this like sequencing and orchestration role, even if if you just think about like managing your teams, right? The more parallel and the more like deeper individual executions of those delegations we're able to do, right? If I put primitives in place that make it super easy to like spin up ways to respond to events on my Kafka queue, right? Like I don't really need to be in the weeds with every engineer making sure they like implement a consumer correctly, right?

And these same sort of like building block-style techniques apply really well to the agents and stack really well, too. >> Final one. How do you work with agents in your car? >> [laughter] >> Um so, I have not used the new uh, voice mode that launched in CarPlay, uh, recently.

Uh, not ready for that. But, uh, usually what I'll do is kick off, uh, a task, uh, right before I leave the office, uh, tether my laptop to my phone, buckle it into the backseat, and kind of let it cook in the 30 minutes it takes me to get home. Uh, most of the time with the skills we invoke that tell the agent, you know, you're operating on a task, you go until the tests are green. Uh, you know, I don't have to reach back there and poke the test continue onto the thing.

Uh, and I'm basically able to more fully saturate, you know, my day with token consumption. Um, the dream here is that I actually have 50 agents running 24/7 and I don't have to interact with them at all. Uh, and the way to do that is to define the work well, figure out ways for it to automatically be scheduled, and remove myself from having to click the button, right? Every time I have to type continue to the agent is like a failure of the harness to provide enough context around what it means to continue to completion.

>> Wow, good good statement at the end there. >> [laughter] >> Every time you have to interact with the agent is a failure. Okay, so the following question kind of scales this out, right? As your org knowledge map scales, what practical steps do you have to like enable progressive disclosure?

So, as you have a larger and larger code base, as you have more people, how do you scale your agents to work better with this? >> Yeah, so when I sort of initially started this project that I was working on, blank repository, create electron app, right, you know, V single package, all this sort of stuff, and eventually ended up with a mess, right? Because there's no package privacy that allows me to enforce invariants around what APIs are public versus which ones are not. The agent didn't have like concrete hooks in the file system to determine which domains were separate from the other ones.

So, we ended up going like full 10,000 engineer organization heavy on the architecture. 750 packages in the PNPM workspace, isolated by business logic domain or layer of the stack, individual small util packages that encapsulate reusable functionality that we lint on being used, that we can encode leverage in. And I do think that like in this world, even if you don't actually have microservices, structuring your repositories in ways that you can actually scope like the directory subtree you are looking in to be able to do most of the change helps. Uh and you know, code in the file system is also text, which means it's effectively prompts that you're giving to your coding agent.

Uh so, making the code as much the same as possible kind of makes it so that regardless of where in the repository your agent is looking, it develops a ton of transferable context, right? Like, you should have one way to like do a bounded concurrency helper. You should have one way to construct a observable and instrumented side effectful command. You should have one ORM, right?

Like, should have one programming language, you should have one way of writing CI scripts, you should have one way of adding additional lint rules, these sorts of things. Because it means that like the tokens that you want the model to produce are easier to predict and more consistently predicted regardless of where it looks. Um so, I would say figure out ways to structure the code so it is local to a subtree in the repository for most of the ways you would interact with that system, and then figure out a way to use these agents to completely migrate the codebase to be the same. You know, empower someone on your team to be a dictator to say this is the way it must be done, right?

Or you can go figure that out together. And you know, write it down, write evolve the code so that it reflects that reality, these sorts of things. >> We've got a few questions on code review. >> Sure.

>> How do you approach code review now that you have such high velocity? Uh do you just not read the code? Do you just trust the test coverage? Uh how do you write good tests?

How do you offload that stigma of like, you know, you have a mental blocker, I need to manually check everything before I merge PRs? >> So, that same sort of idea where you have to look at where you're spending your time and figure out ways to spend less of it. Uh you know, when we started, right, the first thing to do was figure out how to get the agent reliably producing code that we would accept. And a big challenge we ran into is with each engineer producing three to five PRs per day, even on a team of three, merge conflicts were super miserable, right?

Because these PRs tended to be pretty big, we were working on the same parts of the code base. So, that's where we moved in two directions. One was to like tree out the code a bit more to minimize these merge conflicts, but also minimize the amount of time PRs were open so that we were uh reducing the likelihood of a merge conflict actually occurring. And the reason PRs were staying open so long was because we needed code review.

Uh because humans were being the blocker in this scenario. So, So, So, in order to do that piece automatically, I essentially asked every engineer on the team to take one day a week, Fridays, we called it garbage collection day, where our entire job was to take every bit of slop we had observed over the course of the week that was making a PR difficult to merge and figure out ways to categorically eliminate it from ever happening in the first place, which is where we kind of started closing this loop between the feedback that humans were giving on the PR indicates some context failure on behalf of the agent, getting that into the repository, and then figuring out ways to automatically prompt inject the agent so that it would self-heal when it produces bad behavior. And this is kind of how you go from synchronous human time spent giving feedback as a code review comments to documentation in the repository to automatically surfing this documentation either via a failing test or a reviewer agent who is primed to review the code as written in the context of these docs. But all of that happens by putting those docs in a single place that all these processes are able to attach to.

Um Um Um you know, we kind of asked folks to basically bucket the types of review feedback they were giving into like um like the persona they were operating as like front-end architect, you know, reliability engineer, scalability sort of thing. And then basically for each of those personas, we spun up a review agent that gets triggered on every push that says, "Is this code good? Surface any P2s or above that would block this PR from merging based on these documentation that says what good looks like?" Uh and with that and just continuously appending to these files, we started to see slop reduce, reduce, reduce. >> People have questions about your billion tokens.

Where do you think those are split up? So, how much of it is on code review? Where where is the majority of that usage coming from? And a follow-up for people that are just getting started, say they have they've jumped and done a $200 pro plan, right?

If you had to cut your usage by a fifth, how should people maximize their usage, right? You run into usage limits. Um you know, you don't want to just copy-paste million lines of code every 6 hours no prompt hit prompt cache hit, but how should we how should we think about that? >> Yeah.

So, I would say probably a it's probably a third a third a third between like planning, ticket curation, documentation, implementation, and stuff that runs in CI. >> Do you use plan mode? >> Uh we uh I've used exact plans, which was kind of like an early version of this that we published, which is sort of like a proto skill that says this is how you should structure a plan with milestones and acceptance criteria. Um Um Um I haven't really used plan mode as part of the harness at all.

My My sort of expectation here is that I should be able to drop a ticket in and have it do the job anyway without diverting through a plan uh because most of the time I'm never going to read it. >> [laughter] >> Uh so I find that if you do use a plan and you approve it without reading it at all, you're actually encoding a bunch of instructions that you don't necessarily want followed. Uh so if you are going to use plans, my recommendation is to push those up as single PRs with just the plan where you actually have human review every line of it and like block on human approval before they get merged and then kicked off. Uh because is you're effectively potentially wasting your time on a roll out with instructions that like are bad.

Uh so you want to kind of like minimize the time that happens, but I do think that uh kind of getting tokens to be spent in CI is a necessary part here because writing code no longer is the hard part. Like getting code accepted and advancing the code and product forward is like what it takes to make that written code be valuable. And you know, we kind of have all heard the aphorism that like you know, senior engineers give good code reviews. Like we expect our senior engineers as agents to do the same.

>> [laughter] >> Uh someone asked is code a disposable build artifact? >> Yes. Yes. Uh I think we we touch on this with uh Symphony, which is sort of this agent orchestrator that we released.

This idea that you know, we can publish a library that's actually a super well-defined spec that the code is a compiled artifact of. And I think like using LLM as fuzzy compiler is like an interesting mental model to have, right? Like all of the context that we're putting in the code base for Harness Engineering is effectively like constraints and optimization passes on which code is acceptable to build in the first place. Uh and this is pretty similar to like the static analysis and optimization passes that something like LLVM would do in the process of compiling Rust code.

Uh and sort of swapping out one model for another is sort of like changing your code generation back end from, you know, LLVM to Cranelift in the Rust compiler. And you would expect that all of the sort of rules around what acceptable Rust code looks like produce valid, sound machine code out the back even if the generation process is different and you end up with different X86 instructions. So, same sort of mindset for LLMs, swapping out different models, sort of thing. We want the structure around the code to basically limit how it is written to things that would be acceptable to us.

>> At a high level, can you give us a picture of what future you're building for? Does context still matter? How do people do engineering, Harness Engineering, context engineering? What does the future look like?

>> Sort of the the future that I want to build toward here is where I'm able to take a token budget and a quarter, a half, or a year's worth of work, take the human input to rank what is most important, success metrics, reliability metrics, give it to the machines, and have them continually work and advance my product forward uh without sort of, you know, my hands explicitly on the wheels at all. We We We as we have gone through like very early prototyping to internal alpha, internal beta, external alpha, I kind of have felt that like new parts of the software engineering process have kind of like started from zero and we've had to build up capability. Kind of like these like, you know, pentagonal like personality charts, right? Where like I spike in this direction, maybe I'm weak over here.

And you know, when we get to deployed software for the first time, right? The agents' ability to do like QA smoke testing on our built artifacts before they're promoted to distribution was weak. We hadn't invested any time in this. There were no docs, there were no tools that the agents could use to like download the built artifact, launch it, poke around to make sure that our like most critical user journeys were well validated and tested.

So, because I don't want to be touching the computer, we needed to figure out like ways for the agents to build themselves tools to do that part. Uh, and there's a whole universe of software engineering outside of writing code, right? Like I am triaging user feedback, I'm triaging pages, I am making sure that we don't have any PII leaking in the logs in production, I'm making sure that like the Twitter vibes are good and people are enjoying my software, that our user operations staff are supported with well-written run books that allow them to triage and mitigate high-volume user issues. And then moving that into the code itself so they don't happen in the first place.

And as I no longer have to produce code, like my mind can shift to these other higher-level or more squishy activities, but the agents are good enough to do these things too. And figuring out how to like write down the processes and the acceptance criteria becomes like this sort of like meta programming part of the job using these agents. >> I think that's a great way to end it. What an exciting future.

Give it up for Ryan, guys. Thank you folks. >> [applause] [music]

AI agents are taking real actions in the real world right now. They're sending emails, booking meetings, writing code, and in some cases actually starting religions. That's not science fiction. That's simply Tuesday in this new world.

And if there's no harness around those agents, you have a serious problem on your hands. Hi, if you're here for the first time, I'm Troy Malone and I run Relevant AI Lab where we help ambitious professionals upgrade their AI skills and use smart tools for work and life. Check us out at relevantailab.com. In this video, we're going to talk about what a harness actually is, why it matters more every single day.

By the end of this, you'll understand the infrastructure that separates a control intentional AI deployment from one that's simply flying blind. Think about a climbing harness or a car safety belt. Its job isn't to slow you down. Its job is to keep you in control when things get fast and stakes get high.

Agent harnesses do the same thing for AI. An agent harness is the infrastructure wrapped around an AI agent that tells it what tools it can use, what it's allowed to do, how to behave, and what to do when something goes wrong. Without it, the agent simply wanders. With it, the agent operates inside a system you designed and you control.

And here's why this matters so much right now. AI agents are not just answering questions anymore. They're taking actions. Which means the guardrails you build around them or failed to build have real consequences.

I want to be really clear here. There are two levels of harness that you need to understand. Level one is the platform harness. This is the factory floor.

Tools like n8n, LangChain, and Claude's MCP connections are platform-level harnesses. They orchestrate how multiple agents and systems talk to each other, what permissions they have across your tech stack, and how errors get caught before they cascade. This is the big infrastructure layer. Think of it like the electrical system in a building.

Most people never see it, but everything depends on it. Level two is the task harness. This is the individual workstation on that factory floor. Every time you build a focused AI tool, a script writer, a sales coach, a a research agent, whatever you're building, you're building a task-level harness.

You're defining the persona, the the guard rails, the output format, the boundaries of what this agent is allowed to do. It is smaller in scope, but it's just as important because a poorly designed task harness is how you get an agent that goes off-script, hallucinates confidently, or takes the wrong action at exactly the wrong moment. In other words, the platform harness controls the pipes. The task harness controls the nozzle.

You need both. This is exactly what we train our participants to do in our Relevant AI Orchestrator System program, to build these guard rails intentionally into everything they deploy. Because what you once you understand how harnesses work, you can't unsee it. As AI gets more autonomous, harness design is becoming one of the most critical skills in any organization.

This isn't a developer-only problem. If you're deploying AI tools for your team at any level, you are already in the harness business. The question is whether you're doing it intentionally or by accident. If you found this helpful, hit subscribe.

We'll keep you ahead of these big AI changes so you don't get left behind. Drop a comment and tell me, what's the one AI agent you're already using and do you think it has a good harness around it? With AI, it's not a question of if, it's already happening. Make AI your secret weapon with Relevant AI Lab at relevantailab.com.

We'll see you next time.

Finally, after months and months of hard work behind the scenes, I am unveiling the new Archon, a massive overhaul of the AI command center that I was working on last year. It is now the first open-source harness builder for AI coding. So, similar vision, but a very different and powerful use case. Harnesses are the future.

It's the layer on top of your coding agents that orchestrates the different sessions. It's what makes AI coding deterministic and repeatable. We'll talk about what that means. And now with Archon, you can build your own custom harnesses very easily.

No matter how you work with AI coding agents right now, you can take that process and bundle it up into an Archon workflow that you can run across all your code bases, even handling different tasks in parallel. Archon handles all of the messy logic behind the scenes to make that possible. And so in this video, I want to talk about what makes Archon so powerful. I want to show you how to get started with it and even give you some inspiration for the kinds of things you can do with it.

It's a very powerful tool. It's actually hard for me to explain everything in just one video, but a lot more content coming on Archon soon as well. So, the main idea with Archon is you can encode your development process as a workflow, no matter what it is. And so I will link to this repo in the description.

I highly encourage you to try Archon today. It's very easy to get up and running. I'll show you that in a little bit. But I have an example here at the top of the readme for what an Archon workflow looks like.

And so every single workflow is just a combination of nodes, where a node is either a prompt that we send into a coding agent session, or it is a deterministic command that we want to invoke. Because sometimes we want to enforce certain things to happen, like context creation or validation, that we don't want to leave up to the coding agent cuz it might forget to do so. So we want to plan, implement tasks in a loop, run the tests, have it do a review, even adding in a human approval gate so we can address our feedback. We can build ourselves into Archon workflows if we want as well, and then ending with a pull request.

And Archon comes with a skill that I'll cover with you in a bit as well. So we just say like use Archon to build this feature, it knows automatically the workflow to use, it'll invoke it for us, and we have the logs so we can monitor the workflow along the way. Archon also comes with a ton of pre-packaged workflows that will immediately level up your agentic coding workflow. Fixing GitHub issues, creating pull requests from ideas.

We have pull request validation and review commands, even one to help you create full PRDs with human in the loop. So, ton of things for you to use and very easy to create your own custom workflows as well. We're going to cover all of that in this video. I'm going to give you enough of a starting point for you to dive right into Archon.

And then if you want a super deep dive, I'm also doing a live stream this Saturday, 9:00 a.m. Central Time. So come be a part of this as well cuz we're going to get really deep into building workflows and running them in parallel, doing a lot of fancy things. But for now, I want to give you a good overview of Archon, why it's important, where we're heading with agent harnesses, and then we'll get into the setup guide.

So I want to start by talking about the evolution that we've seen that has brought us to harness engineering. So prompt engineering, that was all the rage back in 2022 through 2024. It was all about how can we prompt LLMs to get the single best output? And then that evolved into context engineering, which is all about how can we curate the perfect context for a single agent so it can handle a larger set of work.

We give it all the context it needs and nothing more. And now that has evolved into harness engineering, where we're dealing with many different coding agent sessions, all tying that together through a harness. And so up until this point, everything we've done here is dealing with a single LLM or a single agent. Now we are stringing coding agent sessions together to handle much larger sets of work.

And so maybe you've heard of the Ralph loop before, or Anthropic has built a couple open-source harnesses. There are a lot of harnesses out there, but the problem is that they're not custom to you. Maybe it's a good starting point or helpful to create a proof of concept, but what Archon unlocks for you is being able to create your own custom harness wrapping up your own software development life cycle. And harnesses are a big deal.

It's the tooling, the prompting, chaining different coding agents together, everything to elevate the capabilities of a single large language model. And this is really important right now cuz Claude is about to release Mythos, but it's more enterprise use. Like us consumers, there's no way we're going to be able to afford using Mythos for everything. But what we can do is build a harness around Opus to make it more powerful than Mythos by itself.

And this is proven. There have been studies that have been done that if we take a large language model and we just use it to create some code, the PR acceptance rate is only 6.7%. But if we create a harness, like the Ralph loop but probably more custom and better, we can get even as high as like almost 70% for a PR acceptance. And I don't want to get like super deep into the study here, but the point is we can elevate models a drastic amount by creating a harness where we build in validation, we build in a special kind of context curation.

And that's exactly what Stripe did with Stripe Minion. So I covered this on my channel already actually, but Stripe, they ship 1,300 AI only generated pull requests every single week. And they did this by building their AI coding workflow, their context curation, their validation, enforcing that at different steps of the way in their workflow. So they actually built something kind of like Archon, but it's not open-source, right?

You can't use it like you can use Archon. And with the Claude code source code leak, we found that even Anthropic is leaning a lot more into harnesses with agent teams and features they're building around sub-agents. 40% of their code base is just code for harnesses right now. And so that just is a signal for how much it's a big deal right now.

And Archon, it wraps above Claude code and Codex. So the old Archon was more of a tool built into coding agents. That's kind of why it became irrelevant because these coding agents, they built all that themselves for rag and task management. You know what I'm talking about if you've used the old Archon.

But now the new Archon, it sits above the coding agents and it orchestrates them. So the problem is before, you're dealing with AI shepherding. Like yes, you have your skills and commands and you're running workflows there, but you still have your entire process where you're running different skills and different commands, and you have to remember what comes next, and you have to kick off the code review after the implementation. But now your entire process, you can bundle as an Archon workflow.

Define once, run forever, reusable across projects. You also get to pick where you're injecting context. Like maybe you have a skill that you only need during the validation step, or you have an MCP server that you only want during planning. We have that level of control per node with Archon.

And kind of like the example I gave in the readme, here's an example of going from plan all the way to open a pull request. This is the kind of thing you can build with Archon. And I think you really see here how we're building in reliability through deterministic steps, enforcing validation at certain steps of the way, and human approvals. So we have our plan here.

We can even like build this into a loop where we have the agent create the plan and then we give feedback, and we go in a loop until we go to the coding step. And we do that in a fresh context window. You always want to do your planning and implementation in different coding sessions to remove bias. And then after the coding, we go into the tests, and we run this every single time.

Again, we don't want to rely on the coding agent to remember to do the tests all the time. And then we retry and have it fix issues if they come up. Otherwise, we go to the final human approval gate. When things look good, we then open the pull request.

Just one example of how you can take I mean this is a more basic example of what your agentic coding workflow might look like, but there's going to be different things depending on if you're fixing bugs or refactoring or building a new feature. All of those things you can build into Archon. And the big secret here is the hybrid secret. This is what makes Stripe Minion so powerful.

There are certain steps of the workflow that we don't want the coding agent to decide. Like sometimes we want to curate context in a specific way, or run our tests in a specific way. We have nodes for that that we can build into Archon workflows. But then of course, most of the workflow is still going to be driven by our commands and skills, just sending prompts into our coding agents.

We have of course support for all of that so you can take your existing commands and skills, everything, and build it right into Archon. So I hope you can see why this is so powerful, building these kinds of harnesses for yourself. And so with that, I now want to get into the guide for you, how you can get Archon up and running using these workflows in less than 5 minutes. Now, installing Archon is actually incredibly easy because we can use our coding agent to guide us through the entire process.

All you have to do is clone the repository, open up Claude, and then ask it to set up Archon. Because it will automatically load in a skill that we have in the code base here that guides you through the entire process. And I'll do this right now with you just to show how easy it is. We can do this in less than 5 minutes.

So I'll take this first command here to clone the repository, and then obviously we want to open up our coding agent in this repo so we can load the skill. So we'll change our directory, and then we'll open up Claude. There we go. So now in Claude code, I just say set up Archon.

That is all you have to do. And take a look at this. We'll see in just a second, it'll load the Archon skill, and then it'll kick off this process where it'll ask us all the questions we need in order to help us get our credentials set up and then validate that Archon is working on our machine. So first, Archon is going to check to make sure that we have the prerequisites in place, including Bun, and it'll install it if we don't have it already.

Then it asks what repository we first want to use Archon with. So it's important that you add in your own project, not the Archon repo, because we want to use the Archon workflows on a target repository. And don't worry, it's super easy to add more projects into Archon after. And so, just for this demonstration here, I have a reg YouTube chat application that I'm going to be running some Archon workflows in.

And so, I have the path to this repo cloned locally copied here. And so, I'll just go to option number two, specify a local path. And so, then it's going to ask me, you know, like what is the path? Please paste it.

So, I'll paste it in here. And then, it'll register this as our first Archon registered project. And then, going forward, we can add any project into Archon by just running a workflow there for the first time or right within the web UI. I'll show you that in a little bit.

Then, it asks what platforms do I want to set up. So, the CLI is included by default. You can run Archon through the CLI, ask your coding agent to do so. But, we can also run in GitHub.

And then, we could run through Telegram, through Slack. I'm actually going to set up quite a few of these right now. There are a lot of different interfaces to interact with Archon. Then, I will submit my answer here.

There we go. And now, it's going to walk us through setting up the credentials for each one of our platforms. Then, after Archon installs the global CLI, like it did right here, then it's going to walk us through a setup wizard to set our credentials for any other platforms we chose, like GitHub or Slack. We need to do this in a different window, because we don't want to send our API keys directly into Claude Code.

So, it's going to spin up another terminal process for us to enter our keys, not going to a coding agent. And so, it should spin up the wizard automatically and bring up another terminal automatically. But, for certain operating systems, or if you're trying to run Archon in a VPS, you will probably have to open up a new session yourself. So, you just open up a new terminal, and you run the Archon setup command.

So, Archon is now a globally recognized command, because we installed the CLI. So, this is going to work automatically. In my case, I'm just going to go back to the terminal that it popped up for me automatically. So, we can pick our database.

SQLite by default, this is the easiest. Or, you can set up Postgres if you like. So, I'm going to do SQLite. And then, it asks which AI coding assistant you want to use.

So, right now, we mostly support Claude. We're almost done with Codex support, and then we want to add in other coding assistants later, like the Pi Agent SDK and Open Code. And so, I'll go with Claude right here. So, I'll just do space to select, and then enter to confirm.

And then, how do I want to authenticate with Claude? Well, I just want to use my Anthropic subscription. We are allowed to use our Anthropic subscription as long as it's an application running locally using the Claude Agent SDK. And that is the case for Archon.

So, I'm going to use my global auth. Just going to hit enter here. What platforms do I want to connect? So, I'll just select GitHub, Telegram, and Slack.

So, then it'll ask me for the credentials for those. And this is where it's going to guide you through getting the API keys for each of them. So, for each one of the platforms you select, there are instructions to walk you through getting your keys for each application. So, I'm just going to go through this quickly.

You can take your time to get the keys yourself on your own. All right. Then, after you set all of your credentials, it's going to ask, do you want to install the Archon skill in your project? And I would highly recommend doing this, because then, if we open up Claude Code in our other code base, we can use the Archon CLI to kick off workflows automatically.

Cuz then, if we have Claude open there, we can just say, "Use the Archon CLI to invoke whatever workflow to handle a pull request or an issue or something." And it'll know right away to load the skill, and it'll know exactly how to use the Archon CLI. Again, it's important for any CLI to have a skill paired with it, so that our coding agent knows how to use it. So, I will bring this over. It asks for the path.

So, I'm just going to take the path that I copied earlier for the repo. Just paste it in right here, so it knows where to copy it. And then, it'll ask if you want a non-default docs directory. If you have other documentation you want to load into Archon, probably don't worry about that right now.

That's something that we're working on currently. So, I'll do no there. And there we go. Our setup is complete.

And so, what we do here is we actually go back to the first session where we first said, "Set up Archon." And it tells you this here. So, it says, you know, come back here, complete the configuration there, let me know when you are done. So, I'm I'll just say, "Done." And then, what it's going to do now, that I've set all my credentials, is verify that everything is working. So, it'll test the connections.

It'll even test running a workflow through the Archon CLI. I'll come back once that is done. All right. So, it confirms that our credentials are good to go, without actually reading them, of course.

Then, we list out all of the default Archon workflows to make sure they're available. Those are all the workflows that we have bundled ready for you to use immediately to fix GitHub issues, run a rough loop, create a PRD. There is a lot of value that we have built into this right away. And then, it of course runs a workflow as well.

So, it runs our basic Archon assist one just to make sure that the Archon CLI is functioning, and we're ready to use it on any repository that we want. And then, it gives us a summary at the end. And look at how easy it is for us to just start using Archon right now. So, we just open up Claude Code in our target repository.

You can really do this in any repo you want, because remember, when we run the Archon CLI in a repo for the first time, it automatically registers it with Archon. So, we just copy over the Archon skill into whatever repo, launch Claude there, and then we can just say, "Use Archon to, for example, fix this issue number." Or, "Help me create a PRD." Whatever workflow you want to use, and you can build your own, like I'll show you in a little bit. So, because I chose my reg YouTube chat application as my onboarding repo, I copied over the Archon skill. So, it knows how to use the Archon CLI when I ask it to.

And so, I'm going to open up Claude Code right within this repository, and we are already ready to use a workflow. And so, I'm going to use Archon to help me fix a GitHub issue. And so, going into the issue list for my repo, you can see there is quite a few options that I have to work with here. Well, let's say I just want to deal with issue number one.

And so, all I have to do is say, "Use Archon to fix issue number one in GitHub." That is it. I don't have to provide any more context, because it'll know to load the Archon skill, find the right workflow for the job, and then invoke it. And then, our Claude Code can just monitor it in the background. So, there we go.

It used the GitHub CLI to view the issue to get context, loaded the Archon skill, and it decided, "Okay, I should use the Archon fix GitHub issue workflow." It's one of the defaults that we have bundled. It is a very powerful workflow, by the way, because it does full investigation, fixing, and validation before it creates the pull request. And that is the end result is we'll have a pull request that has the final implementation from the Archon workflow. So, we can see right here that in Claude Code, it is running as a background process.

So, we can track the logs here. We can always ask Claude Code to give us an update for how the workflow is running, or we can view the workflow in the web UI as it is running. Okay. So, right here, we are focusing on using the CLI to run an Archon workflow.

That is the most convenient way. But, we also have a web interface. If you want to more visualize what's happening with your workflows, we can also view the logs for the workflows more easily that we run through the CLI. So, I'm going to go back to the terminal where I had the Archon repository open with Claude Code.

So, all I have to do, if I want to start the back end and front end of Archon, is I just have to ask my coding agent to do so. It is that easy. So, I can say, "Spin up the front end and back end of Archon." And it's going to, based on the review, understand the commands to run, and then get everything running for us as background processes. And so, take a look at this.

If I go into the browser and go to port 5178, right? Cuz that's where it says it's running right now, I can see my conversations. I can see the workflows that are currently active. So, if I go into the dashboard here, I can see that it's using the Archon fix GitHub issue workflow.

I can view the logs for it to see what is currently happening. So, this is the node that we just completed. Now, we're currently investigating the issue. So, we started with web research, then now we're investigating.

We can see all the tool calls. So, all the logs from Claude Code, if you want to dig into all the individual actions that it's doing as it's going through this Archon workflow. And so, every single one of the nodes here is either a deterministic action, like a bash command, or it's a session with Claude Code. And again, in our Archon workflows, we can determine when we go from node to node, do we want to start a brand new session with Claude Code or continue the conversation?

We have a lot of flexibility for token management, making sure we keep our context lean when we're dealing with our coding agents. One of the big things that Archon gives us that we were missing before. All right. And while we wait for this workflow to finish, it's going to create a pull request at the end, I want to show you what it actually looks like in the YAML.

Cuz remember, all of the workflows in Archon are simply defined as YAML files. It is so easy to improve the existing ones that we have in Archon, and even create your own. That's one of the last things I'll show you in this video. And so, within the .archon folder in the Archon repository, we have all of the default workflows.

These are also bundled into the CLI. So, when you run the CLI from any other repository, it'll automatically have access to these. So, take a look at this. We have all of these different workflows.

The one that we're running right now is Archon fix GitHub issue. And [snorts] so, we have a description for the workflow. And this is important, because this is just like Claude Code's skills, where the description is what we first give to Claude Code. Like, "Hey, here's a workflow from Archon that you want to use when the user is specifically asking for you to fix a GitHub issue." Right?

We don't want to load the entire workflow into context for the coding agent. That's way too much. It only needs this brief description up front. So, it uses this to determine if it should analyze and run this entire workflow.

We can define the provider up front. Again, we're supporting more in the near future, the default model used for each of the nodes, and then we have the list of nodes. This is the step-by-step that we're going through. So, exactly what I have defined in the YAML document, we can see right here in the web UI.

We see the full execution, all the different nodes that we have, the different branches and decisions that can be made. All of this is a direct visualization of what we have in the YAML right here. So, for this workflow specifically, we are going to first extract the issue number. So, based on the prompt that is fed into this workflow, it's kind of like running a sub agent in Claude Code, we're going to grab the issue number, and then we're going to classify the issue.

Is this a bug that needs to be fixed or a feature that we need to build? Because that's going to determine what comes next, right? Do we have to investigate the issue or plan for a new feature? And we have the prompt here that we're sending into the model for this step specifically.

And one of the most powerful things, I know that token consumption is a big deal right now, especially because of rate limits with Anthropic. One of the powerful things we can do with Archon is specify the model we want to use for the individual nodes. So, certain nodes, like classification, they don't need a lot of reasoning power. So, we can make it a lot more token efficient, a lot cheaper by just using Haiku for our model.

And then we go into the research phase. In this one, we're just going to use the default model. So, we don't specify it here, it just means that it'll use the default model of Sonnet. And we don't have the prompt in line, we're actually using a command.

And so, for every single workflow that's in this folder, and then the commands are sort of like the extensions that we run in certain nodes. So, for example, we have the Archon web research command right here. So, I'll just open this really quickly. It's just like commands or skills in Claude, where it's just a longer prompt that we're going to invoke for this node specifically.

So, we do our web research, and then we investigate if it is a bug or we plan if it is a feature request. That's the the first decision that we can see right here. So, in the case of the workflow we just invoked, it went down the path of investigation because looking at our issue here, it's definitely a problem that we're addressing, not a new feature that we are adding. And so, not like I need to go like in super deep detail for every single node here, but I'm just trying to show you the idea of like how we can take a pretty comprehensive workflow and turn it into this single process that we run with Archon.

It's more than just writing some code. It's doing classification, it's doing investigation or planning, it's implementing, validating, and then creating the pull request at the end as well, and even doing further review after that point. And so, we have a lot more faith by the time this workflow finishes that the pull request is really ready for us to review and merge. And yes, it takes a good number of tokens to go through this many steps, but that's why we lean on being able to specify the model at each step of the way.

We're able to use Haiku for a lot of this here. So, we're just kind of giving more of a set of guidance around the coding agent. That's why we're calling it a harness, right? Like, this is sort of a harness wrapping many different Claude Code sessions to work together to fix a GitHub issue.

And there are so many amazing workflows that we have available for you that you can use out of the box, and then anything that's not here, you can just create by yourself. So, we have the adversarial dev harness as an Archon workflow. I showed this in a live stream that I'll link to right here. We have a comprehensive PR review workflow.

We have one to help you create issues. So, like investigating a problem creating a GitHub issue, idea to PR, this is a very comprehensive one. We have one with human in the loop. We actually have human in the loop with Archon.

So, we can pause at any given node to ask for your input. So, we have this interactive PRD, where it'll have you sort of ideate with a coding agent to create that initial spec for a new application you want to create. We have the Ralph loop built as an Archon workflow. Uh man, there's just so many things that are here already.

We even have an Archon workflow to help you build more workflows. We'll use this in just a second here. So, yeah, literally like no matter what you want to do with your AI coding assistants, it doesn't matter how many coding agent sessions you need, you can bundle it together into an Archon workflow, adding in reliability through deterministic nodes, like you always want to run validation at some point, or you always want to curate context in this way. The sky's the limit for what you can build with Archon.

All right, so I'm just going to keep showing you some really cool ways to use Archon. I'm going to keep using the GitHub issue fix workflow because that is my most used, but there are so many amazing workflows you can use. And by the way, you can just ask it in the web UI here, what projects and workflows do you have? And so, the Archon agent in the UI is actually kind of special.

It has all of the context around Archon, like our registered projects and workflows injected in at the start of the conversation. And by the way, in the web UI here, you can just click add project to give a GitHub URL or local path if you want to register more. So, right now, this is my only registered project, as you can see from the drop-down here. And then here are all of the available workflows, which this will also include any that you build on top yourself, but these are all of the default ones shipped with Archon.

So, now, for example, I can say fix GitHub issue, and then if I go to the issue list here, we're currently handling this one and the other screen. So, we'll just do number three. So, I'll do number three for uh the rag YouTube chat project. So, we don't even have to call out the exact repo name, it can reason based on what we said and what context it has loaded to know which workflow to kick off.

So, take a look at that. It invoked the Archon fix GitHub issue workflow specifically in our rag YouTube chat application. So, it does all of the routing for us from the web UI. And we can go into the logs, the beautiful screen.

This is the exact same view we saw earlier, but this time we actually invoked it from the web UI. Very cool. And another really powerful thing I want to show you really quickly is that we can invoke a ton of workflows in parallel. And so, for example, I'm back here with Claude open in my rag YouTube chat application, and I can just say use Archon to fix GitHub issues 5 7 8 9 10 and 11.

So, the last six in that list there that we haven't touched yet, I just want to rip through all of these at the exact same time. And so, we can invoke Archon directly from our project, or we can have Claude Code open within our Archon repository, and we can point it to any code base if we just give it the path. And remember, it will automatically register Archon at that point. So, now it used the Archon CLI.

Boom, look at that. Six times in a row, all of them are running as background processes, so we can see that right here, and we can continue to have it monitor for us, or even just ask us along the way, like, you know, like give us a status update. So, we can view it in the user interface. Like, if I wanted to go back to the UI now, we can see that we have all of these workflows currently running, or we can just have Claude Code constantly check for us.

We could, for example, use the slash loop command in Claude Code to say, you know, like every 10 minutes, check on the workflows and restart if there's a failure or something, which there usually isn't. But yeah, so all six workflows are running, progressing through the early DAG stages. So, yeah, most of them are on the classify step right now, figuring out is the issue a new feature or a problem that has to be addressed. Super cool.

All right, so I'm back, and we have the pull requests for all of our Archon workflow runs. So, we can see them all complete in the web UI as well. We can click into the logs just see what happened every step of the way. And of course, we can go see our final result.

Going to the repository for the first time here, I'll click on pull requests, and boom, there we go. We have eight new open pull requests handling each of the issues that we handled with an Archon workflow. All right, the last thing that I want to show you really quickly here, and I'm going to expand upon this a lot more in the Archon live stream, is how we can build our own custom Archon workflows. Because yes, there are a lot of very powerful workflows that we have as the defaults for you, like some of that I already showed you in this video, but there's always going to be an opportunity to build your own, something very custom to you, or if you want to take another framework like GSD or B-MAD and bring it into Archon, or take a strategy like beads for memory, like anything you want to do, you can build it as a custom workflow for yourself.

And you can take advantage of this workflow builder workflow that we have. I know it's very meta, but all you have to do is open up Claude Code in the Archon repository and just say, use the workflow builder workflow to help me make an Archon workflow. If I had a dime for every time I said workflow in this video, I would be a rich man. But that is all you have to say, and it's going to automatically load the Archon skill, and then ask you questions.

It'll give you a chance to obviously describe the workflow you want to build, and then it'll run this full builder afterwards to create the YAML structure, and then you will immediately be able to run the workflow on any code base. So, for example, something fun that I thought I would try right now for this video is creating an Archon workflow that essentially takes the idea from beads. So, beads, it's pretty cool. It's an open source repo that gives persistent structured memory for coding agents.

And so, I want to essentially build the idea that we have here into an Archon workflow. So, I'm just going to copy this repo, paste it in here, go into my speech-to-text tool and say, I want to build an Archon workflow that incorporates beads. Basically, just taking all the ideas from beads in a simple sense. So, I want you to search through this repository, understand how it works, and then build this as an Archon workflow so that we can use beads to create any new feature on any code base.

So, there we go. I'll send that off, and it's going to do some research and thinking for me, and then obviously invoke this workflow to create the final YAML. So, I'll show you that when it's done. And there we go.

We have our full workflow created. It starts with exploration, then it decomposes the feature request into individual tasks. and We implement them in a loop with progress tracking, validating everything at the end as well. So, taking a lot of ideas from Beads and building a full Archon harness around it.

So cool. We can also view the full workflow within the user interface. We're working on a workflow builder, so it's like N8N but for AI coding. Super cool.

A ton of awesome things that we're actively working on right now with Archon. It's just in beta. Probably going to be some bugs that you'll find when you try it. A lot of new things that we're going to be adding over the next couple of months here.

It is my biggest passion project. And so, please give it a shot. I think you'll really like it. And also, I would love to see you at the Archon livestream this Saturday at 9:00 a.m.

Central Time. Going to be diving a lot deeper into building workflows and running them and showing you all these really cool features in Archon. So, I hope to see you there. Otherwise, if you appreciate this video and you're looking forward to more things on Archon and AI coding and agent harnesses, I would really appreciate a like and a subscribe.

And with that, I will see you in the next video.

Hey, how it's going guys? In this video, we're going to look at hardness engineering, a new concept to build production grade AI agents. Let's jump in and see what do we mean by hardness. Now, if you look at here on my screen, I already have a claw artifacts that I'm going to walk you through named harness engineering and also a notebook demo to explain that how we can really code it and use it.

You use this harness to build reliable AI agents. Now you know if you look at in last couple of years large language models have become commodities like we don't care about language models anymore because every week we are seeing a new LLM and that does not make a lot of sense because even on the leaderboards Sli or Opus 4.8 8 or GPT solve the new model are not really uh far ahead from each other like on the benchmarks there might be a bit improvement by a percent or two and there's no drastic changes so LLM has already become a commodity so everybody's talking about AI agents and how do we build better agents to solve complex problem or you know do some kind of business transformation that's what's happening so if you look at today when you talk about agents it's not only about model that what LM do you use? It's about the harness. Okay.

Now the harness is basically the real mode. It's the combination of you know your prompts, your rules, your memory systems, your evals, uh the disaster, the recovery management, all the governance part of it. I I mean the architects who talk about observability, the governance, the loop, etc. That's what really matters.

the tools, the permissions, the sandbox and everything the combination that basically combines everything and we call it a harness. Now the agent is basically the combination of model plus the harness. So the model is swappable. You can swap models like you know using routters or any kind of a new LM which is coming out.

You can always you know swap it. So that's what we're going to see and there have been a of course an evolution from prompt engineering to harness engineering. We first talked about prompt engineering where we used to write our uh you know intent into text format and then we moved to context engineering where we realized that prompt is not enough to build better agentic system because agent needs more context in form of system prompts in form of knowledge in form of rag in form of a lot of other things and then now suddenly we're looking at the harness because we are we realized that most of the agents are not going into the production It's okay to build a P or MVP but you know when it comes to enterprises you need control you know you need scalability you have to scale that and for all those right regions we have seen the rise of harness engineering that's important and if I look at some of these numbers like this number if you look at here openai internal product like how many lines of a code a team at openi script in 5 months you know zero lines written by hand even at entropic they're all using agentic IDE or you know coding models uh and they're writing um this many millions of you know uh lines of code has been written by uh these kind of models and if you look at the optimization at of cost as well 954 versus 368 same 100 legal questions you know and uh the expensive run used the uh the most powerful model alone and passed like 14 that you see 14 out of 100 over here plus 18 out of 100 past the cheap run past 18 using a cheaper open model but with a better harness. That's what it does, right?

So you also have to keep that in mind. So that's why harness is important. I'm going to see what do I mean by harness. By the way, uh the harness control loop and suddenly you know before harness we have to talked about loop engineering.

You would have seen a concept related to loop engineering. Now this entire harness control loop that you see are combination of like this this control loop that we call it guides before the act and sensors after the act. Okay. So every harness tool you know it fits in one of these four boxes.

You know either you know if you look at the computational and inferential. Inferential is when AI has been judged. You know you'll have an agent.md file a plot.md file. You'll have your prompt conventions.

You'll have your different skills as reusable playbooks. Now you create skills that you use it later on uh with different frameworks and different agents and whatn not and we have schemas and type definitions. We have our config files. We have our templates and whatn not in terms of prompts and everything.

Then the sensors part which is after the act. We have test compl compilations. We have uh static analysis. We do the checks and the logs most of the obserability part.

Then we have the evals. we do review by another agent or we look LLM as judge evals and human review gate like with the feedback and all. So whatever you are doing you know these loop one thing on the other will exist in this kind of loop if you are building production grade agents. Now this is the boundary map.

Okay. Uh this is the choice button upgrade of course uh ladder. If you look at the first one uh so when you work with generative AI systems you either build a chatbot you either you know either you build a copilot or a workflow or an agent and so on and so forth there are differences fundamentally there are differences chatbots they talk but they never execute you know text and text out ask them questions they return workflows you know each step either sequential hierarchal and whatnot you know so that's the workflow okay agent now agent is the model decides each step uh if you look at this with the help of this harness and harnesses everything the permissions the sandbox you know the event logs the entire obserability part of it uh and the events part of it so and that's can also be used later on to align the agent so that's called agent alignment right so this is something that you should look at it over here I'm not going to go deep into it seven responsibilities of a harness okay and none of these live inside models because models are anyway static now the whole reason we are using agents are for dynamicity right you need dynamic things you task to be performed. You need you know uh connectors to connect with tools and what not.

So the model and versus this execution tools context life cycle observability verification and governance. The whole idea is to build reliable production grade agentic system and without all of these you can't do it and that's why the entire harness loop is important. These are the seven responsibilities of a hardness starting from execution you know you look at the resource limits you look at the token maxing you look at the timeouts and whatn not then we have the tools about for discovery you know you will use MCP for connecting with MCP servers then the context what the model sees each round the context engineering everything that we do with the context pollution the context overhead and whatn not the life cycle you know when to start when to interrupt you'll write custom rules uh you'll have privileges and whatn not. Observability which is very important that entire telemetry thing event log stresses when the agent is failing latency how it can improve so on and so forth.

The verification like you know you look at most of the time see sometimes agent makes blunder you know they delete your production database even if the permission is not given and what not. So how do you look at this particular thing and the governance where we look at the permissions approvals audit and budget these are the seven responsibilities of a harness. Okay. Uh model only does the reasoning part nothing else you know to that.

Now the feedback flywheel the goal is to not have better answers for the team that gets smarter every time it uses AI. That's the agenda of building better AI system that can improve by its own or evolve by its own. This is how it works like four thing that you see over here. So you know it captures signals write back.

So it write backs and I'll show you. It write backs all the mistakes and learn from it. the alignment of agent which is important. So we have to keep that in mind.

Uh here we have a few things technical dev debt and cognitive debt. Uh what degrade symptoms repayment how do we improve it? This is mostly for HDLC. Uh even if you're writing prompt make sure that you are writing spec driven prompts right prompt which are spec driven and that's what GitHub does it by the way.

GitHub has an HDD based framework. Version control of your prompts are important. Giving identities to your agents are important. you have to review them like code because you have to do dynamic prompting.

You cannot just write prompt one time and keep relying on that. Uh you always have to test the AI output refactor and write learnings backs and whatn not. Now I do have some code I'll show you that. Okay, if you look at here uh I'll show you an example.

uh this is how you build a starter harness you know on your report you'll have an agents.mmd and I'll show you that which will have few instructions few rules you know you'll have your decisions you'll have your failures and then you'll have your check harness and then you'll have continue you'll wire that into your continuous integration pipeline okay you triage that right classify justify uh this is available on my github by the way these slides I'm not you know covering it uh as I said that models every week we are seeing a new model and so models are not anyway any anymore the mode the mode is the hardness the mode is how do you build the agent so you look at the eval score you look at the review time uh the incidents uh if the agents are failing when what kind of tools fail and what not right so that's important the budgeting is important how do you optimize it how do you optimize the context and there are repos like headroom for example that basically helps you optimize your uh entire context thingy when you're building a workflow so that's important uh this is okay these are some honest caveats that I found out working on it that it's not as simple that we see okay so keep this in mind that why you know it's not worry about model anymore you look at the hardness part of it and I'm going to show you a quick demo over here what I mean by that so if you look at this notebook and I'm running it on Google collab you can also do it like anywhere you just need a notebook or a Jupiter uh notebook to do that now as I said a today are built with the harness with a loop right model plus harness harness Harness is everything around it. Model is the brain performs the reasoning. Harness you have right to rules and tools, permissions, memory and everything. Okay.

So most it's behavioral thing. Most of an agent's interesting behavior the lifts in the hardness not the model. Okay. That's important.

Now system and who decides the next step chatbot workflows agent plus harness and whatnot. Right? So I'm using openAI model but you can feel free to use any model you want. So I am using lang graph uh this is an this is a this is an agent that we have created in lang graph right which is graph based stateful and you know we are using openi models and we you can see we are using gpd 5.1 over here right now the demo project right this demo project is very simple just to explain you how harness works what do we mean by harness engineering you can see one bug one failing test a tiny three file project there's an average function You just enter the division that you see it over here.

So average list of one two returns one instead of 1.5. You see this there also a protected file config.py that no agents would ever touch. That's basically I'm a rule that I'm writing. We'll use it later to show governance in action.

So I'm going to show governance and how what do we mean by AI governance for agents right? Uh a very small part of it. Governance is a very broad field. Uh so we're not going to cover everything.

Uh we also define our first harness component. You can see run test is a computational sensor which is a deterministic pass fail signal that does not depend on anyone's opinion. So if I run this you'll see it will say failed. Okay.

So if I just run this you can see test passing false uh failure test average test calculator assertion error. You can see it over here. So this is on the purpose. Okay.

Now so there are different levels. You can see level zero chatbot uncertaintity lives in the in the conversation. Basically a chatbot just you know regions over text you give it. Okay.

It can explain the failure brilliant brilliantly but it has not read a file and a command or fixed anything. So the first engineering rule of the whole stack never treat a description of an action as an action that happened. A chatbot that says I fixed it is hallucinating a harness it does not have. So keep that in mind.

Right. So if you look at here, it says this pi test run fail. The test log which is a dictionary explain the most likely root cause in three sentences. You have no tools.

Don't claim to have run or fixed anything. Now let's just run that. So we're going to we're going to do lm.invoke. We using length for that with an openi model.

You can use any model you want. You can see it says the average function is returning one stand of 1.5. Then when called with one two so it's likely doing integer division otherwise truncating blah blah blah. Right?

So here what we have defined if you look at here you say average uses integer division so average one two returns one instead of 1.5 but if you look at here the first one function doing one instead of 1.5 and it says don't claim to have run or fixed anything right so that's the first one chatbot one about the hallucination now level one the workflow the steps are known so code decides so you can look at the when the procedure is fixed run test collect log summarize write it as a graph where your code owns the control flow the LM appears as one node summarizes and never steers right so look at this state graph lang graph what to do so when you use state graph in lang graph we explain what to do the agent can do that right so if you come down we have a class diagnosis state a node you can see code decides this always run first this particular thing run test and there's a summarization node that you can see and then we just compiling it over here with this workflow And just let's run it. So look at here. It says it will take a bit of time. The average function is returning the integer one instead of the correct floating point average 1.5 for the input likely due to the integer division or incorrect calculation.

Very nice. Now the next level agent the next step depends on live evidence. Fix the failing test can't be fixed. Can't be a fixed flowchart.

In general the cause could be any of 20 things. We hand the model tools and let it choose each move. Read, hypothes, hypothesize, edit, rerun and verify. So we are not giving it no rules, no permissions, no audit trail, no budget.

This is the naked agent. Every demo source and thing you should never say like most of the people they say this. They just go white and they just do it. So agent is not an upgrade.

The complexity cost you pay for uncertaintity. You are doing token matching. You are building single multiple agent without harness. Right?

That's what we do. Let me just run this over here. And it takes a bit of time to do that. Okay.

You can see I found and fixed the bug. All test now pass in calculator.py. If you come here, there's a file called calculator.py that have been created. So if you see this, it say in calculator.py the average function used integer division.

This is your function. This returns an integer by truncating any decimal part. So average incorrect blah blah blah verification and this and output you can see now it probably worked the model is good but notice everything you cannot answer right now. Did it touch any file it should not have?

We don't know permission no log. Okay. And this is my calculator.py file if you see over here which will have a basic Python function that you see define add AB return AB and the number of it and use two division. So we get a float result when appropriate.

So this is a simple Python function you can see it over here the same thing verification I ran the test suit before fix one failed one pass after fix all test pass so it did not you we don't know that because there's no permissions and there is no log what did each step cost no budget no m okay is is I fixed it actually true you trusting the model's own claim we don't have any fact for that now those these all three or four gaps that you see that's what a harness harness closes it. Okay. Now we are looking at the last level of level three agent plus harness. The model proposes but the harness controls.

So we will rebuild the same agent with four layers around it. Harness layer you will you will have a guide before the act. So an agent MD rules injected into the system prompt. You will write system prompt right to give it to your agent.

Now you're going to have an agent MD file a markdown file and the vocab is inferential guide. So this is the G4 while you just for inference the governance layer protected file policy enforced you know we are enforcing a policy in the tool not politely requested in the prompt. Basically this vocab is for permission. So we are also giving permission observability every tool call appended to an event log.

So you can later audit it and see about the costing you know what went wrong and so on and so forth. And the verification once the outputs of the task are done or generated the harness reruns the test itself. So it never trust done it basically validates that and that's the computational sensor. So we are looking at the inferential guide we are looking at the permissions we are looking at the audit and we are also looking at the computational sensor.

So the harness basically creates all these four things wraps around your loop. Basically it's a loop. So it you know it wraps the loop right and then the agent makes sure that it's not going to make any mistake. So we're building a very reliable agentic system with these principles right now.

That's what we're going to do. So let's do that. So you can see the calculator.py file. I'm going to write some bug introduced test passing false.

You know this is my file and this is your test calculator file. See now okay we have changed the file has been changed. Now this is test calculator.py file. I'll show you the file by the way everything.

Okay, now let me just minimize this guide. You can see the guide. We are creating this agents.m MD. Okay, so I'm just going to run this for now and let's show you harness agent ready.

You can see it says project rules for AI agent. Look, let's look at agents.m MD. And what we are doing in agents.m MD, it says project rules for AI agents. Never edit the config.py file or agents.m MD.

They are protected. So the agents cannot even touch this config.py. after any code change run the test suit before claiming success. So it has to run those test suit that has been written keep diffs minimal fix the bug don't refactor or reformat if a rule blocks you say so don't work around it so basically you are writing some instructions to enforce that on your agent that's your agent MD now the fun part we give it the honest task plus a booby trap like you know we also ask you to repoint the production API endpoints is this which lives in the config.py file so this is my config.py pi by the way if you look at this here this is an API endpoint and this can be anything it's protected from production right so let's do that so I'm just going to just give me a minute so let me just run this what we are doing here if you look at harness agent invoke and we're just running the running it and we'll see the output here is what I and what I can't do now see the output fix the failing test bug fixed the failure was in average function that we know in calculatorpy already so it's fixed the bug is fixed and all test now pass it ran the test suit for us before it just said okay I fixed it it has to ran the test suit it has to give you the validation the the fact and then you know okay it has done it or not right requested change to config pi current content of config pi project rule one says never edit config or agent they are protected so because of that rule I'm not allowed to modify config by including changing API endpoints which is fantastic.

So it it cannot change it because we have you know enforced the policy on that. So that's that's how you create the harness. Without harness your agent will you know go into haywire right? It will have no idea what to do by the way.

Okay. And now let's look at this uh this thingy the cost thing the harness report. I'm just creating a harness report. You can see the verified fix harness ran test itself pass protected config pi untouched.

Yes. Tokens in and out. If you look at this cost of this task, audit trial and everything is over here. The config the fail calculator blah blah blah.

Now you can see both both are the highest working. A denied entry, the agent tried to config which did not work and no right attempt at all attempt at all which is fantastic. Right now the feedback flywheel how do we you know write the lesson back the cognitive debt that will accumulate right and how it can be used to align the agent better and performs better in future that's what we're going to do here so let's just do that I know everything will be rewritten and all okay so you can go through it later this will be available on my GitHub repo now this is what we have built agents.mmd in the system prompt which I told you inferential guide before the act type of test bio run test uh the sensor computational protected event log recussion limit so on and so forth so keep in mind when you're building agents don't worry about selecting a framework or selecting which is a better model of course you can look at the leaderboards and LM arena or any kind of benchmarks and just select a model focus on the harness guys the engineering behind the harness how do we going to create a control loop how are we looking at the governance and observability part of it. If you can do that then only you can build better you know agentic system.

So this notebook will be available on my GitHub repo and this claw artifacts as well. This will also be available basically this is you know in format of PPT that you can use it you know uh for your any kind of session or workshop if you are taking it but keep this in mind agent equals model plus harness right that's very very important uh for you to understand uh this is what I wanted to show you in this video if you uh like this video please hit the like icon if you have not subscribed the channel yet please do subscribe the channel guy that motivates me to create more such videos in the future. Uh here is my channel AI Anytime. Go and subscribe it.

Here is the GitHub account uh GitHub here AI anytime. More than 400 public repositories that you can utilize open source project. If you have any thoughts, feedbacks, let me know in the comment box. You can also reach out to me through my social media channels.

Find those information on channel banner and channel about us. That's all for this video. Thank you so much for watching. See you in the next one.

[music] >> Hello everybody. Everybody's head turned up. Hello, hi. How was lunch?

Was it good? You didn't like it, no? It's like British food. Anyway, hi.

I'm Tejas. I'm I'll be your first speaker this afternoon. Tejas, that's pronounced like Tejas. Don't worry, I'm not Hopefully my my joy in AI is and I've had the privilege of working at a number of different places over my career in one form or the other.

It's just been an absolute joy to learn from the best. Today, I'm a AI developer advocate at IBM IBM IBM where we we do things with AI, believe or not. We train frontier models, we build harnesses. It's really it's a fun lab to work in.

But that's not what I'm here to talk to you about today. Today, I'm here to talk to you about AI harnesses. AI harnesses. Before I move forward, I would love to just have a show of hands.

How many of you are like confident in your understanding of AI harnesses? Like you're like I could present this on stage today. Look around. Look around.

No, seriously, look around. That's why we're doing this talk. Okay, this is my hope. I want you to If I ask you this at the end of the talk, right?

I want you to be like, oh, I I I get it now. That's the whole point. I have literally nothing to gain from this other than I I shared knowledge, okay? Because also this term is kind of everywhere.

You may have heard it used like 52,000 times today. And it means different things to different people cuz like in the machine learning world, it means like a glorified test suite for machine learning models. But in the AI in the AI world, it means something different. And so today, we're going to understand this in detail.

It's a deep dive, but it's 18 minutes long. So let's let's move forward. Um I want to start by talking about why harness. Like why do we use harnesses?

And the reason for this is because we pay rent to companies that give us compute, give us inference, give us tokens in return. Some of you maybe work for companies that have frontier models like Anthropic or Google or whatever and you maybe What was the term? Token billionaires, yeah? Um Um Um I'm not that.

I am maybe with Watson models, but but the vast majority of us aren't token billionaires. We we pay rent. We literally $20 a month for Claude Pro. And then you get a context window that's limited and you get like, you know, you you don't get the full hog, so to speak.

And the model you rent is is a black box. Like they could at any time, I'm not saying they do, but they could if Opus is somehow not available, they could serve you Sonnet even though it says Opus. You would never know, right? And so it's just a big There's too many variables that we cannot control.

So why harness? Because the name of the game with harness is reliability. Um I really hope I'm not supposed to stand in front of this white line and then I'm just not in the camera. Anyway, whatever.

It's reliability. It's it's making sure that the agents we build do what they do, period. Irrespective of the black box model, irrespective of the of the the thing we rent and so on, okay? Now that we understand why harness, let's talk about what a harness even is from first principles.

Like let's let's take it all the way back to harnesses that we know and understand. If you've ever, you know, climbed a mountain or something or you've seen someone This is a harness. It's like mountain climbers literally will like harness themselves to what? To a mountain because it's stable.

And they can't go off the rails, literally. They they anchor themselves in something stable so that they can't drift too far. Okay, that's that's what a harness is by design. When you have any dog owners here?

You have dogs? You you walk your dog on a harness, okay? That's why? Because your dog doesn't go and bankrupt you with tokens.

Okay? That's what a harness is. But the problem is if we think about what harness, there's really two types. There's one from the machine learning world, which as I mentioned is kind of like a test suite and a test runner.

You would give a model some inputs and you see the quality of the outputs. That's not This is not ML engineer Europe. We're going to talk today about the agent harness that is common in AI engineering. Okay, so what what is an agent harness?

An agent harness and and this is kind of the money shot here. The agent harness is I'm not making money off this. It's just an expression. The agent harness is everything around the model that gives it grounding in reality.

It's literally the thing that ties it to a stable environment, okay? An agent So Claude code, for example, can be considered an agent harness. And some of you would say, oh, no, it's a coding agent. Absolutely, it's a coding agent.

But it's a harnessed coding agent. An agent harness has more or less the same typical suspects, moving parts. Number one, it's got a tool registry. Almost like so Claude code, cursor, codex, they have tools to read from the file system, to write, to execute bash commands, right?

They have a tool registry. They have a model and some of them allow you to choose a model, some of them allow you to not. They have a model. They have primitives for managing context.

Almost every harnessed agent runtime today will compact its own context, right? That's that's that's the job of the harness. Guardrails are another part of a harness. For example, max steps.

Anyone using max steps? Do not do more than five tool calls. That's a guardrail. And so if if you do that, you just kill the kill the run, right?

An agent loop is another part of an agent harness, which is crazy. This is what some people I've spoken to preparing this talk will say, wait, isn't a harness just the agent loop? No, it's the stuff around the agent loop. In fact, it could be a loop around your agent loop.

It could be an NM loop. And we'll look at that a little bit in some code. And then finally, there's a verify step. This is, for example, in a coding coding agent, after the work is done, a verify step would be, hey, let's let's run lint, let's run tests, let's make sure nothing broke, right?

So almost every I'll use code coding agents as an example, but you could have a harness for anything. And it's it's amazing cuz it really grounds black box models in a stable environment that you control, okay? I'd like to show you you you a demo. And what we're going to do together is we're going to build a harness, a bare bone baby's first harness.

Let's call it a poor man's AI harness together so we understand from first principles how this works. We're going to build a computer use agent that has a job. The job is go to Hacker News and upvote the first post, okay? It's a computer It's a browser use agent.

We're going to use a really bad model intentionally. We're using GPT-3.5 Turbo, which is like 2023, right? But we're going to harness it so that it can actually do the job. And we're going to save money.

So let's I've spoken too much. Let's just get into the demo. Uh Uh Uh And and so let's Welcome to my project. This is my project.

Hello everybody. This is the entry point. Can you see that? Is it too Yeah?

You want it bigger? Let's do bigger. Okay. So this is not Actually, this room is too bright.

Let's do light mode. It's I It's not my nature, but sometimes. That's better, yeah? Okay.

So we have we have a model and we're trying an old LG Sorry. We We We We shouldn't have seen that. No, we'll we'll try an old model. And this is the prompt.

This is the This is the task. This is literally my prompt. Upvote a story I just described it. For the purpose of this demo, we will not change the prompt at all.

Because a lot of us think, hey, my agent is not doing what it's supposed to do, so I just need to prompt it harder, right? That's not always true. I need to change the system prompt. We're not going to touch any prompts here.

We're just going to build a harness and the outcome will change. We we log some things to the console and then we start a browser session. Okay, what's a browser session? It's literally just Playwright.

Not Playwright MCP, like Playwright Playwright. Where this is just a class I made with an open method that launches Chromium and gets a context and makes a page. And then navigate We're just literally calling the Playwright functions, yeah? This is This is just traditional engineering.

So we create a session, we open the session, meaning a browser window in a context. And then we create our tools and we give that browser session to the tools. And we create a context and we give the task, meaning the prompt here, to the context. Now, create tools is literally what it sounds like.

It's here. There's just some types and create tools is a function that takes a browser session and gives you like tools. And these tools are not I didn't invent this. This is from OpenAI's SDK, okay?

So you have the name, the description parameters, and execute, the way you actually call the tool in your runtime. And and there's just tools for I made this. It's very easy. Um So that's my tools.

And then create context, you may think, whoa, context engineering. Absolutely not. It's This is my context. There's nothing here.

It's just a system prompt. Literally, the most basic system prompt and the user's task. This is basic basic. And then we have run loop, which is just running the agent in a loop.

So what it's doing here, we can actually just look at this, too. While true, so it is an agent loop. And we get a response from the agent and we see if the response says stop, meaning if the LLM says, I'm done, then we return the value. If we get any other response, we don't do anything except add these events into a trace.

So we just push history into a big list of history. Does that make sense? And so that's all we're doing here. This is just a loop where we just collect events until we're done, okay?

So this is super basic. Now let's see how it works. So I'm going to come over here and I'm going to do Are you okay, sir? Do you need water?

I'm going to npm run agent. Um And so it's going to open Chromium. It's going to Okay, Hacker News, so far so good. Click upvote.

Oh, no. So we we hit a login screen and then it kind of panicked and crashed. But look, it it lies. You see this?

Um this is a problem. And so what's the solution? Prompt it harder? No.

Change the system prompt. Always login with these credentials included in the system prompt. No. So how do we then solve this?

And look, we because of my logging, we can actually see it just clicks the upvote button and then considers it a success. It doesn't verify. This is the job of a harness, okay? So now incrementally, we're going to slowly start building a harness.

Um And so, let's just move I'm not going to write code here. I'm not going to live code because we don't write code anymore. We inspect diffs. Right?

Anyone write code by hand? You don't Maybe actually you do belong here. Anyway, so um I'm kidding. So, this is um This is the first change we're going to make.

This was our index file. And we have this run loop that I showed you, but now we're going to add one thing to it, which is default guardrails. We're going to create some guardrails, okay? Um what do our guardrails look like?

Well, let's go and look at it in the editor um with guardrails over here. And so, we have some types, but these are our guardrails. We have two. Max iterations, meaning if you do more than six steps, I'mma kill you.

And max messages, meaning if you have more than this many messages, I will compress the context. These are just guardrails, okay? A little utility to combine them, and we just we can compose them here. We could do like as many as we want.

So, now let's go back to our changes. That's the guardrails. We if we go back to the agent loop we actually use the guardrails here in this diff. And so, we include the guardrail functions, and we can see that here what we're doing is we're checking how many messages have we accumulated, and we just like trim the context if it's too much.

Um but what I did want to show you is here at the end um we we push context size, which is some more metadata about what we've done with our guardrails, okay? Um our context compressor is extremely basic and extremely naive. This is what it does. Um let me actually open this with syntax highlighting to spare Um this is what it does.

So, what we're doing is if we always keep the system prompt and the user prompt and the most recent two messages. So, if the guardrail is triggered, we always remove everything after the system prompt and the user prompt in the middle, and we keep the last two messages. This is super naive. Don't do There's better ways, but this is where babies first.

We're We're getting there. So, we we're starting to have a harness, but it's not called a harness, but this is really like a pregnant harness. Like it's almost born, okay? And so, what we're going to do is let's just call it a harness now.

So, I'm going to show you another diff where we Here, check this out. Index, we've deleted almost everything. Um and we've moved it into this file called harness. Let's go look at our entry point now.

It index, it's it's all gone. So, the prompt is there. But this is it's like 19 lines of code, and we just have run harness. We've taken all the logic from here and hidden it in a function called run harness.

And as you would expect, run harness does exactly the same thing as we did in the index, okay? Nothing new is here except maybe like a print function, which is just console log. Is this clear so far? Yeah, we just moved stuff.

Now that we have something called a harness, we can actually use it. And let's solve the problem of lying first before we solve the problem of logging in as me. Yeah, because it says I I upvoted, it did not. I want to know.

So, what we're going to do is we're going to add some guardrails and and have it tell the truth. Like if you failed, tell me the truth. Um how might we do that? Well, we'll check it out here.

So, many many things changed. Um Or not. I don't know. Hang on a second.

Yeah, okay. Did Many many things changed. So, we run harness and we added a third argument here, which is a verify step and max attempts. Max attempts goes to our guardrail.

So, if if you took more than three tries to do this, just give up. And if we go to the harness, we added a lot of things um that are just manual code. This is not different prompt. This is my logic.

Um the main logic is run harness no longer wraps over the code we moved, but we moved that to a different function called run harness attempt. So, if we if we come to run harness Let's go here. I need to check the branch out, sorry. Yeah.

So, now if we go to run harness attempt, we'll collapse this. I'll collapse this. I'll collapse all of these. And if we go to run harness attempt, now this is the same thing from our index.

We just moved it into a function called run harness attempt because our main run harness is just a loop that runs no more than three times, okay? Is this clear? So, we're just enforcing the max steps, but at the harness level for safety. Um then we have run harness attempt that calls it.

We have this function called verify successful upvote. I wrote this. This is deterministic. That's what I want to show you.

What does this do? Well, we see if You remember we were tracing in the agent loop, we're just adding history events. So, we reflect on that, and we see if there was a browser click on the upvote and if it's successful, but really successful, then we say true. But there's a huge butt here, which is we have now cases for failed login.

If there's a tool named harness auto login, and if the message starts with failed then we return early and we say no no, this failed. We're We're removing the lie, okay? Similarly, unrecovered login redirect. We look over our agent loops tools that we've been pushing into.

Um and if we see that the harness auto login didn't run and now we're on the page that is the login URL, then again, we just fail. Okay? Um and so, we're what we're doing is we're just adding like if this happened, if this happened, you just just fail. Return early.

Is this clear? This is what a harness does. And so, let's run this now with the harness. Uh npm run agent.

And And And now it's going to go on Hacker News, and we're going to repeat the same cycle. Okay, it's going to come here, and now it's still failed, but look, it stopped lying because our harness checks the tool history and actually sees what happened. This is what a harness is supposed to do. Great.

This is already like half the battle won because step one to solving a problem is admitting you have one, okay? Test-driven development vibes. So, now that we we're failing correctly, we can succeed. And I'd like to show you that in the last diff, and then we'll finish the talk here.

So, number four. Um we have a whole new function. It's called login handler. Uh I'll add some syntax highlighting here so you don't go blind.

Uh but here, create login handler. This is This is all it does. It runs every agent loop just before we push to the traces, and it This is what it do It checks the browser session's current URL. And if we're not on a login page, it just says cool, I don't I return I have nothing for you.

This computationally is not costly at all, right? If you're not on the login page, but if you are on the login page then it we fill in a temporary This can be an environment variable. It can be secure, you get the idea. But we fill in credentials and submit the button programmatically from the harness, not from the agent, deterministically and securely because this file has access to any secrets I want it to, right?

And so, this How How is this called? Well, this is called in the agent loop. So, if we go back to our agent loop and notice we were pushing traces, yeah? This is where we push the traces.

Just before, if we have a login handler we call the login handler just before this in the agent loop. What does the login handler do? Well, if we're not on the login page, it does nothing. If we are on a login page, then it quickly will inject credentials and submit the form and then take you back.

It will also add, as we can see here, it pushes a message into the queue saying, "Hey, I'm the harness. I logged in. You're good now." Is this clear? Yeah?

So, the the harness is is is literally harnessing the agent to something stable, something deterministic. That's what it's for, okay? Let's run this now and see what happens. So, npm run agent.

It's going to open Hacker News, and when it gets to the login, now that harness step, it logged in and it upvoted the first one, and it closed. Amazing. So, successfully upvoted a little snitch for nilux, uh rank two, uh succeeded after six iterations, and I can click this and go into Hacker News and actually see indeed it was upvoted, um and it I can unvote now, which means it was upvoted, right? So, um the agent used the computer, logged in as me with my harness that I just made here on stage.

That's the purpose. Is this clear so far? Do you understand the role of a harness? Look at you nodding.

This is music to my ears. Fantastic. Something to my eyes. I don't know the It's beauty to my eyes, kind of weird.

We don't have a expression for that. Let's land the plane. I'm done. I think my work here is done.

What does this look like in practice? Why Why do I care so much about harnesses? Because they run the world. Models are non-deterministic.

And you want to do more with less. You want to use a cheap model. Use like Quinn or something, or even something smaller. Use GPT-OSS.

It's free. And with a great harness, you can go very far. That's why. At IBM, we create a open-source project that we deploy in the enterprise that allows very large companies, huge companies, in their private like data-sensitive areas to perform rag operations on all kinds of things, teams, calls, and PDFs, and invoices.

Um We We build It's called open rag, and it's it's rag I don't know if rag is cool or not anymore, but open rag has a hell of a harness that provides enterprise-level security to like asking questions with internal very very siloed data. And And that's kind of where the harness engineering comes in. So, let's summarize. We covered a lot of content.

Was it a deep I think it was a deep dive. It was a deep dive in like 18 minutes or so. Um we went pretty far. I It's not It should not be lost on you that I did not touch the prompt once.

I did not change the system prompt. We just built a harness and the outcome radically changed. And of course, we can add secrets, we can add tokens. Um yeah, we did a lot.

In the end, I hope you understand what a harness is, the value it can present, and how you can use it. What's next? Um Look, I I don't have a crystal ball like everyone else here, um but it's not lost on me that 2025 was the year of agents. Yes?

Uh 2026 is the year of harnesses, I'm pretty sure. Everybody How many times is this word used here? Um I think I would hope I think it'd be pretty cool if 2027 was the year of dynamic on-the-fly generated harnesses. How cool would that Like you tell an agent, "Hey, do this for me.

Buy me a flight ticket." Whatever it may be. And then before doing the work, the agent creates a harness. This similar to plan mode. Any of you using plan mode?

But But on steroids. The The agent creates an actual harness, self-aware. It knows, "Oh, I can maybe hallucinate here. I can maybe" Creates a harness, does the job, and returns back to you, guardrailed and everything.

That is so cool. Dynamic on-the-fly harnesses. I would I I this is honestly the next logical step towards AGI, and I would love to see it. I don't know if this is just me being uh you know weird guy with ideas, but um I think that's kind of the direction.

So, with that um I'm almost out of time. I would be really remiss if I didn't spend the last like 30 seconds saying thank you so much. The slides are on GitHub uh as uh am I uh and so I'd love to chat more. Thank you.

>> [music]

We all know Fable 5 was one heck of a model. Then it got pulled. But if you know how to build a system that makes any LLM perform at that level, then you'll never have to worry about chasing the next big AI breakthrough or a company or government rug pulling you ever again. And that's what I'm going to share with you in this video.

What an agentic harness is, how they work, and how to get started building your own so you can get long horizon creative autonomy without Fable 5. Plug in any model you want. So in this video, I'm going to introduce you to what an agentic harness is. I'm going to show you one of mine in action right now.

And then we're going to build one on our own using a template or simply prompting our AI to do it. And then I'll give you some more tips on getting started. Before we begin, I just want to encourage you to hit that subscribe button because this type of sovereign AI usage, like the real foundations of AI, is what I like to explore on this channel. I don't think anyone should be dependent on the shiny objects, the next big hyped tool out there, the next big hyped model.

We should know how to use this intelligence from the ground up. So when the rug gets inevitably pulled out from under us and we lose the best model ever, we're not completely screwed. Just a few hours before Fable was pulled, I built a 240 listing directory. Fable came up with the SEO AEO strategy.

It scraped the whole web and it created the entire architecture and infrastructure of this sovereign individual-minded directory. It categorized things. It made an LLMs.txt. I rendered the whole thing with Astro.

And it was just really impressive So I got really excited about Fable. And then Fable was pulled and builders, including myself, were really bummed because it did such a good job at long horizon autonomous work. But the truth is, we don't actually need Fable to do long horizon autonomous work. Opus, other models, are all intelligent enough.

What you need is a good harness. An agentic harness is the rig that keeps the building loop for an AI going. When you know how to build your own harnesses, you're not going to be as vulnerable to the rug pulls that might happen due to a government or a company or whatever it might be next. So, in this video, I'm going to show you all about agent of harnesses, how do they work, and then we're going to set one up to keep building my directory for the next steps cuz I can't use Fable anymore.

So, let's talk about the loop. Now, what an AI is going to do when it's building an agent, it's going to read what the prompt is, it's going to pick a tool, it's going to run the tool, it's going to check it, and then it's going to find out if it's done or not, and then it's either going to start over or move on to the next thing. That is what the AI agent loop looks like. Okay, so let's unpack that.

You have AI, the intelligence, the LLM, that is the brain. Okay, that's here in the middle. But, the harness itself is going to be the body, the fingers, the arms, what it needs to actually get stuff done that actually is good. So, you have the context, which is like a knowledge base, you have the tools, which might be MCPs or using the terminal.

The harness has systems for verifying that the work is good. It has systems for determining if the work is done or not, and then it has a memory, like what actually happened and what needs to be next. So, in essence, an AI agent is really a model plus a harness. And then, this is really important to remember, in my opinion, because Fable was the model.

Okay? The reason Fable was so freaking good because the Claude code or Claude app is the harness that makes the agent so freaking good. So, when you don't have Fable anymore, what do we do? Don't fret because we can create this on our own, and in some cases even make it better.

So, a good harness can run with any model. You can run it with GPT, you can run it with Gemini, you can run it with Claude, and perhaps most importantly, you can run it with open-source models. The model is probably not going to be run on your local machine, though it might be able to, especially maybe a year from now, it might be completely normal that we run harnesses on a machine. I've been using Goose a bit with local models and Odysseus and we're getting to a point where this is going to be a thing.

But for now, what's important to understand is that while we're renting the intelligence, probably, we can own the harness. And I'll also call it a rig in this video cuz that's what we call in the AI companies Academy. You can own the rig, which is like the software, the body, the robot that you can plug the intelligence into. And if you can own that, then you're not as when the government tells Anthropic to shut down its model.

Okay, so let's address the two problems with letting the model itself hide inside the harness, which isn't exactly what Fable was. But for sake of the metaphor here, let's go with it. Problem number one is you rented it. So, perhaps it's not Fable 5's intelligence that made it so impressive.

Obviously, Opus 4.8 could have made that directory, but the harness, Claude Code, which Fable 5 was designed to really use and maximize, was the body for Fable. So, Opus can still use Claude Code. It's just not going to be as good with it unless you set it up to be as good as it can be, which we're going to do later. So, imagine the harness lived inside of Fable, you rented all that.

So, if it's no longer available, it's gone. Problem number two is you have no leverage when it's not yours. And this is the main point I'm trying to drive home here, is you want to know how to build these things to plug in any model, whether it's local, remote, whatever. So, then the next time a big black swan event happens, you're not screwed over.

I'm just going to keep saying that. The Agent of Carnus is your ticket to running powerful AIs without worrying about where the intelligence is coming from. In a previous video, I mentioned Odysseus. That is a great workspace.

It's not quite a harness in the same sense we're talking about here. Here, we're talking about being able to one-shot entire projects, long-horizon autonomous creative work. So, you can build the harness yourself and then run it with pretty much any model. Now, an agent, like I mentioned, is essentially a model plus a harness.

You can still plug a weaker brain into a great harness and it'll still ship. It'll still do a pretty good job, the better your harness is. You could have a genius brain, but no harness, and it's still just going to be a chatbot. It's not going to do anything.

It needs to have the agentic capabilities to do stuff. So, to me, this is like the elephant in the room. Fable 5 was amazing because of how it works with Cloud Code. So, what I've been saying for a long time in the AI Captains Academy is you want to build your tools, your workflows, your pipelines, your harnesses with frontier models, and then execute with open source.

Why? Because with the frontier models, Opus, GPT, Gemini, it's going to cost more, but it's going to do a great job. Even in Odysseus, you have teacher mode, which will call a frontier model to create a skill or procedure that the open source model just doesn't do well. But then, once you've got a quality harness built by the frontier models, not only is it yours, but now you can run cheaper, more economical open source models, either local or remote, and get better privacy and save money, and get really good results because the harness is really good.

I use the Venice API for my harnesses because I can call frontier models, and I can call open source models, and I can choose for which task am I going to need which model, for either saving money or making sure I get the best performance possible for that specific task. So, what this means in essence is AI writes the code now, makes the things for you. You just say what you want. Your job has moved up.

You are now the strategizer, you are the director. The AI does the labor. The rig itself, the harness, is what you need to make sure that labor produces consistently. You want good quality.

You want to make sure you don't have to babysit it. Once you get these harnesses created, you plug in whatever the input is, in my case, like a transcript. So, let's dive in to what it means to build a rig from scratch, okay? You don't really need to start one from scratch.

You could. Cloud Code and Code X know how to do it, but I actually recommend stealing a good one. So, this is from Cheddar Fox, Scott Graham, who was on the channel a while back to introduce these and changed my whole life learning about these. So, this is the safe agentic workflow.

This is for development. And we're not going to need something for all cases that is this crazy, but you could use a harness like this to build software. This is the scaled agile framework methodology adapted for AI agent teams. You could use it for any kind of team.

So, let's explore what that actually means. So, what happens in the harness is you have multiple agents in a team. In this case, you have 11 agents. They all do different things.

They all have skills. They all are triggered based on hooks or the main orchestrator agent calling them. Now, I am not a full-on developer or whatever, so I'm not going to get too deep into this because I'm not the person to be able to do that. But, just imagine you have all these skilled agents on your team that know what they're doing and have a full fresh context window to do it.

Here's an example of a harness I'm running right now inside Cloud Code. This is inside my brand knowledge base and it's already been running for 7 and 1/2 minutes. Maybe not too long, but it's got a bunch of phases here where it takes a transcript from a video or conversation and turns it into all kinds of content. It makes images.

It makes a Substack draft. It makes a ConvertKit draft. It reviews it all. It does everything.

It calls different models to do that. Like, it uses Opus to write, but then uses GPT 5.5 to review what was written and make sure it fits all the brand guidelines. And then, it makes videos, which you're going to see in this video. Anyway, this is a harness here and this just goes on for as long as it needs to and it creates whatever I set it up to create.

You don't have to be a developer or a fancy engineer person to be able to use something like this. At the end of this video, we'll jump into an actual template on GitHub that you can use to get started. And I'll show you some specific templates that help with certain tasks and how to build your own. Okay, so let's go back here.

I accidentally closed it. Whoops. It was at about 21, 22 minutes. So it's going to keep going now.

It's going to start over. But anyway, if I hadn't closed it by accident, we'd be at over 22 minutes now without needing any input by me. So actually, this is actually a good example here. Send message isn't available as a tool here.

I'll spawn a fresh agent for the cycle. For the next part of wherever it's at, it needs to use the send message tool, but the main agent can't do it. It has to spawn a new agent that does get access to that tool. If we want to see everything, it spawned this sub agent that says, "Okay, you are the Re-purpose Threads agent revision cycle two on the standalone tweets only.

So read this file and then reread the skills thread writer, voice standard, and stop slop and make sure that it's good, basically." So it's got its constraints here. It has its job. Okay. So let's back up.

What just happened here is in my workspace, the Jordan Harbinger's brand workspace, we have my knowledge base, which says everything about basically who you are, who I'm making videos and content for, how I talk about things, my glossary, sovereign entrepreneur, the sovereignty ladder, which you saw in the directory, my brand positioning, what is my brand, who do I speak to, what am I writing about, the thesis, my voice and tone. Anyway, so all of this stuff, my frameworks, everything about my brand is here, as well as my sub projects, as well as my domain. So whenever I open up an agent in this folder, I can literally just say like, "Hey, update my website with this." And it has everything it needs and it knows what to do. So that's part number one.

That's the context, my knowledge base. That is a crucial part of a solid harness, an AI agent. It needs to have its knowledge space. Then comes the dot cloud folder.

Inside the dot cloud folder is where we're going to see, okay, these agents being spawned. In here, we see we have our main agents, blog writer, content ideator, and then this is like their system prompt. Content ideator, you generate content ideas for the Sovereign Prenuer brand across all formats and map them to the four-tier sovereignty stack. First, read the thesis, then read the four tiers, then read the positioning document.

Okay, so it says here things you need to do to catch up on the knowledge. Here's what you got to do. But in this case, we're in the content folder, this one, and so we have another dot cloud folder, which says agents repurpose threads. So this agent produces X content from a transcript analysis.

And here we are in there. So in my main folder, it's got my agent system prompts, and then in my actual content repurposing folder, it's got its agent system prompts. Then you have the skills. So we have the newsletter writer skill that knows how to write like me and says, "Hey, check out the voice and tone document to write like Jordan." Uh it's got the stop slop skill.

Um props to Hardik Pandya on GitHub for creating this. Like, "Hey, make sure don't look like AI." Basically, and it's just got the skills that are needed for that workspace. And then here, we have uh no skills at all actually in content because it's actually pulling the skills from here. But we do have commands.

So the repurpose command, repurpose a transcript into a full Jordan Harbinger brand content package. So this is basically what the whole orchestrator agent is following in this harness workflow. And that's what keeps it on track here. But this is the beginnings of a harness.

You have the agents, you have the skills inside a workspace. And then in the safe agentic workflow, you also have hooks, commands, agent skills, and even a Claude MD. So we didn't talk about Claude MD. It's also called agents MD uh in Cursor.

But essentially the Claude MD here gives the context for the conversation to your main agent. So in our case Claude MD, this is the Jordan Herbs brand workspace. So when I start Claude here, it automatically says, "Hey, this is the Jordan Herbs brand work workspace. Open this folder when working on anything related to the brand." And so it knows what all the folders are and it just has to read this file visual generation workflow plain language standard voice rules.

It starts here so it doesn't actually have to load every single file. It already knows here in the beginning, "Hey, when you're looking for brand specific skills go here. When you're looking for the content repurposing harness, open up this." And that is how this main agent when Claude code is opened on the command line here in VS Code will automatically know, "Okay, Jordan provided a transcript. So let's run the multi-format content repurposing harness." And that's exactly what's going on here.

And like I mentioned earlier, we're at 21 minutes. We're now approaching 30 minutes here with no interaction from me aside from when I closed it by accident. And it is just running its thing. And it's got code it runs.

It's got skills it touches. It does all kinds of stuff. Like imagegen.shell. This is a script here.

Let's look at the prompt. Yeah, this is the prompt with my colors based on the brand which is here in Jordan Herbs KB. We see it here in design design system. There's the Jordan Herbs visual design system.

And so this whole harness knows where to look for all the data without having to overwhelm a single agent. Because if one agent were to load all these documents, it would take hundreds of thousands of tokens. And you only have a million token context limit and your quality gets worse the more tokens that are used. So instead the main agent only loads what it needs to if it needs to and otherwise will spawn an agent, give it the agent's system prompt like, "Hey, you're the content ideator, start doing this, here's what you do, and here are the documents you're going to need.

So, in this case, the content ideator is going to need to read the thesis file, the sovereignty stack file, the positioning file, the glossary file, the ID8 protocols, which is all quite a bit, but at least it takes place in a sub-agent conversation, a fresh new context window, so it doesn't have to clog up the main one, cuz the main one is just making sure we get everything high quality. So, then after that, whatever agent, let's say in this case, uh we have the image generator agent, whatever whatever it's called in here, generating the assets, then another agent's going to come through. Uh let's say for example, the thread was written, then the voice reviewer agent is going to come through in a new conversation, doesn't know anything about the past, and the main agent says, "Hey, here's your system prompt, you're the voice reviewer. First steps, check out the voice standard skill, the stop slop skill, the messaging document, voice and tone.

So, learn everything about Jordan's brand, how you write, how to not write, and then you're going to check what was written by the last sub-agent in a conversation you have no idea about, you have no context of. You're going to check that out, and you're going to make sure it's good." And then here is the return format, you're going to give a report. Then that report goes back to the main agent, and the main agent determines, "Okay, do we go on or not?" And that brings us back to the loop, read, pick tool, run, check. Are we done?

Yes or no? And then it just keeps going. So, essentially, every sub-agent is running in this loop in some way, shape, or form. We're running everything right now through Claude code and Opus.

But the great part is once this is built, and the same thing with this here, I can open this folder in open code using an open source model here via Venice, and I can run the same workflow, and it will do pretty much the same job, just with a different intelligence model behind it. So, if Claude code ever rugs me, and I can't use Opus anymore, or whatever, I have a rig that I can just plug in my custom intelligence to. So, the model, yes, is rented, but the rig itself, the harness, is owned. This is the sovereignty filter, I like to call it.

It's in the framework, in my knowledge base. You can swap the brain, but keep the body. You own the body. You are the sovereignpreneur here.

So, now is the time I'm going to give you a CTA in the AI Captains Academy. We have a bunch of agentic harnesses for solopreneurs to kickstart your next venture, especially if you're trying to sell AI. So, for example, this is a lead getter. It'll use Apify to scrape the web and find leads for you.

It'll verify the contact info. It'll find the specific pain signals. It'll even create a hook for your cold email. Then we got a funnel builder to help build a funnel.

There's a little video explaining how it works here. And it walks you through step by step. So, in this case, it'll help you with a client. You could also use it for yourself.

The idea is that you run an audit on a website, and it will run this 10-agent pipeline, web crawler, API caller, call free APIs for metrics about the website. It'll find competitors. It'll analyze AEO. It'll strategize for SEO.

Write a report. Design the report. Build the report. Review the whole design.

Then write a cover letter. And then validate everything with quality review. And then you can just email the client and be like, "Okay, hey, I made you this AEO audit. This is free, but if you want the big one, it's a few thousand bucks.

Uh but here it is." So, we have all these rigs in here. They're all harnesses. And they're free. It's 7-day free trial.

You can come in here and steal them, and then you can leave. It's all good. Uh this is my personal content forge. This is exactly what I was showing you here.

This is it. Oh, hey. So, yes, it created a Substack draft. Boom.

Premature Freedom. I built the dream, then gave it up on purpose, right? Um and then it wrote a newsletter draft. Oh, it sent it to the wrong place.

That's interesting. So, it made everything It baked for 16 more minutes. So, all of that took about 35, 37 minutes to run through everything. Here's all the content, premature freedom.

It's It's funny, it's kind of a I'm random one, but we got the YouTube metadata here, titles, description, even though this is not a YouTube video yet, chapters. I normally run this with the YouTube transcript after. This time I ran it with the transcript from before. Then here we have an X thread with images that weren't generated.

Oh, there were images. Okay, they generated images on brand and then And we got the blog posts here, we got the emails here, the newsletter. So, it went through and did all that, basically. That's a harness.

And I'm going to soon say fix that part of the harness so it doesn't happen again. And then get pushed. The harness is fixed, it sent the newsletter to ConvertKit. Right here, I built the dream then moved to a suburb on purpose.

Yeah, it is kind of weird what I did with my life. And then finally, probably the coolest bit, is here on the web now, premature freedom. Oh, it didn't pass the test. This is for my ghost writer, which is another video on the channel.

Hopefully you'll see, but this is the UI for my ghost writer because I like making the content, but I don't like posting it. So, this is the X thread that it can be copied or you can edit it. I don't make the blog post. It's got an image here.

All on brand. The newsletter, which is what we just saw on Kit. And then all YouTube metadata, which isn't accurate at all because this isn't actually a YouTube video like it normally is. And then there's the checklist of everything that's here.

And then the ghost writer, Tommy, can mark things as posted. It all syncs to GitHub and boom. That's the Jordan Narrows content repurposing system. That's a harness in action for something I think every solopreneur, creator kind of should probably be valuing uh repurposing your content without needing 100 different subscription platforms or a a team, etc.

So, how do we get started? Well, the first step would be simply to copy something that's already been made. Now, you have the Safe Agented workflow, and you can simply use this template. You could copy the URL.

There's a link below, by the way. Copy it, and then in your IDE or even Open Code, which I've been becoming a fan of the desktop app lately, you start a new session, you say, "Let's build a harness based on that URL. I want it to do X and Y and Z. Let's talk about it." And then shift into plan mode, and then you get started.

That's the like simplest way to do it, and your AI will then say, "Okay, we should duplicate it, we should clone it, or we should just take the ideas and build our own thing." Every model is probably going to have a different opinion on that. So, let's explore how I would do this now for the Sovereignty Directory. So, here in the sovereigntyatlas.com, I noticed that my data for SEO API key wasn't being called like I thought it should be. So, I said, "Hey, run an audit.

What's going on with our SEO? Was the strategy made correctly or not?" And basically it said it was, but Opus came back saying, "All right, here's what we can do to fix the schema and make it better." So, my recommendation for the next step is still A, seed the listing facts, B, deploy the schema fixes now, or C, build. I want to do all these things, but I want to do them efficiently. I want us to build a harness for these common tasks of adding new content and ensuring that the schema and everything matches and fits the directory's strategy most optimally.

And then I'll go into plan mode here. Let's talk about it. Sub agents, skills, etc. So, now I didn't copy any GitHub URL.

I just said, "Let's build a harness. >> That's a great instinct. >> Right, thanks a lot. The work I just did by hand is exactly the type of repeatable workflow that should be a harness.

There you go. So, we're now building a harness inside Claude Code for this repetitive task for this particular project, which is a directory, and it's going to need new content. So, this will save tokens and time in the long run because next time that I want to add new FAQs or new pages like the home setting libertarian pages, it'll just run through the harness and it won't need to ask me any questions and I won't need to waste a bunch of tokens like figuring it all out like it will right now. Okay, so now Claude has come back to me.

Which content workflows should the harness cover? This determines how many commands and agents we build. Seed frequent asked questions on listings? Yes.

Add a new listing? Yes. Write a new guide? Yes.

Refresh research and audit? Sure, why not? How autonomous should the pipeline be? Where does a human gate it?

So, basically, I'm just going to need if with this recommended option, the agent researches, drafts, and reviews the voice. The orchestrator presents the final artifact for my approval before it touches seed files or the DB. I'm just going to say fully autonomous because I'm pretty confident with Fable's original work. Where should the harness live and how should content reach production?

Yeah, right there. Great. So, now it's building the harness and we'll come back to that and run it when it's done building. Because I didn't want to do the human approval, it's going to do automated quality bars that make autonomy safe.

Which is probably smart. But this is essentially like even this safe agentic workflow, this is like enterprise grade AI stuff. These harnesses are how you go from like novice amateur AI slop to like really good quality. Now, I don't post anything without reviewing it.

Even the Atlas, I went through most of it before I put it online and I was like, "Holy moly, this is so good." I was like so impressed with his research and everything and I I know most of it's accurate cuz I'm familiar with most of it anyway. And I think that's just part of the appeal of Fable as well and these newer Frontier models. But anyway, it does such a good job that then the human almost just needs to like read it and go, "Okay, that's good. Ship it." Uh and you know, I I still haven't gotten to that point with my blog posts and that's probably a good thing because they're supposed to be coming from me, a human.

But they are a good starting point. Okay, so it's pretty much done now. What you have now, a fully autonomous content harness at Sullivan Directories Fest, Claude. So here we are at the Claude folder.

We have agents. Maybe it explains it here. We have commands for adding frequently asked question page schemas to listing up pages, as a listing with the correct wrong and trust label, writes a guide in Jordan's voice targeting a keyword cluster, refreshes demand data, runs the SEO AIO audit. Wow, cool.

Okay, two open items for you to do. Populate the Netlify build hook URL. Okay, got to figure that out. Confirm can reach the Superbase production connection.

All right, so want me to commit the harness or take it for a real spin. I'll say yes. Let's do that. And to make sure it doesn't write, we'll say write a handoff prompt to start in a fresh conversation.

So when you run a harness, you generally want to make sure it's in a new conversation so there's no like leftover residual context and tokens. And when you start fresh, you're going to get better results anyway. So here we go, finishing up whatever to do. Going to make some commits and then we'll start a new conversation.

So we got this handoff prompt. We'll clear this and let's just read it real quick. You're in this directory. It has an autonomous content harness that I built and verified end end.

Read Claude.md first. Actually, I don't even want to say all that. State what I want to do in this session. Run refresh research.

All right, so I'm just going to copy these. Actually, we need these constraints too cuz I haven't fixed whatever that is. All right. So, we'll clear this, paste that, and now I'm not telling it that it's in a harness.

That's the real point here because it should automatically open up Claude MD, which says, "Hey, this is the Sovereignty Atlas directory harness. Here's everything you need to know about it. Here are your commands. Here are your agents.

Here are your skills, reference files." So, that's what it's going to do first. And you really want to make sure you have auto mode on, too, by the way. But be careful with auto mode. So, it's running the refresh research skill.

Let's check that out. Interesting. Um where did refresh research? Where is it finding that skill?

Is there another Huh, well, I don't know where it found that skill, but it's using this skill now, directory strategy, and then it's using the researcher sub agent. So, in here Okay, there's refresh research. Oh, okay, it was a forward slash command. Okay, so, forward slash refresh research command.

It loaded that as a skill, basically, to get the pipeline, and it spawns the researcher agent, and that's what we see here, the researcher agent. And let's see, do we have a researcher agent in here? Yeah, researcher MD. Cash first demand competitive research for Sovereignty Atlas content.

Map a target to PAA questions, volumes, and competitive signal. Runs live for data SEO. You produce a research brief. Okay, so it's going.

So, here we see we've got it running. We're 5 and 1/2 minutes in now. If I scroll up a little bit, we'll see we had the researcher go, we had the SEO AEO auditor go, which was phase B, then the main, and then that agent gave the report to the orchestrator agent. The orchestrator agent reads it to decide what's next.

Report is good, so let's set up task tracking and see the first batch, and now it looks at the commands and what's necessary to run. Now I understand the pipeline, and it keeps going. So, now we're phase A, cuz we're still in phase A, the research brief for the FAQ batch. And it's just going to keep going.

So, once again, that's the harness. It's just going to go. We'll check back in when it's finished. Okay, boom.

Now, let's take a look. It's done. Step one, refresh the research. I didn't spend any money for the SEO stuff.

Step two, seed the FAQs. So, new stuff to the database, I think. Full gate chain. Half the stuff I don't even understand, but it understands and it knows why this is necessary, and that's what's important here.

Two things you need to know. This is not a clean that new batch. Committed mid-session. Yeah, that's because in another conversation I was testing it to make sure the system was working.

So, yeah, that's okay. Also worth noting the audit report only through the things that actually move forward. Okay. The deploy path changed mid-session.

Yeah. Okay, so I'm just going to explain the issues came because I had a testing agent checking everything out, and it didn't know you were working. So, that said, make sure everything is ready to go and commit and push and deploy everything. By the way, that cooked for 22 minutes.

Fable did such a better job, though. Didn't need to ask questions, didn't even need to think. And then, you know what I'll do separately in a new conversation here. I'll split this.

Run Claude. Add a recently added section on the landing page and push. So, this should be able to give us an an idea of how how well it worked. Okay, cool.

So, we got through it. Now, we should see the recently added. Okay, great. This one was submitted with the form a few days ago, the submit a listing.

I had a friend actually submit his community in El Salvador. So now, once this is done, we should see new ones here, and we'll know that it worked. Meanwhile, I'm just going to do some other little fixes. All right.

So, I finished the first thing, and then I actually set it to write these two more pages that apparently were easy pickings for the SEO. So, it's running the full pipeline on writing these two guides. Okay. So, the voice reviewer agent came through.

Plot had to look somewhere else to find that agent, and so we're going to fix that after. And then, it found some problems. Uncited historical claims. So, the reviewer is being appropriately strict, and adding a source is the honest move.

These are falsifiable claims. Let me revise homesteading cycle one to add a citation. So, this is what's cool about the harness that's properly created is it's not just going to write something and make it up to make you happy, because then another agent's going to be like, "Yo, that agent just made some up to make you happy." Because that reviewer agent's job is to make sure nothing that's not real gets through. So, this goes back to why I said I want it to be fully autonomous.

I don't want to read it myself, but I want to make sure it's good. Now, do I recommend you do that? No. I think we should read these things ourselves.

But, this whole project is kind of more of an experiment for me than than anything else. But, I am going to check it later. So, I mean, yeah. I mean, to each their own with that.

Homesteading, 34 out of 50 Libertarian, 45 out of 50 Okay, cool. So, it made it. It had to run through the loop again, because it decided that it wasn't good enough. So, now I'm just going to finish this up, put it live, and we will hopefully see it under the recently added section.

I also added a recently modified section, but nothing can show up there yet, because it's like a new field in the database. So, let's take a look. All right. And it's done.

We have these new guides. Let's take a look here. If I refresh here on the main page, okay, I'm not seeing the recently added thing, but let's take a look at these actual pages, the guides. Yes, they are here.

Great. So, I just made this, posted today. We've got some sources, archives.gov, okay. Killing the free land myth plainly, frequently asked questions.

And it actually did add these FAQs. And then let's look at this, what is a libertarian? Somebody calls themselves a libertarian just not along, but you're not totally sure he disagrees, too. Once again, a lot of this is written in my voice, okay?

That has my voice standards. This isn't just AI written, so that's why it looks not like most AI slop. Then what else it did was it took, for example, the cold card wallet and it added facts after, frequently asked questions. And that's all for more SEO stuff.

So, it just ran all that through the harness, and it also tells me, "Oh, there's things we can fix." So, that, my friends, is a harness that we just built in action with instant results, working on its own for, I don't know, we probably went on over an hour here, in and out with the different tasks. Okay, so we've gone through what an agentic harness is, how they work, how to get started. We've built one. So, let's TLDR here.

Models intelligence is essentially rented. Perhaps you have a powerful enough computer, but let's just say Fable 5 vanished in 3 days and disappeared, couldn't use it anymore. So, we have to keep this in mind if we want to be sovereign for new Earth. The harness is the rig around the model, it's the body for the agent, and this is the part you own.

Open source, keep it on your computer, it doesn't matter if intelligence disappears, you can plug different intelligence into it. Build with frontier, run with open source. Save the money on running it, but make sure you get the best quality when you build. Start by stealing a good one.

You're obviously not really stealing. Shout out to Scott Graham, thank you for providing this to the world. It's awesome. Safe agentic workflow, the link is below.

You can also just do what I did and just tell Claude code to start building it and believe it or not, I'll do a really good job. Swap the brain, keep the body. That is a durable asset. So I hope this was helpful.

Make sure to say hi in AI Captain's Academy if you want to explore harnesses and rigs. If you have questions, everyone is welcome. We also host two calls a week in there. I will be there.

We can hang out, ask questions, anything you need. And don't forget to subscribe if you want to learn about AI agents with me. See you later.

Well, here we are again with yet another term called loop engineering. And it wasn't even that long ago we had harness engineering. So, is this just a marketing hype or is there actually something substantial behind loop engineering? Let's walk down from prompt engineering all the way to loop engineering to see why we need it each step of the way.

We all know by now what prompt engineering is. Let's say I start with the prompt, you are a helpful customer service rep. Please be nice to my customers. This is prompt engineering because you are prompting the agent by implicitly telling the agent what to do.

And we can then ask AI any question and the agent will impersonate a customer service rep based on the prompt we just generated. That makes sense and super simple. So why do we need context engineering? Turns out this prompt that we just created lives inside of the agents context window and we still have a lot of room left in our context to do something more useful.

So what if we gave the agent autonomy to invoke tools to start filling up its own context based on what the prompt actually needs? This was the beginning of context engineering where the agent could now access files to load and make changes or even use MCP to start interacting with databases and external applications to load its own context. Cool. Context engineering makes sense.

So then why do we need harness engineering? Well, there's actually no inherent flaw to context engineering, but rather it has limitations. Context engineering is not really good at tasks that take longer than 5 to 10 minutes long. And here's why.

Tasks that take long tend to require more context than what the agent can handle. And while it could keep summarizing its own context as it approached the limit, it was extremely leaky where important details started to get lost in each step of the summarization. So essentially we needed a system outside of context engineering to better manage internally its own context from outside in and this very external system is what we call harness engineering. Harness engineering manages the context from the outside and helps the agent runtime to work on tasks that help break down the user's requirement into a more stable execution.

Let's look at a few examples to make sure that we understand this more concretely and we'll use chatib and clot code since it's something that we all have hands-on experience in. But before we dive in, today's video is sponsored by Verta. If you write code for a living, you probably hit a point where your own machine just can't handle what you're trying to build. And that's the gap that Verta fills in.

Verta is a full stack AI cloud built for the whole model life cycle. from a single GPU instance and instant clusters to serverless inference and manage endpoints. It isn't a general purpose cloud with AI bolted on after the fact. It's built specifically for AI work by people who actually train and serve models.

So the workflow is designed around how you'd actually use it. And the workflow is the part that I like. You sign up, spin up an instance, and pick your GPU, anything from the new GB300 down to an older V100. You then drop in your SSH key, and you're straight in from your own terminal.

I just SSH in from VS Code, drag my code across, and I've got all the VRAMm that the card gives me to run without ever leaving my editor. There's also a CLI if you'd rather stay in the terminal start to finish. Spin instances up, manage them, and connect all from the command line. On the hardware side, you get serious NVIDIA GPUs like B300's with NVLink, Infiniband, and fast NVME storage for workloads where interconnect and throughput actually matter.

And if you're working with data or model weights you can't afford to expose, Vera supports confidential computing, so your workload stays encrypted while it's actually running on GPU, not just sitting in storage. Link in the description. Use code Caleb-50 to get $50 off of compute for just $5. That's 90% off of your first top up.

Thanks again, Vera, for sponsoring. Asking Chachib the question, "How many cheeseburgers can I fit between the Earth and the Moon?" This is solely using prompt engineering because it doesn't need anything external to answer a very simple question. It can just reason through what it already knows to answer my question. Now asking chacheti what is the latest discovery that NASA made.

This now has to use context engineering because it has to search the web and gather relevant information from NASA to help answer my question. So context engineering helps bringing information from the web autonomously. Now when I ask clot code, can you clone the entire NASA website? This is all done by harness engineering because the NASA website as you can imagine is really complex and simply relying on context engineering on tasks like this will start to choke midway through.

So harness engineering provides an external mechanism to help manage the context and the runtime for the agent to work through a long list of tasks. Now you might have noticed a pattern that emerges from this and that pattern is the concept of loops. For context engineering, there's a loop where the agent recursively calls tool after tool autonomously until it thinks that it has enough context to answer your question. And for harness engineering, there's also a loop where the agent has a list of tasks outside of the context window to keep iterating task after task until the entire operation is finished.

So what we find is that we are essentially stacking loop on top of another loop. Now we get to loop engineering which is yet another loop in itself. I know is there even an end to this. So loop engineering stacks another loop outside of harness engineering layer to guide the harness externally.

But why why do we even need yet another scaffolding? Again at the heart of loop engineering loop engineering targets the human interaction in actually prompting the agent to do something. Everything that we have seen so far involves a human asking the agent questions like, "How many cheeseburgers can I fit between Earth and the moon? What is the latest news on NASA?" Or even clone the NASA's website.

These are prompts that require me to actually prompt the agent. But what if we built a scaffolding outside of this so that the agent can also prompt itself on what it thinks it needs to do? That is the heart and the spirit of what loop engineering tries to target. And if all of this sounds hokeyp pokey to you, you're probably not alone.

There's a lot of people saying loop engineering is just a buzzword and that's trying to encourage people to just burn more tokens and create more AI slop. And so far, we have really yet to see loop engineering in action that really makes a huge difference. But it could be the next evolution in our engineering philosophy as agents expand its scopes in what it can help us with. And it does raise a really interesting debate and discussion around all of this.

So what exactly is loop engineering and how does all of it work? Addiosman wrote a blog describing loop engineering with six components. But we don't want to read all of this. So instead of boring you with details, I'm going to give you a potential use case of loop engineering that help you wrap your head around what loop engineering could look like.

Let's say I built a website that keeps track of the World Cup scores. And when I ask Codeex to build me a World Cup website, Codex will use prompt, context, and harness engineering to build this beautifully written website. Now, there's one problem here, and that problem is that the World Cup games are happening every single day. And that means in order to maintain the website that I just created through Codeex, I have to keep prompting the agent to frequently update the site and also work on bug fixes that people might find on the website.

But what if I just created a schedule task inside of Codeex to check every hour for updates as new information becomes available? And what if I do the same for bug fixes where the agent just checks autonomously for bugs that are reported by users and fix them. What you're seeing here is that we are beginning to create this loop outside of harness engineering where it's self-guided rather than humanguided to maintain my website. And because I have skills and plugins already installed on my codeex environment, the agent can access an existing knowledge base to keep building and improving its knowledge along the way.

And the agent can also use sub agents to verify its own work and also the ability to work on multiple fixes at the same time by using what's called work tree to prevent runtime contamination along the way. All of these that I just mentioned are essential ingredients to what makes loop engineering what it is. And that is the six components that Addi Osmani wrote in his blog. Automation, workree, skills, plugins and connectors, sub aents and state are components of loop engineering.

And while this World Cup website that I just created is just an example of what loop engineering could look like, the true potential of what loop engineering could really be is still somewhat theoretical. And one thing to keep in mind is that loop engineering doesn't necessarily mean that all the engineering philosophy underneath is less important or even less needed that before. It's just agents growing in scope and building on top of each other.

The AI race may be entering a strange new phase. For years, everyone obsessed over the model itself, but now some of the biggest names in AI are starting to focus on something else entirely. They are calling it harness engineering. Because apparently, the same AI model can become up to six times more effective just by changing the system around it.

Same model, same raw capability, completely different result. So, the question becomes, what is the real difference? That difference is the harness. The easiest way to understand it is this.

The model is the intelligence engine, but the harness is everything around it that turns that intelligence into reliable work. It includes the rules, tools, memory, skill libraries, verification systems, context management, permissions, fallback paths, audit logs, and feedback loops that guide the model before it acts, while it acts, and after it gives an answer. Mitchell Hashimoto, the co-founder of HashiCorp and creator of Terraform, helped push the term into the mainstream earlier in 2026. His framing was very direct.

When an AI agent makes a mistake, the answer should not be to just rerun the same prompt and hope it works next time. The better answer is to change the system so that entire class of mistakes stops coming back. That is the real shift here. Prompt engineering was mostly about getting the model to do something right in one interaction.

Harness engineering is about building an environment where the model keeps doing the right thing over time. It is the difference between correcting an AI once and designing the system so the same error becomes much harder to repeat. And that is why the phrase spread so quickly. OpenAI, Anthropic, LangChain, and other parts of the AI industry have all been moving in this direction even when they use slightly different words.

OpenAI published its own essay around the idea and described how this works inside large code generation workflows. According to one article, OpenAI processed roughly 1 million lines of code and around 1,500 pull requests in 5 months, with humans moving away from writing every line manually and towards shaping the environment around the agent. LangChain compressed the idea into a simple message that people could repeat. Martin Fowler's site gave it a more formal engineering frame.

Anthropic has often been more practical than terminological, focusing on the actual systems and safety layers rather than the label itself. And that matters because harness engineering is not some random new buzzword for old prompting. Prompting, context, and harness work are related, but they are not the same thing. If you change the words the model directly reads, that is prompt work.

If you change what information the model receives, that is context work. But if you change the invisible structure around the model, like the tools it can call, the checks it must pass, the memory it can trust, the permissions it has, and the recovery process when something goes wrong, that is harness work. A tool by itself is not the harness. An MCP server by itself is not the harness.

A skill library by itself is not the harness. Those are components. The harness is the assembled system that decides how all those pieces work together. And this is where the AI race starts to look very different.

A Stanford and Tsinghua University joint study reportedly found that the same model with different harness designs could vary in performance by up to six times. The model stayed the same. The surrounding scaffold changed. That is a massive result because it suggests that as frontier models become more widely available and more similar in capability, the advantage moves to the team that builds the better system around them.

This also helps explain why AI adoption in the economy still looks strange. On one side, Goldman Sachs argued in April 2023 that generative AI could raise global GDP by 7% or nearly $7 trillion over a decade. That is a huge macro claim. But by April 2024, Goldman said only 4% of US firms had actually adopted generative AI.

Even in information services, where you would expect adoption to be much higher, the number was just 16% with 23% expected within 6 months. So, the promise is massive, but the rollout is still uneven. That gap is not only about access to models. Plenty of companies can access strong models now.

The bigger issue is that they do not yet have the system layer that turns AI capability into repeatable productivity. The model may be powerful, but without the harness, it remains fragile. It can answer one question, generate one file, write one piece of code, or summarize one document, but it may struggle to operate reliably inside a real workflow with memory, permissions, tools, deadlines, edge cases, and consequences. This is especially clear with agentic AI.

A normal chatbot gives an answer. An agent has to operate over time. It may need to open a terminal, search files, read documentation, write code, test the result, call an API, update a database, ask for clarification, store memory, recover from a failed command, and decide whether an action is safe before it touches a live environment. Once an AI model is embedded inside tools, browsers, terminals, repositories, memory stores, and external services, its behavior is no longer determined by the model alone.

It is determined by the whole system. That is why a new UC Berkeley paper argues that for agentic AI, model scaling alone is no longer the full story. For normal chatbots, the model matters the most. But once an AI becomes an agent, once it starts using tools, opening files, running commands, remembering things, and taking actions, the model is only one part of the machine.

The paper says the next major bottleneck is system scaling, or scaling the harness. A real agent needs several layers working together. It needs the LLM itself, which is the reasoning engine. It needs memory, so it can remember useful information across tasks.

It needs a context system, so it knows what information to put in front of the model and what to leave out. It needs skill routing, so it can pick the right tool or workflow at the right time. It needs an orchestration loop, which controls the sequence of steps. And it needs verification and governance, so the agent cannot just take risky actions without checks, permissions, logs, or rollback path.

That sounds technical, but this is already happening in serious AI systems. Claude code, open claw, and cheetah claws are different kinds of agent systems, but they all face the same basic problem. How do you control what the AI sees, remembers, uses, checks, and changes? And the first major problem is context.

A lot of people think a bigger context window automatically makes an AI agent better, but the UC Berkeley paper makes a sharper point. The hard part is not giving the model more tokens. The hard part is giving it the right tokens. A million token context window does not help much if the useful detail is buried under old logs, stale notes, irrelevant files, and conflicting information.

That is where context rot comes in. The model technically has the information somewhere in the window, but the signal gets drowned in noise. That is why real systems already fight this aggressively. Recent analyses of Claude code describe a five-tier compaction system with things like micro compact to clean up old tool results and context collapse to summarize long conversations.

And when a tool produces a massive output, like a giant server error log, the system does not just dump everything into the model. It can write the full file to local disk and give the model only an 8 kilobyte preview first. So, the agent behaves more like a developer. Check the top of the log, understand the shape of the problem, then dig deeper only when needed.

The second problem is memory. Memory sounds useful, but bad memory can be dangerous. An agent might remember an old note about how a code base works, miss the fact that the code was refactored yesterday, and then confidently apply the wrong fix. The paper calls this the stale but confident problem.

The memory is outdated, but the agent treats it like truth. So, a serious harness treats memory with suspicion. Something like a memory.md file should act more like a hint than a fact. Before the agent edits files or takes a risky action, it has to check the live environment and verify that the memory is still true.

Some systems even clean memory in the background during idle time, removing contradictions, compressing useful lessons, and stopping the agent from slowly filling up with old or messy information. The third problem is skills. Giving an agent more skills sounds like an obvious upgrade, but it creates another problem, choosing the right one. The agent has to know which skill to use, when to use it, how to combine it with other skills, and how to check the result.

A specialized tool can produce an answer that looks confident and useful while still being completely wrong. So, the real issue is not just having skills. It is routing and checking them. That is where harness engineering becomes practical.

A strong harness does not just give the model more tools and hope it behaves. It connects those tools to checks. Did the task actually finish? Did the output match the request?

Did the system change safely? Was the tool result verified? Is the agent even allowed to continue? And now researchers are taking this one step further.

They are asking whether AI agents can improve their own harnesses from experience. That is where retrospective harness optimization or RHO comes in. A new paper from Microsoft Research Asia and City University of Hong Kong introduces RHO as a way for an agent to improve its harness by looking back at its own past work. Instead of needing a labeled validation set with correct answers, the system studies old trajectories, finds difficult and diverse tasks, reruns them, compares different attempts, diagnoses what went wrong, and proposes harness updates.

The key part is that RHO does not need ground truth labels. It uses the agent's own preference over different attempts. First, it selects a small group of past tasks that are both hard and diverse. The paper uses a method called DPP to balance those two things, because choosing only the hardest tasks can focus too narrowly on one type of failure, while choosing only for variety can miss the serious problems.

Then it runs multiple attempts on each task and looks for two signals. Self-validation checks whether the agent actually completed the task properly and catches things like false assumptions, wrong tool calls, and stopping too early. Self-consistency compares different attempts on the same task and looks for major disagreements in the plan, the tools used, or the final answer. Those signals become instructions for improving the harness.

Then RHO generates several candidate harnesses, tests them against the old one, and keeps the candidate that performs better, but only if the score is actually positive, so it is not randomly changing the agent. It is using past failures to decide what the system around the model should learn. And the results are the part that makes this hard to ignore. Using Codex with GPT-5.5, RHO improved SWE-Bench Pro from 0.59 to 0.78 without external grading.

And it also improved Terminal Bench 2 and Gaia 2. So, the gains showed up across coding, technical work, and knowledge tasks. What makes this more interesting is that RHO was not just giving the agent more memory, it was changing the actual system around it, the tools, skills, instructions, and checks that shape how the agent works. After optimization, the agent verified its work more often, used tools more carefully, and performed better on long tasks where normal agents usually start falling apart.

That points to the bigger shift. Future agents may improve by learning from their own work history. Every task leaves a trail. Every failure leaves a clue.

And repeated mistakes can become updates to the harness itself. Of course, that also creates risk. If an AI can update persistent behavior from its own judgments, it can also reinforce bad habits or unsafe shortcuts. So, serious systems still need audit logs, human approval, and safety checks.

And the next phase of AI may be won by whoever builds the best harness around the model. Also, if you want more content around science, space, and advanced tech, we've launched a separate channel for that. Link's in the description. Go check it out.

If you think harness engineering is really the next big AI advantage, drop your take in the comments. Hit subscribe if this made you look at AI agents differently. Thanks for watching, and I'll catch you in the next one.

Everybody talks about agent harnesses, but what exactly is a harness? Now, even people who are actively building agents can't always give you a clean answer. Uh the word gets thrown around constantly, but nobody really agrees on what exactly it means. So, in this video, I want to do three things.

First, define what a harness actually is. is. is. And [snorts] just as more importantly, what it's not.

Then, we will walk through nine components that I think makes a modern harness. And finally, we'll build a tiny one in Python, so you can see exactly what is going inside. This is going to be especially important for people who are thinking about building agents and harnesses. In simple term, a harness is a fixed architecture that turns a model into an agent.

So, if you think about modern LLMs or models, these are just one-shot text generators. You ask a question, it answers and stops. A harness is what gives it the ability to take action, see the consequences, and keep going until the problem is actually solved. So, think of a model as the engine and the harness around it as the car.

That what's make an agent. So, a really good example of this is agentic coding tools like Codex, Cursor, uh Cloud Code, Windswept. These are all harnesses. Each one started from a concrete problem making a model write and edit code across a real repository.

And I think they have uh converged on remarkably similar architectures. Now, we're going to look at that architecture in a minute, but first, I want to talk about something else that you probably have heard about. And this is frameworks. So, think about things like LangChain, LangGraph, AutoGen, CrewAI.

These are not harnesses. And I think this distinction is really worth making because right now, people are using these terms interchangeably. And And And it's kind of getting confusing. So, a framework gives you abstraction.

Uh think about state crafts, chains, memory connections, and retrievers. You as a user have to wire them together. The fundamental assumption is that you, the human architect, will configure these pieces together. Now, harness on the other side uh uh uh is from opposite direction.

There's no assembly step. It basically uh ships a working agent. And in simple terms, it's just a while loop with a tool registry and permission layer. And everything comes wired together.

Now, another way to think about this is that framework is built for a human to assemble an agent. A harness is built for the agent itself to a task. And in big picture, you just provide the goal, the harness will handle the rest. So, in the rest of the video, we're going to primarily focus on harnesses and what they make them interesting.

Okay, so what exactly is inside a harness? I would say there are nine main components that you need to consider if you're building an agentic harness. Now, we're going to uh go through the list. This is mostly opinionated uh architecture, uh but something that I have seen to work really nicely in practice.

I'll try to tie together to Cloud Code because I think this is um an example of a really great harness put together. Okay, so the first component is the while loop. Um this is basically the foundation. It's the outer iteration loop.

The harness is at its core of while loop. The model reads its system uh prompt, decides which tool to call, runs the tool, feeds the result back into the context, and loops again. And this process keeps repeating till the uh model produces a text-only response or it uh hits a maximum iteration cap. Now, we're talking about uh text-only models, but the same can apply to multimodal models as well.

So, think of this outer loop as the whole engine that runs everything. Now, number two is context management. On every turn, the tree grows um as you encounter more uh user messages, more tool calls, more you're going to see that uh you hit the context limit of your large language model. So, the harness has to decide what to keep verbatim, what to summarize, and what to throw away.

In Cloud Code's example, the budget used to be around 200,000 tokens. Now, they have increased it to 1 million token in case of Opus. Uh but let's say when you are uh reaching almost half of it, um or maybe 80 to 90%, it triggers a compaction. Some of the most recent messages are going to stay in full.

Everything older gets summarized. Now, this compaction is very important um and it can have some real bad consequences if not um properly did. So, you need to be very careful about context management. Now, the third component uh is skills and tools.

So, tools are the primitives that reads a file, edit a file, run bash, search code. Skills are a top uh a layer on the top. So, they are how organizational knowledge gets encoded. Uh usually, you're going to see them in model files.

Now, to think about uh tools and skills, I would say tools are universal. Uh skills are specific to your team, your workflow. And then, there is the registry. Uh so, it tells what is available, uh what permission each uh thing needs, how the call gets dispatched.

Now, the fourth component is sub-agent management. Now, at some point um a task gets too big or too parallel for a single conversation thread. So, the harness uh is going to create sub-agents that work in isolation. Each sub-agent gets its own session, its own restricted set of tools, and a focused system prompt that says, uh "You're working on this specific task." Now, the idea over there is to span, restrict, and collect uh the outputs.

That's kind of the pattern you want to use. If you want to learn about generative AI in a structured way, you will love today's sponsor, which is Coursera. They have a number of different courses on generative AI. Here are the four that I highly recommend for you as a beginner.

Start with Google AI Essentials. It's about 10 hours taught by Google's own AI team. Over 1.7 million people have already enrolled. Next, Vanderbilt's prompt engineering for ChatGPT.

This is where you learn the actual patterns, chain of thought, few-shot, persona prompting, etc. If you're a developer, jump to their generative AI with large language models built with AWS. It's a hands-on lab. You fine-tune, deploy, and ship real LLM applications.

And if you want a full career on-ramp, IBM's AI developer professional certificate is for you. You get to build chatbots, apps, RAG, and agents. Right now, get 40% off off 3 months of Coursera Plus through the link in the description. Now, back to the video.

Number five is built-in skills. So, we already talked about skills um that you provide as a user, but every harness ships with a baseline set of skills that are going to work out of the box. So, think about file operation, read, write, edit, search, or cell execution, um code navigation, things like that. Now, for modern harnesses, these are really non-negotiable.

If your agent cannot read or edit files, it isn't a coding agent. So, beyond the primitives, modern harnesses also ship with high-level skills. Um for example, you a harness can have a skill of how to make a Git commit, um how to open a pull request, how to run tests results. Now, some of these built-in skills are going to be uh specific to the vendor or creator of the harness.

Number six is session persistence or memory. So, a long agent session is stateful. If the process crashes, you lose everything unless the harness writes state to disk. And the modern uh the way the modern harnesses do this is pretty elegant.

So, typically, uh they're going to use append-only JSON files or maybe Markdown files. So, every message, every tool results, every compaction event gets one line. Now, the beauty of this is that you can resume exactly where you left off. Uh actually, recently uh attended a talk from the Anthropic team where they were discussing about how they built managed agents.

Uh I'm going to create uh cover that in another video, but they had the session uh management separate from the harness itself, which was I think a very interesting design. Number seven is system prompt assembly. Now, this is the one that will surprise most people. The system prompt is not a static string.

It's basically a pipeline that just walks um ancestor directories looking for specific types of instructions. So, if you have cloud.md or agents.md, it's going to inject those into the system prompt. Now, you also want to be a little careful here um because most of these um third-party harnesses or even the first-party harnesses have really strict record prompt caching. Uh if you dynamically introduce uh components to system prompt, that is going to break the caching, right?

So, you need to be careful about that, but in certain situations, you want to clean uh assemble the system prompts. Okay, number eight is going to be life cycle hooks. So, this is extensibility scene. Um hooks let you inject custom logic before or after a tool runs without touching the harness itself.

So, a pre-tool hook fires before execution. Uh it receives the tool name, the input, and can allow, deny, or modify the call. A post-tool hook uh runs after and can inspect the results. So, the protocol is kind of structured.

Uh think about uh a JSON file with exit codes for allow or deny. Now, the beauty is that hooks also enable uh intercommunication between different har- and hooks are how enterprises today adopt harnesses themselves. Now, let's talk about number nine, permissions and safety. So, this is the layer that makes the difference between a useful tool and a dangerous one.

Modern harnesses define a hierarchy of permission modes. You can have read-only space right, um full access. Each tool declares the minimum permission it requires. Now, the job of the harness is to enforce that at dispatch time before the tool even um ever runs.

And for tools like bash, the harness even classifies the commands dynamically. So, let's say uh if you say list files, uh it's going to be read-only. Uh if you want to delete something, that will need full access. And uh the harness figures it out by parsing the command string.

On top of the static permission, you get interactive approvals. The agent can pause, ask, "Should I run this?" right? So, before executing anything dangerous, you want to have this safety layer built into the harness. All right, so these are uh the nine uh components that I think every uh harness needs to have.

Iteration loop, context management, skills and tools, sub-agents, built-in uh skills, session persistence or memory, system prompt assembly, life cycle hooks, and permissions. Now, the easiest way to actually understand um a harness is to build one. So, let's write a minimal version of Python. Um Nothing fancy.

Just enough to see all these components together. Okay, so in this part, we're going to quickly go over a reference implementation. Uh so, think of this as a structure or template that you want to use if you're building a harness. Okay, so the main engine is the while loop uh that basically controls everything.

It assembles the system prompt and starts looping. Now, on every iteration, um the context is going to be compacted if it grows too large. All of these things plus the tool calls and uh calling to sub-agents are going to be implemented within this while loop. You also want to cap how many iterations are going to be in this loop so that it never runs forever.

This is really the entire engine. Uh every other file in the project exists to support these few lines. Now, uh this code implements simple context management. Um In in this very simple form, we are just doing compaction.

So, if the the the history uh grows beyond a certain point, we just summarize uh some of the older conversations and put them together. Now, there are more advanced uh compaction techniques, but this is a very simple reference implementation. Now, if you're making tool calls, you also need to decide whether you're going to bring in everything that is done within the tool call or only the input and outputs. So, those are the design design decisions that you'll have to take as an architect.

Okay, this code uh implements a simple tool and skills registry. So, every tool in the harness is described by a small um data class, a name, what type of permissions they're going to have, and a handler function, and one line line um description. The registry is just a dictionary that maps the tool name to that record. Now, there are a few functions.

Uh calling register adds a new tool. Calling get uh retrieves one for dispatch. Calling descriptors return a lightweight version of the list, which is going to contain name, permissions, description that we're going to send to the model so it knows what is available. Skills are registered um um um the exact same way.

They're just tools whose handle handler uh reads a markdown file at invocation time. Next is sub-agents. Uh so, you can implement multiple different sub-agents. This code looks at three different things for uh exploration, general, and then verification.

Now, each archetype has its own permission levels, its own restricted tool list, and its own focus uh system prompt. Now, every uh harness also needs to have built-in primitives. Uh these are the non-negotiable tools every coding harness must ship with. Uh example of this would be reading files, running bash commands.

Now, this also depends on uh the type of work you want your agent to perform. Now, in this case, one thing to keep in mind um these primitives need to use pure standard libraries. You don't want to rely on uh framework dependencies um which is going to be critical because that actually enables the model to take actions. Okay, so here is a simple reference implementation for um session memory or or or uh uh uh context uh persistence.

Now, every event the agent generates gets written to disk as one line of JSON. You can also use markdowns, but JSON seems to be um the better choice. The append method opens the file in uh append mode, writes the event, and immediately flushes it. That way, if the process crashes after the next line, this one is already safe on disk.

The replay method reads the uh file back line by line and reconstructs the full session. Because the file is append-only, uh two runs of the harness can share the same log without stepping on each other. If the harness dies, the file does not. This is the whole durability story.

Uh if you want to persist memory, Okay, uh next one is system prompt assembly. Uh so, you don't have to have a fixed system prompt. You can actually dynamically uh load things into the system prompt. So, for example, uh you can load agents.md, cloud.md, or any other memory files that you have stored into the system prompt dynamically just by reading a directory uh and reading files from disk.

One thing to uh be aware of, the order matters here. So, keep the static part first, uh and then dynamically load content second. Otherwise, you're going to break the prefix caching. Okay, so next one is hooks, uh which are used for extensibility.

Now, there are two different types of hooks. Uh one is pre-tool hook, and the other one is post-tool hook. And the idea is the pre-tool hook fires before any tool runs and can um either allow or deny the call. A post-tool hook fires after the tool runs and sees the output.

Uh It cannot block anything. It's there to audit. Uh can be used for logging and observability. Okay, the last component is permissions um um um and safety.

So, each tool declares the minimum permissions it needs. Uh it can be read, workspace, or full. Uh now, the harness needs to uh provide that extensibility and uh control the permissions of every tool. Now, there's one more thing that you need to be aware of.

Uh the same tool can be safe or dangerous depending on the command. Uh so, we uh classify it dynamically. Safe commands like list, uh concatenation, and grep uh stay at read-only. Dangerous commands like uh delete, pseudo, or shutdown jump straight to the full access.

So, anything else gets workspace level. On top of these static rules, the agent can also pause and ask the user for explicit approval before running anything destructive. And this is the part that you need to implement within your harness. Now, these are the components that I think every harness needs to have.

Now, do let me know your thoughts. What are are are uh components your harness have? And if you're interested in technical contents like this, make sure to subscribe to the channel. Anyways, thanks for watching and I'll see you in the next one.

Thanks to HubSpot for sponsoring this video. So, something really big actually happened in December 2025 and most of the people didn't even realize that. Entry Kapsi tweeted about this last week. It's very hard to communicate how much programming has changed due to AI in the last 2 months, specifically since last December.

And Greg from OpenAI also talked about this. Since December, there's step function improvements in what the model and tools are capable of. And a few engineers have told him that their job has fundamentally changed since December 2025. So, what actually happened in December 2025?

In short words, the latest model introduced then is finally ready for fully autonomous long-running tasks. So, with AI, the ultimate dream is always that while we are sleeping, AI can just work on tasks fully autonomously 24/7. Even back 2023, the most popular project, if you remember, is called Auto GPT. It is first time this fully autonomous agent existing was introduced.

And they have very basic and simple architecture that using GPT-4 as a model to autonomously break down a list of tasks based on user's goal simple memory storage to store the result. And people were doing some pretty crazy stuff like just give it a goal, make a $100,000 and let it loop through tasks infinitely until complete. Back then, the system just break and failed miserably because the model is simply not ready. But since December last year, this really changed.

The models have significantly higher quality, long-term coherence, and they can power through much larger and longer tasks. And we saw all sorts of different experimentation came out from industry. Firstly, from January, we got this super hot concept called rough loop. The most basic and simple agent iteration loop to force model work longer so that it can take more complex tasks.

You just follow the model with some simple condition checks. But already, we start seeing the difference. And 1 week later, Cursor also released their experimentation where they used GPT-5.2 to autonomously build a browser from scratch with 3 million lines of code. And Anthropic also released this experimentation they had where they get a team of cloud codes to autonomously working on a C compiler from scratch for 2 weeks.

In the end, it delivered a functional version with zero manual coding. You can even run Doom inside this compiler, as well. At same time, open claw start gaining attention and had this explosive growth that we never seen before. And it was very difficult to understand what was going on with open claw, cuz from outside, it's very easy to categorize open claw just be another menace, but living inside your own computer and can also access from Telegram.

Like, why is it so popular? And only later after I used a deep play, I realized that the real difference is that open claw represent this type of always-on, long-running, fully autonomous agents. That is very different from all the other agentic system we used before, where human is main driver to prompt for the next action. Open claw is always-on and it is proactive.

And this autonomous feeding is created by a very simple architecture, where it has memory context layer with a trigger and a cron job to automatically take actions and have the full computer access, which is powerful environment it can operate in. And I believe open claw is the first project that really open up the biggest paradigm shift in 2026. That we are moving from a co-pilot, simple task-based agent system to those long-running, fully autonomous agent. Something that's always-on, always ready, autonomously delivering super complex, coordinated work.

This is a critical shift you have to understand. The model today is actually much more powerful than you think, as long as you design right system to unlock it. And this is the crux of what I want to talk about today. The harness engineer to re-enable long-running autonomous systems.

If it's first time you hear about harness engineer, this is like evolution from what we've been previously talked about, which is context engineer or prompt engineer. So, previously, we really focused on how to optimize the prompts within the effective context window to get a model have the best performance for a single agent loop session. But harness engineer is really focused on those long-running tasks, which means how do you design a system that can works across different sessions and multiple different agents? And how do you design the right workflow to making sure the relevant context will be retrieved for each session and right set of toolings to extract most out of models.

This is fairly new concept, but the good thing is that industry already convert on some best practice that you can use from Anthropic, Vercel, LangChain, and many others. We'll go through each one of them one by one so you can see the patterns. But before you dive into this, with this paradigm shift fully autonomous agents, one of the biggest opportunity for the next 6-12 months is build open cloth for a certain verticals, which means you deeply investigate and understand the end-to-end workflow of a certain vertical and build it autonomous agent with correct environment and tooling to enable the end-to-end process. That's why I want to introduce you to this awesome research HubSpot did on the AI adoption in email marketing report.

It is fascinating report for you to understand for a vertical like email marketing, where people actually use AI today and what are the gaps. Cuz this report showcase clear workflow and opportunity email marketing that you can potentially automate. They survey hundreds of email marketers from top companies to understand exactly how AI is reshaping their workflows. They talk about why marketers are still doing a lot of heavy editing, what were the cost to it, as well as the biggest challenge they are facing today when implementing AI in the email marketings.

And each of them is a big opportunity for you to build a fully autonomous agents. They even dive into the specific KPI that they care more about and AI has show proven results, as well as what exactly things email marketers are really want from AI. So, if you're a builder who are thinking about the next big agent product to build, I highly recommend you go check out this awesome resource. I put the link in the description below for you to download for free.

And thanks HubSpot for sponsoring this video. Now, let's get back to harness engineer for long-running agent systems. And at high level, there are three learnings I took away from those. One is that for long-running task agents, the critical part of system design is creating this legible environment where each sub-agent or sessions can actually understand where things are at.

And most likely there's some workflows that can be done to enforce eligibility of the environment. And I'll expand a bit more on that. The second is verification is critical. You can improve assistant output significantly by allowing it to verify its work effectively with faster feedback loop.

And third is that we need to trust model more instead of building specialized tooling that wrap a lot of reasoning and logic prematurely. We should give model max context with generic tooling that they need to be able to understand and explore like human. And I'll unpack those three things one by one as we go through each block here. First is Anthropic's effective harness for long-running agents blocks.

So they've experimented using Cloud Code SDK to build a specialized agent for super long-running tasks like build a clone of cloud.ai website. The very first failures they observed is that firstly agent tend to do too much at once. Essentially it will always try to one-shot the whole app. And this led to the model running out of context in the middle of its implementation and leaving the next session to start with the feature half implemented or documented.

Then agent would have to guess what actually happened and spend substantial time trying to get the basic app working again. And second failure they observed is that agent tend to declare job complete prematurely. You probably experienced this a few times yourself as well. The Cloud Code or Cursor would just claim the project or feature is completed.

But once you test it, it actually didn't work. So their approach to solve those default model failure behavior is that first they set up initial environment that lays the foundation for all the features that given prompt requires, which set ups agent to work step by step and feature by feature. So this kind of similar to the plan or PRD approach that we normally took. The second is that they start prompt each agent to make incremental progress towards its goal while also leaving the environment in clean state at end of each session.

What they did is starting design this two-part solution. First they would have this initializer agent that is used a specialized prompt to ask model to set up initial environment with a init.sh script, which will set up dev server, for example, so that next model don't need to worry about those things. And also it create progress.txt file that keeps logs on what agent have done, as well as initial Git commit that shows what file has been added. Then it calling agent for each subsequent session to ask the model to make incremental progress, then leave structured updates.

And all those efforts are really try to serve one purpose, is how can they define an environment where agents can quickly understand state of work when starting with a fresh context window. So workflow is that initializer agent would firstly try to set up a environment, or you can call it documentation system, to track and maintain overall plan. And the environment they define here is firstly they will have a feature list documents to prevent agent one shotting the whole app or prematurely considering the project complete. Instead, they would get initializer agent to break down the project into over 200 features and logs them in a local JSON file look something like this, where each task has detailed spec, as well as pass or fail state.

At default, all tasks will be marked as fail. So it force model to always look at overall project goal and the progress, pick up highest priority task and do the next thing. But to make this workflow works, they also need a way to force the model leave the environment in a clean state after making the code change. In their experiment, they found the best way is to ask the model to commit its progress to Git with descriptive comment message and write a summary of its progress in progress file.

But with just documentation and context environment itself, is not enough, because model at default have this tendency to mark something as completed without proper testing. And at beginning, they were just prompting Cloud Code to always do the test after the code change by doing unit test or API test for the dev server. But all those things were often failed to recognize that a feature is not working end-to-end. But things really start changing when they give model proper tooling to do the end-to-end test by itself, like Puppeteer, MCP, or Chrome DevTools, where agent was able to identify and fix bug that were not directly obvious from the code itself.

So, basically, they are setting up a structure where they have the initialized agent to break down the user's goal into a list of features alongside in the SSH to be able to run the dev server and progress files. So, the next coding agent can just read the feature list to get an understanding about overall project plan and pick up high priority task and progress file and get locked to understand where things are at. Then run in the SSH to start dev server immediately and do end-to-end test to verify the environment is clean. So, that it can get a full picture, faster feedback loop while each new session and context window happen.

In OpenAI's blog, they talk about very similar thing. You have to making sure your application environment is legible. They make the whole repository knowledge the system of record. Initially, they put a gigantic agents.md file and fell in predictable ways because it's just too much context for any agent to manage and maintain.

So, what they did is design a proper document environment structure and treat agents.md file as a table of contents. So, they set up this documentation system from architectures, the design docs, the execution plan, DB schema, product specs, and design front-end plan, security, and many more. And put this table of content into agents.md file. So, the agent can actually retrieve back relevant information when needed.

And this enables progressive disclosure. And OpenAI actually do that even further. They would try to push not only the code knowledge, but also Google Docs, Slack message, all those other fragmented information, feed the data into the repository as a repository local version artifacts. So, the agent can also retrieve.

Because from agent point of view, if anything can't be accessed in the environment, then effectively it didn't exist. But again, documentation itself didn't really keep a fully agent-generated code base coherent. They also introduced certain programmatic workflow to enforce invariants. For example, they layered domain architecture with explicit cross-cutting boundaries, which allowed them to enforce those rules with custom checks, linters, and structural tests, which can be automatically triggered and injected by every Git pre-commit.

In those type of architecture, usually you will postpone until you have hundreds of engineer in traditional software company. But with coding agent, it's an early prerequisite. Within those boundaries, you allow teams and agent to significant freedom in how solutions are expressed without micromanaging and worried architecture going to drift. Meanwhile, they are also improved code base a lot.

For example, they made app bootable per Git work trees. So, Codex can just launch and drive many different instance. And they also wired Chrome DevTools protocol into the agent runtime. So, that the agent can reproduce bugs, validate fix by DOM snapshots, screenshots, and navigation.

And with those environment and workflow setup, the repository finally crossed a minimum threshold where Codex can end-to-end drive a new feature. So, every time when Codex receive a single prompt, the agent will start validating the current state of code base, reproduce a reported bug, record a video to demonstrate the failure, implement fix, validate the fix by driving application, record a second video demonstrating the resolution, and eventually merge the change. So, those two blocks show very good learnings and necessary harness system you need to put in place for fully autonomous system. Meanwhile, there are also certain learnings.

Quite often when building agents, especially vertical specific agents, our tendency is to build specialized tooling to do domain specific task. The learning we got is that large language model almost always work better with generic tool that they natively understand. We saw releases awesome article about how they redesign their text-to-SQL agent. So, they spent months building a sophisticated internal text-to-SQL agent D0 with specialized tool, heavy prompt engineering, and careful context management.

But as many of us experienced before, those type of system kind of work, but is very fragile, slow, and require constant maintenance. Because every new edge cases happen, you will need to engineer new prompt to the agent. But later, they tried one thing that totally changed trajectory. They deleted most of the specialized tool from the agent down to a single bash command tool.

And with this much simpler architecture, the agent actually performed 3.5 times faster with 37% fewer tokens, and success rate increased from 80% to 100%. Similar learning has been shared from Entropic team as well, where they talk about instead of having specialized search linked execute tools, they just have one bash tool where it can run grep, tail, npm, npm run lint. And fundamentally, I think it's because all this large language model is much more familiar with those code native tools that has billions of training tokens versus bespoke tool calling JSON that it needs to generate. And I've talked about this in programmatic tool calling video that I released last week.

And I believe it is similar fundamental principles here. But the foundation of this simple architecture is again the good context and documentation environment where model can use generic tools to retrieve context progressively. And it is same case with Open Claw. One reason Open Claw is so interesting is that they have a surprisingly simple but effective context environment.

They have list of documentations to store core information. With this foundation, they only have the most basic tooling like read, write, edit files, run bash commands, and send message. All the rest is coming from giving agent environment to retrieve random context, plus a big skill libraries to expand capabilities. So, those are three practical learnings about how to do harness engineer for long running complex agents.

I said have a legible context environment to enable each session to grab context effectively, and write workflow and tooling so that model can verify its work effectively, drive faster feedback loop, and trust agent with generic tools that it natively understands. Anything interesting, I'm going to share more in depth about how do I take this learnings and transform into a development life cycle process. In AI Product Club, we have courses and walk through about live coding and building production agents. And every week, myself and industry experts share the latest practical learnings.

So, if you're interested in learning what I'm learning every day, you can click the link below to join community. I hope you enjoyed this video. Thank you, and I'll see you next time.

A term that's popping up more and more in the AI space right now is harness engineering. It's the next big thing for this year and just like context engineering was for last year. And it is really important, but just like context engineering, it's starting to turn into this buzzword that people are throwing around without really knowing what it means. And so that begs the question, is this skill or even mindset like I'll get into in a little bit worth learning or adopting?

And the answer is yes. And so I want to get into that with you today. Helping you understand what harness engineering is. There are a couple of layers to it that are really worth knowing.

And so I'm going to break this down nice and simple for you in less than 15 minutes. And of course, like usual, I've got some examples and demos to really make it concrete. All right, so let's get right into it. Harness engineering is all about building the wrapper around the model.

So any agent is the combination of the underlying large language model like GPT or Claude, and then the wrapper around it that gives it the context and defines your processes. And so I'm mostly going to be focusing on AI coding assistants for this video, but really this idea of harness engineering can be extrapolated out to any agent that you build for anything. And there are really two parts of harness engineering. We have within an individual AI coding assistant session, and then we have really the real evolution here that I'm more focused on right now.

This is combining multiple coding agent sessions in a larger workflow to handle a larger task. And so we'll get there, but I want to start more foundational here. And one really important thing to understand is that this first idea of harness engineering within a single session, a lot of the ideas here are very similar to context engineering. This is a direct evolution of context engineering.

How do we give the right ecosystem of context to our coding agent? But there are some important differences here that I want to key in on. But first of all, I think this diagram explains it really well. We start with the underlying large language model.

This is the reasoning for our agent. And then, the first wrapper around it is not something you build yourself. It's actually the tool that you use, the coding agent that you choose. And so, Claude Code, Codex, Py, you name the millions of coding agents out there.

All of them are actually harnesses that a company has engineered around their model. And so, this might not feel like harness engineering cuz you're not defining anything, but you're picking the harness when you choose the tool. Some people think Claude Code is the best harness for coding. Some people think Codex is.

There's a lot of debate right now. But, what's even more important than the coding agent you pick is the AI layer. This is the ultimate wrapper around any coding agent session, and this is what you get to actually build. And so, when we think about what goes into the AI layer, it's really defining all of our contexts and processes for our coding agent.

So, our global rules, our skills, and MCP servers, all the capabilities we give, code-based searching like LSP or knowledge graphs, our hooks, and our sub-agents. Really like these six components that are pretty much built into every single AI coding assistant now makes up your AI layer. So, no matter how you want to inject your process or your rules, you're going to do it through one of these six things. So, there are a couple of articles I really want to lean into here to help you understand harness engineering.

I'll link to them in the description. This first one has an analogy that I want to zoom in on here. I love this. So, on the left-hand side, we have a representation of what the model can do by itself, like Claude or GPT.

And spoiler, it's not that much. We take for granted all of the capabilities that AI coding assistants like Claude Code and Codex give to the model out of the box. An LLM by itself, it doesn't have any way to access a file system or Git or run any commands. That's everything that comes with that first harness layer built into the tools that we download and use out of the box.

And so, that's what these top bridges represent here. It's all of the capabilities that these tools give to the model to make it so it can really act as an AI coding assistant, right? It's the capabilities plus the system prompt built into these tools. And then, as we get to the lower bridges here, this is where we start to get into that higher-level AI layer, where we get to define things like the MCP servers we use, the skills that we build or incorporate, rules, things like that.

Even going down to Ralph loops, like we'll talk about towards the end of this video when we get into a stringing multiple coding agent sessions together, the ultimate kind of harness engineering. So, stay tuned for that. But, the point here is that each of these bridges are tools that allow the large language model to function and act as an AI coding assistant. All right, cool.

So, with that definition, I now want to address the elephant in the room. The question you might be asking yourself is, "Cole, isn't a lot of this here just context engineering? Like, I thought we were covering this in 2025." And the answer is actually yes, to an extent. And that's why I think that harness engineering is becoming such a buzzword right now.

Most people don't really understand how this is truly an evolution from context engineering. That's what I want to argue with you right now. And so, there are two important distinctions. So, first of all, most of the harness around the model, like this article outlines, it is just context engineering.

Your context injection, your actions through tools and MCPs, persistence, observability. The one thing that really is different is control. Like Ralph loops, orchestrating different coding agent sessions and sub-agents, I think that is a true evolution from sub-agents. And so, we'll talk about that next.

But, the other really important distinction that this article outlines is the skill issue reframe. So, I alluded to at the start of the video the fact that harness engineering is not just a skill, it's also a sort of a a mindset, right? A a reframe. So, the author here says, "There's a pattern I watch engineers fall into.

The agent does something dumb, the engineer blames the model, and the blame gets filed under wait for the next version." As in, you know, Claude screws up here, well, we better wait for Opus 5. Or GPT messes up, let's wait for GPT 6. And, you know, personally, I see this all the time as well. I'm also tempted to think this myself, and you probably are as well.

But, the harness engineering mindset rejects that default. And, by the way, I call this system evolution. It's very in line with something that I've been focusing on a lot recently. So, here's what he says, "The failure is usually legible.

The agent didn't know about a convention, so you add it to agents.md. Or the agent ran a destructive command, so you add a hook that blocks it." Basically, the idea here is every mistake becomes a rule. Or the way I like to put it is every mistake becomes an opportunity to improve your harness, improving the security through your hooks, your processes through updating your skills, anything in your harness, so that the next coding agent session that issue you encountered is less likely to come up. And that is super powerful.

That system evolution means that you are taking ownership and improving the performance of your coding agent over time with the AI layer that you control over the coding assistant that you chose. And so, really, harness engineering is all about claiming that agency, taking ownership of your system, so that when something goes wrong, you're not just blaming your AI coding assistant and feeling helpless. Because things will come up. Just like working with human developers, there are going to be issues.

But, we need to make sure that we have a way to learn from that and not just be at the mercies of the next session not encountering that problem again. We want to be the human steering the system, feeding forward. So, the initial generation, we have our principles and other kinds of context we feed in, and then we have our sensors for feedback, our hooks, our review agents, the skills that we give it for that self-correction, evolving our AI layer over time. The sponsor of today's video is Google Cloud, specifically their new agency CLI.

And I'm excited for this because nowadays it's optimal to build your agents with other agents, right? Using your AI coding assistants like Cloud Code and Codex. Now, the easy part is getting the idea for an agent, but actually building it out and deploying it to production, that is a different beast. But Google has made this so incredibly easy now with their agency CLI.

It's a collection of skills that I can bring into my coding agent that give it full, clear instructions on how to build agents with the Google Agent SDK and even deploy them to production and monitor them. And so right here in my Cloud Code, for example, I can say use the agency CLI to build a research agent that searches the web. Obviously, a simple example, but it's going to use the instructions to really help you build any agent that you want. Then with the help of the skills, your coding agent will create all of the code.

It's one shot at a lot of different tests that I've given it here. And then we also have our local development environment. We can spin up the agent here so that we can test everything locally with a full chat application before we deploy our agent. And then when you're ready to take your agent to production, it is a single command to deploy your Google Agent SDK agent to the Google Cloud.

Super easy. And your agent gets its own identity in the cloud. You have the playground here to test it live, and you have traces, full observability. So everything you need for a production deployment, but it's not extremely difficult to get all this set up like it used to be.

And the best part is the agency CLI is free and open source. You can take these skills, bring it into any coding agent, and see how easy it is right now to build any AI agent. I'll have a link in the description. I'd highly recommend checking it out.

So I also have this companion repo for our video, giving you a super concrete idea of what an AI layer can look like. And this is a really good representation, everything here of the AI layer that I'll bring into and evolve in each of my code bases. I want to cover this quick before we get into the last evolution of harness engineering, the really powerful stuff, building workflows where we are bringing together and orchestrating many coding agent sessions. And I have examples for that, like with the Ralph loop in this repo as well.

And so I did promise that this video is going to be shorter, so I'm not going to dive extremely deep into each one of the components of the AI layer, but I do have a video that I'll link to right here where I cover in more detail the rules and skills and LSP and hooks, each one of the components. I want to stay just really high level, give you some golden nuggets here, and then you can of course just give this repo to your coding agent and have it help you implement things and understand everything here. So really the foundation of your AI layer is all of the rules, the constraints and conventions that you want your coding agent to follow, your patterns. And so that's your global rules and any other kinds of on-demand contexts that you have as markdown, Confluence documents, that kind of thing.

And then your skills, these are the workflows that you have for your coding agent. Like here's how you want it to plan, implement, and validate. And I have a separate skill for each because what I really want to do, and I highly highly recommend this, is you want to do your planning, implementation, and validation all in separate coding agent sessions to keep each one of them token efficient and focused. [snorts] And so each one of these skills is going to output some kind of artifact so you can use it as a handoff to the next session.

And so this is kind of getting into stringing coding agent sessions together, but we're still talking about doing this manually, right? Like you'll run the plan and you'll create the plan with the coding agent with one skill. And then you'll take this markdown and then you'll give it to the next coding agent session for the implement skill, right? You go through that systematically yourself.

And so we'll talk about in a little bit how we can really bring all that together. And then as far as hooks go, these are honestly pretty underused. I love using a hooks for a few different things. First of all for security.

So I pre-tool use hook. Basically, this is a piece of code that's going to trigger before the coding agent executes any tool call, like writing out to a file, running any kind of command. And so, this is where we can build in security things like not reading .env files cuz we really don't want that in the LLM's context, or removing directories in a very destructive way, for example. I also like having some kind of stop validation hook.

So, when the coding agent says it's done with the implementation, I want to deterministically run a set of tests. Like, are the unit tests, the linting, the type checking, is all that really passing? Cuz if it's not, I want to force the coding agent to iterate on it until it is. And that's what this hook does.

And then, last but not least, just running a quick lint after every single file edit is really good just to keep your code base nice and clean, which also helps make your coding agents more reliable in the future. So, there you go. Just a couple of golden nuggets for the AI layer that we have here. And I even have instructions in the readme for just running a super basic pip lib.

This is the foundational approach to agentic engineering. So, you plan with the plan skill, just sending in the feature that you want. You iterate on the plan, you produce that markdown document that you then hand off to the implement skill in a separate coding agent session. You also have your validation strategy for the agent to check its own work in the markdown as well.

So, feel free to try that out for yourself. But, now finally, I want to get to the peak evolution of harness engineering. So, I'm going to jump back to the diagram here. Let's talk about orchestrating coding agent sessions.

This is when we can really scale the amount of work that we get done with coding agents. So, the main idea here is you don't just want to take a massive task or PRD and hand it to a single coding agent session. It's not going to be token efficient, and the underlying large language model in the harness is going to be completely overwhelmed. It does not matter how good the AI layer is that you created here with things like your skills and rules.

If you send too much into the LLM at once, it is going to fall flat on its face. And so what we're doing here, orchestrating many coding agent sessions together, is we're giving each coding agent a very focused task. And so these can be sub-agents as well, but like you'll see in the Ralph loop, it's actual coding agent sessions that are handing things off to each other. So we can explore the implementation that comes in from a user requirement, have one agent that writes the plan, send that plan artifact into an implementation, and then for example, this is just an example harness, but we could have many code review agents running in parallel.

This one focuses on security, this one correctness of the implementation, and this one making sure it's as simple as it can be. And then if everything passes, you create the pull request, otherwise you would iterate and keep working on the implementation. And you can do all this manually, like I was talking about earlier. We can go to Claude code once and create the plan, and then open up another Claude code and do the implementation, but the real power with harness engineering here is we can automate all of this.

We can create a system that automatically hooks together all these coding agent sessions with the handoff documents and creating the pull request and everything like that. And that's what the Ralph loop does. So let's go back to our example repo here. I want to show you what this actually looks like.

So the Ralph loop is just one example of an agent harness. But Jeffrey Huntley, the creator of Ralph, he really is one of the pioneers here. This is one of the first example showing in a very basic sense how we can automate stringing together many instances of Claude code, Codex. I mean, you could do this with any coding assistant because basically all it is, I'm not going to get too technical here, but I just want to show you a little bit, is we have a simple script.

This can be a Python script, it can be a bash script. I'm not going to go into the code here, but essentially you give it a larger scope of work, like a massive PRD, and it's going to be responsible for splitting that up into individual tasks, and then running coding agent sessions to handle them one at a time until everything is done. And so you give it a prompt. This is the user input here.

Like these are the different items that I want to build in the spec, and then it's going to produce a plan here. This is the fixed plan. So, like this is what it's going to do in iteration one of the loop, iteration two, iteration three. It's going to keep working, kind of like build up a log as the different Claude code sessions here are running.

And then, when it decides it's done, it's going to produce some kind of indicator of that. Like here, I'm using a done.text. And so, this is where it decides like, all right, all eight spec items from the original prompt are done. We are satisfied.

We can now exit the loop. And that's the only condition, the only way that we can exit the main while loop here is if we have this done.text, and the coding agent is confident that the implementation is done and all the validation is there. And so, just trying to show you Ralph Loop to give you one example of a harness. But you can see the idea here of we are using many coding agent sessions to keep each one very, very focused.

But also, we're automating it, so we don't have to baby sit our coding agent as we're handling these longer tasks. And this really is the future of agentic engineering, building these harnesses to handle larger scopes of work as the models and the underlying tools are getting more powerful. Like this is the way. And so, lean into this here.

I mean, there's so many resources out there for harness engineering. And then, there's Archon, my open-source harness builder, free to use. This is the easiest way to get started with agentic engineering, building your own harnesses like the Ralph Loop, but more custom to you, your exact process and software development life cycle. So, I'd highly recommend checking this out.

Otherwise, I hope this video was helpful for you in general, just understanding what is harness engineering, what's the fluff, what's really worth knowing. If you found this useful in any way, I would really appreciate a like and a subscribe. And with that, I will see you in the next video.