On 30 August I went to my first hackathon, the Daytona HackSprint at Entrepreneurs First in Shoreditch. Outpost ran it, with Daytona, OpenAI and Parallel behind it, and the brief was seven hours of building with agents. Daytona's part is sandboxes: small cloud computers you can start, control and throw away from code.
The room
I did not really know what to expect walking in. I had never been to a hackathon, I was the youngest person there, and most of the room looked like they had done this plenty of times before.
The venue was unreal. Entrepreneurs First is in the Shoreditch Exchange, a huge open floor with long wooden desks, plants everywhere and their neon sign glowing on a purple wall. Daytona and OpenAI had banners up between the desks, and one of them just said You Can Just Build Things, which is a very good thing to read at half nine on a Sunday morning.
The people were the best part. Everyone there had come to actually build something, and between coffee and the last push before demos I ended up talking to people who had been doing this for years and were still happy to explain how they were going about it to a first-timer. I left with a handful of new people in my phone.
Seven hours goes very fast. Somewhere in the middle of the afternoon you work out how much is left to do and how little time there is, and the whole room goes quiet while everyone does the same maths. It was stressful in the best way, and I loved it.
Where the idea came from
A guy I know at Cambridge built Zoral. It sits on someone's laptop, watches them do their job, learns it, and then starts doing the job itself on the same machine. He applied to YC with it and did not get in, but he made the top 10% of applicants. I had been talking to him about it, so it was on my mind going in.
When I read what Daytona does, it looked like the right thing to build a version of Zoral on. Zoral learns on your laptop and works on your laptop. I wanted to try splitting those up: learn from a recording made on my Mac, then do the work in the cloud, where one recording could run on a dozen sandboxes at once with a different row of data in each.
What I built
It was called Owari for the first few hours and Crewmate after that. You record yourself doing a task once. Crewmate works out what the task was, turns it into a plan, and runs that plan on up to 12 agents in parallel, each on its own Daytona sandbox with its own row of a spreadsheet.
The problem is that a lot of the software businesses run on is old and has no API or MCP server for an agent to talk to. Claude Cowork and ChatGPT can already use a computer, but they work through one inference at a time, so for a task that has to be done five hundred times over they can end up slower and more expensive than a person doing it.
A small Electron window floats over everything and records your screen. The recording gets cut into about 90 stills and sent to a vision model, which sends back a plan. A dashboard then replays the recording at 8x speed while the plan builds up next to it, so you can see which bit of the video each step came from. You can change or delete any step, and then you press run.
Not clicking where I clicked
The recording is made on a Mac at one screen size and run on Linux at another, so saved mouse positions are useless. Instead, every step points at a control by what it is called, like the link named Leads. When it runs, each sandbox reads the live accessibility tree through Daytona's Computer Use API, finds the control with that name and acts on it. I made the plan's schema reject any step with an x or y in it, so coordinates could not sneak back in.
The first version matched on a control's name and its role, and plans that were right kept failing. A model watching a video can read a button's label, but it has no way to tell whether that button was built as a link, a button or a table cell. In my own test app the Open button turned out to be a table cell. So I changed it to match on the name and only use the role to pick between candidates, and after that plans that got several roles wrong still ran every step.
A plan instead of an agent
The obvious way to build this is an agent that looks at the screen and decides what to click at every step. That takes seconds per click, does something slightly different each run, and with several sandboxes going it would not have finished during a demo. So the model runs once, when it reads the recording, and what comes out is a fixed plan. A ten step task ran in about 13 seconds. Later on I added one exception, where the model gets asked for help only if a step fails.
The part I liked most was pruning. When you record yourself you make mistakes. You open the wrong menu, go back and try again. The model is asked to separate what you meant to do from what you actually did, and it lists the dead ends it left out along with why, like opening Settings to look for a report template and finding everything greyed out.
Each sandbox boots from a snapshot I had already logged into by hand, so Crewmate never handles a password. Values from the spreadsheet get filled into each copy of the plan, and if a step's condition does not apply, that sandbox finishes as skipped rather than failed. In the run I was happiest with, one sandbox did all ten steps and saved a report, and the other searched, found no matching record and skipped.
Three agents at once
I built it with three coding agents running at the same time. Codex did the recorder and the part that reads the video. Claude Code had two sessions, one on the dashboard and one on the server and the code that drives the sandboxes.
Before any of them started I wrote a spec and a dependency map for them to share. The plan format was frozen as a JSON schema, each agent owned its own folders and was not allowed to edit anyone else's, and the map said which parts depended on which interfaces, so a change to one of those could be passed along instead of two agents quietly disagreeing about it. There were no merge conflicts all day.
That was the most useful thing I took away. Letting agents do more on their own is easy. What takes the work is setting up the structure and the limits that let several of them build separately and still agree on what the system is.
Why it did not win
By the end of the day it worked. You could record a task and watch it run on several cloud computers at once in real time. It did not win, and looking back I get why.
However I explained the parallel part, what people saw was basically a worse version of Claude Cowork or ChatGPT. Those already watch a screen and do the task for you, and they are built by teams with a lot longer than seven hours.
It was also my first hackathon, and I did not really understand what the day was for. I treated it like building a product: get the architecture right, stop it leaking sandboxes, handle the edge cases. That matters if people are going to use the thing. At a hackathon it is time taken away from the demo.
What I know now is that you are judged on a few minutes of demo. You need a sentence that says what it is, a moment where people actually react, and something that makes sense from the back of the room. Most of what I was proud of was underneath, where nobody watching could see it.
I still left with a lot of API credits and a much better idea of how to run agents at scale. Next time I would pick the demo first and build towards it.
