Getting better at Orchestrate
Orchestrate is a 24-hour hackathon organised by HackerRank, where developers design, build, and ship an agent. Every submission is judged on four signals: the code, the performance of the agent build, the AI chat transcript, and a 30-minute voice interview in which participants defend their architecture to an AI judge.
After looking through a range of Orchestrate submissions, this blog is meant to help developers understand what makes a build stronger and closer to real engineering work.
Orchestrate was built around a simple belief: developers will be evaluated not only on whether they can write code, but also on whether they can design systems with AI, test them, reason about their limits, and explain their decisions clearly.
That is why each part of the submission looks at a different signal. The code shows how the system is built. The output shows whether it works. The transcript shows how the builder used AI during the process. The interview shows whether the architecture can stand up to questions and whether the developer understands the architectural decisions.
The problem statement may change across editions, but the core skills stay similar: system design, reliable outputs, good use of AI tools, and ownership. A submission becomes interesting when we can see how you thought through the system.
How to Think About the Build
Do not think of the four submissions as separate things to finish at the end. They are connected.
A strong submission usually reads like a build story. You start with a system design, turn it into a working prototype, test the outputs, use LLMs wisely and take the judgement on when to push back, and show where you made the final engineering call.
Submitting your Code
The code zip is where the system itself is visible.
It should be runnable and readable. Anyone should be able to open it, find the entry point, understand the main flow, and see where the agent decisions happen.
A good README makes the work easier to trust. It should show setup steps, dependencies, environment variables, the command to run, and the files the program expects. This is the same expectation a teammate would have during a code review: they should be able to understand the flow, see the important decisions, and know where to look if something breaks.
Agent Architecture and Design
The useful signal here is whether there is a real system around the agent. How does input enter the system? How is context built? What format should the response follow? What gets checked before the final output is written?
A single LLM call is rarely the whole agent. If every input goes into one prompt and whatever comes back is written directly to the output, most of the architecture is hidden. There is no clear place to inspect context construction, schema validation, retries, error handling, or fallback behavior.
A stronger design gives each part of the system a job. The code can load and normalize inputs, build context, call the agent, parse the response, validate the schema, enforce allowed labels, retry malformed outputs, log failures, and decide when a case should be flagged instead of guessed. The snippet below shows how every input flows the same path: loaded, cleaned, contextualized, answered, verified, and flagged rather than guessed when it fails
This does not need to be a complicated multi-agent graph. A simple architecture is often better if the boundaries are clear. If you use one agent, show what sits around it. If you use multiple agents, explain what each one owns, how they pass context, and how conflicts are resolved.
A useful self-check is to trace one input through the system. Where is it read? How is it cleaned? What context is added? Where is the LLM called? What schema does it return? What does the code verify after the response? What gets logged? When does the system retry, abstain, or ask for review? Where is the final output written?
Code
The code criteria is mostly about engineering hygiene.
Names matter. File names like helper, final, utils, or test2 make the reader work harder than necessary. They also make it harder for you to explain the system later. Use names that reflect the role of the file or function.
A cleaner structure usually separates the main concerns. Input loading, prompts, agent logic, validation, and evaluation do not all need to be in one large file. They also do not need to be split into twenty tiny abstractions.
The small things matter here too. Hardcoded local paths, secrets in code, undocumented manual steps, generated files mixed with source files, or code that only works because of a folder on one machine all make the system harder to trust.
Prompts
Prompts are part of your implementation, so write them with the same care as code. Keep them clear, compact enough to avoid wasting context, and reusable enough that the same structure can handle similar cases without being rewritten each time.
A useful prompt explains the task, the allowed outputs, the evidence the model should use, the format it must return, and what to do when information is missing. If the output must be JSON, say that clearly. If the model should choose only from a fixed set of labels, include those labels. If uncertainty calls for review rather than guessing, say that too.
This matters because challenge inputs can be messy. They may be incomplete, ambiguous, or include text that tries to pull the model away from the actual task. Your prompt should make the task rules clear enough that the model does not treat every input instruction as something to obey.
Guardrails
Guardrails are the parts of the system that keep a fluent answer from becoming a wrong final answer.
With LLMs, failure does not always look like an exception. Sometimes the model returns valid-looking JSON with the wrong label. Sometimes it invents a category. Sometimes it explains the decision in a way that sounds fine but does not actually support the output.
Validate inputs before the model call. Validate outputs after it. Reject unsupported labels. Retry malformed responses within a limit. Add a fallback for missing evidence. Apply deterministic rules for cases where the model should not get the final say.
You will not catch every failure mode. But the common ones should not silently pass into the final output.
Reliability
This is usually where the difference between a demo and a build becomes clear.
A reliable system behaves reasonably across the boring cases, the edge cases, and the cases that are annoying to handle. Empty rows. Missing fields. Ambiguous evidence. Repeated IDs. Inputs that contain instructions the system should ignore.
Run the system on the full dataset. Do not only inspect the cases that worked. Look at the ones that felt off. Did the model over-answer when evidence was missing? Did similar cases get different labels for no clear reason? Did one bad row break the entire run?
This often separates a quick prototype from an engineering submission.
Evaluation
Evaluation is the process of checking whether the system behaves the way you intended across the cases you care about.
Pick some cases, decide what you expected, compare the output, and write down what you learned. Look at both the final decision and the reasoning behind it.
The loop matters more than the size of the benchmark. Build something, run it, inspect where it failed, change the system, and run it again. That is the habit we want to see. LLM systems rarely become reliable because the first prompt was perfect. They improve because someone looked at the failures and made the system less fragile. If you use an LLM as a judge, treat that as part of the evaluation design: define what it should look for, pair it with problem-relevant metrics, and check whether its judgments match the behaviour you actually want.
Error Handling
Think through the failures that are likely for your design. You do need to make sure bad intermediate states do not quietly become final answers. Log failed rows. Continue safely when possible. Mark uncertainty when that is the responsible thing to do. Do not write unsupported labels to the output just because the model produced them.
This part is not glamorous, but it shows whether the system was built only for the happy path.
Preparing the Output
In real engineering work, this is similar to the feature running in production for real users. The question is not only whether the system was built, but whether it behaves correctly in the cases it was given.
A useful final pass is to open the CSV like someone who has never seen your project. Are all required rows present? Are any IDs missing or duplicated? Are the columns exactly what the instructions asked for? Are labels from the allowed set? Do similar cases receive similar treatment?
The justification field should help us understand why the system made the choice. If evidence mattered, refer to it plainly. If the case was uncertain, make that visible. If the system chose review, explain what triggered that path.
Handling All Cases
Some cases are straightforward. Some are incomplete. Some are risky. Some are ambiguous. Some may contain distracting instructions. Your system should not treat all of these the same way.
Consistency is the thing to watch. If two similar cases are handled differently, there should be a reason. If you cannot explain the reason, it is probably a prompt, guardrail, or validation issue.
In real systems, most failures happen in the spaces between the happy paths: missing fields, conflicting signals, inputs that look valid but are not, model outputs that do not match the schema, and cases where the safest answer is to escalate or abstain. A strong submission shows that these cases were considered before the final output was produced.
Building the Chat Transcript
Plan, build, and review mirror how real engineering work happens: first you shape the problem, then you build with AI, then you review the output the way you would review code.
What matters is whether you led the process. A useful transcript shows how you understood the problem, broke it down, asked AI for help, reviewed what it produced, and corrected the system when it failed. The provided AGENTS.md is there to capture that trail.
A good engineering trail makes the work easier to trust. If the AI wrote code, show how you reviewed it. If it suggested a complicated design you did not need, show that you pushed back. If something broke, include the error and the reasoning that led to the fix.
Plan
The planning part should show that you understood the problem before jumping into implementation.
It can be short. If you give AI a vague request like build the solution, it is hard to see your judgment. If you give it a shaped problem, your ownership is much clearer.
Useful planning usually sounds like an engineer setting constraints: here is what the system needs to read, here is what it needs to produce, here is where the model should reason, here is where code should enforce rules, and here is what should happen when evidence is missing.
Build
The build section of the transcript should have technical specificity.
Name files. Name functions. Mention schemas. Talk about thresholds when they matter. If you add retrieval, say what is being retrieved and how it is passed to the model. If you add validation, say what makes output valid.
Review
After the system runs, spend time with what it produced. Treat this like code review, except the thing you are reviewing is the system’s behavior as much as the code. Look at what the AI produced, question it, ask what would break, and then iterate on the prompt, validation, or system design. The transcript should show some of this loop.
This is very close to real engineering. You build something, it behaves strangely, and then you have to decide where the fix belongs and how to fix it.
Talking through the Judge Interview
The interview is where ownership becomes visible.
Think of it like a real architecture review or handoff. Engineers are often asked to explain why a system was built a certain way, what can fail, what was intentionally left simple, and what they would improve with more time.
Know Your System
Start by walking through how the system design works. Explain where the model is called, what context it receives, what it is allowed to decide, what the code validates, and where the final output is written.
This includes the architecture, important files, main functions, prompt, schema, and failure paths. It is fine to say AI helped you write parts of the code. What matters is whether you can explain the code you submitted and the choices behind it.
Show Judgment
In the interview, good judgment looks like being able to explain why and how the system works the way it does, what you kept simple, where you added guardrails, and what you chose not to automate.
This is where product thinking and edge-case handling belong. What would a user need to trust the output? What information should be visible in the output because it helps someone act on the result? Where should the system ask for review instead of pushing forward?
Edge cases are part of judgment too. What happens when required information is missing, signals conflict, the input includes distracting instructions, or the model returns something outside the expected schema? You do not need to claim every edge case is solved. You do need to know where the system can bend or break.
Show Technical Depth and AI Fluency
A good answer here is usually specific: how the architecture works, where AI helped, where it introduced risk, and where plain code did the cleaner job.
Be ready to explain what you asked the model to do, what you kept outside the model, and how you checked the model’s work. A fluent developer can talk about prompts, context, validation, retries, evaluation, hallucination risk, and failure recovery in plain engineering terms.
Communicate Clearly
Clear communication matters because interviews are short. Start with the direct answer, add one concrete detail, tie it back to the task, and then stop. You can always add more if the judge asks.
This is also where uniqueness and honesty fit. Point to the part that actually improved the result, and be precise about limitations: what you tested, what you did not test, where the system is fragile, and what you would improve next.
What’s next from here?
Orchestrate is built to help developers become more AI-first: to build with AI, test the system, understand the limits, and explain the choices.
Read more about the experience from the developers
Can you give a brief on what those things are that tell some chat transcripts are good and some are bad, and how you rank candidates based on the chat transcripts they provide