But it felt like solving a different problem, one that did not have a name yet.

The largest available context window at that time was 32K tokens. We started with an 8K model and, within weeks, hit our first context length errors. The obvious fix was to switch to a bigger model when conversations exceeded 7K tokens. But as I debugged why we were even hitting those limits, I realized something important: the problem wasn’t the size of the context window. It was what we were putting in it.

We did not have to just engineer prompts, but also the context itself. Which information to include, when to include it, how to structure it, what to summarize, and what to discard. This was a different discipline entirely, one that would become the foundation of every scalable AI application I have built since.

Back then, nobody called it “context engineering.” Now, it is the difference between a demo and a production system.

Context Window Illusion

Context feels infinite when you are processing single documents or short conversations. Production changes that illusion very quickly.

Here is what large context windows do not protect us from.

Cost explosion

Every token sent to the model has a cost, both on input and output. A 100K-token context on every request is not using the available capacity. It is spending the budget on information that often does not contribute to better outcomes.

Latency degradation

Models do not scale linearly with context length. A tenfold increase in context does not result in a tenfold increase in latency; it is often worse. In production systems, users are unlikely to tolerate long response times simply because context was not curated deliberately.

Quality deterioration

This is the most subtle failure mode. Models perform worse when buried in an irrelevant or weakly related context. As the number of tokens in the context window increases, the model’s ability to accurately recall and reason over specific information decreases.

This matches how LLMs work. Transformer-based architectures allow every token to attend to every other token in the context, resulting in n² pairwise relationships for n tokens. As context grows, attention is spread thinner. You can think of this as a fixed attention budget that gets depleted with each additional token.

State management chaos

When context grows without bounds, it becomes difficult to reason about application state. What information does the model currently have access to? What shaped its previous response? Without clear boundaries, these questions become hard to answer.

Why Context Engineering Isn’t Prompt Engineering

In the early days, much of the industry’s attention was on prompts. There were countless blog posts on few-shot examples, chain-of-thought reasoning, and role-based prompting. These techniques are useful, but they address only part of the problem.

Prompt engineering asks: How do I instruct the model?
Context engineering asks: What information does the model need, and when?

Consider a code review assistant.

A prompt might say:
”You are an expert code reviewer. Identify security vulnerabilities, performance issues, and maintainability concerns.”

Context engineering determines everything around that prompt:

• Do we send the entire pull request or only the changed files?

• Do we include full files or only the modified sections with surrounding context?

• When do we inject relevant documentation or coding practices?

• How do we maintain state across multiple files?

• What information from earlier reviews should influence later ones?

The prompt remains relatively stable. The context is dynamic, stateful, and intentionally scoped.

As AI applications evolve from single-turn interactions to long-running, multi-turn systems, context engineering becomes increasingly central.

Context Engineering

After spending a few years building production AI systems, I have found that reliable applications tend to rest on three context-engineering pillars.

Summarization: Compressing History Without Losing Meaning

A common approach to conversation history is simple accumulation: append every message to the context. This works initially, and then fails abruptly.

Effective summarization is not truncation. It is signal preservation.

For a code review assistant handling a pull request with dozens of files, the LLM does not need verbatim memory of every earlier file. What matters is:

• Architectural patterns in use

• Security issues identified across files

• Repeated coding-standard violations

• Shared dependencies that were modified

• Bugs that may cause cross-file breakage

Maintaining a running summary that is updated as the review progresses keeps the active context focused while preserving coherence.

History describes what happened.
Context captures what matters now.

For long-running tasks that approach context limits, summarizing and restarting with a distilled representation is often more reliable than continuing to append raw history. Many modern tools already apply this pattern implicitly.

Chunking

A code review assistant does not need the entire codebase in context for every decision.

Instead:

• Parse the pull request into changed files and diff hunks

• Index chunks with metadata such as file paths and change types

• Retrieve only the chunks relevant to the current review focus

If the model is reviewing a Rails route that imports a utility function, pulling in that utility’s definition and usage examples is usually sufficient. Test suites, build scripts, and unrelated APIs add noise without improving the review.

This approach improves both token efficiency and output quality by narrowing the model’s focus to what is relevant for the current step.

The same principle applies beyond code: documentation, prior review comments, and best-practice references should be retrieved selectively rather than batch-loaded.

Increasingly, systems are shifting toward just-in-time context retrieval. Instead of preloading all information, agents maintain lightweight references and fetch details dynamically as needed. This mirrors how humans work: we rely on indexes and retrieval, not full memorization.

Stateful Context: Right Information, Right Time

One recurring failure pattern in production systems is treating context as a dumping ground: collect everything and hope the model infers structure.

Reliable systems make state explicit.

For a code review assistant, this might include:

• Current review phase (initial scan, deep analysis, final summary)

• Files already examined and their complexity

• Issues identified by category

• Patterns detected to avoid repetition

• Cross-file dependencies that require coordination

Each state determines what enters context. Early phases load metadata and structure. Deep analysis loads specific diffs and related files. Final recommendations load summaries grouped by severity.

This approach designs context through awareness rather than accumulation.

A useful technique here is structured note-taking. The agent writes notes to external memory and selectively reintroduces them into context later. This enables persistence without continuous context growth.

Patterns That Scale

Several patterns consistently improve context quality in production systems:

1. Semantic compression: Do not just truncate directly; understand what information is semantically important and preserve that.

2. Hierarchical context: Maintain context at different levels of granularity. High-level summary always present, detailed information retrieved on-demand.

3. Temporal awareness: Weigh recent context more heavily than distant context.

4. Scope-based retrieval: Match context type to task phase. Different review phases need different contexts. Security analysis needs authentication flows and data validation code; performance review needs database queries and API calls; style review needs coding standards and formatting rules.

5. Explicit state tracking: Track progress outside the model instead of relying on inference.

6. Progressive disclosure: Let the LLM incrementally discover relevant context through exploration. When it identifies an imported function, it can fetch that function’s definition. When it finds a database query, it can retrieve the schema. Each interaction yields context that informs the next decision.

7. Multi-agent architectures: Delegate focused tasks to specialized agents with clean context windows and merge distilled results. Rather than one agent reviewing the entire PR, specialized agents handle focused tasks with clean context windows. The main agent coordinates with a high-level plan while subagents perform deep technical work or use tools to find relevant information. Each subagent might explore extensively, using tens of thousands of tokens or more, but returns only a condensed, distilled summary of its work.

8. Context-efficient tools: Design tools that are token-efficient and unambiguous. Make sure there is no bloating of tools, leading to confusion for the LLM on which tool to use.

Building With Context in Mind

Start with context budgets. Decide upfront: this feature gets at most X tokens of context. Then engineer to that constraint.

Instrument everything. Track what context is getting sent, what the model uses, and what it ignores.

Test context degradation. What happens when conversations get long? When users upload large documents? When does your knowledge base grow?

Design for stateful context. Your application has a state, conversation phase, user intent, and task progress. Use it to determine the context to be sent to the LLM.

Treat context as a first-class concern. Treat context as part of core architecture, not a post-hoc optimization.

Think in attention budget. Every token you add depletes the model’s ability to focus on what matters. Be ruthless about what earns its place in context.

Conclusion

We now have models with massive context windows, yet production experience repeatedly shows that smaller, better-curated contexts lead to more reliable systems.

Context engineering is not about working around limitations. It is about building systems that remain predictable, efficient, and understandable as they scale. It reflects a shift from asking “How do I prompt this?” to “What information does this system need at this moment?”

Context will not manage itself.

Do not take context for granted. Engineer it intentionally, measure it constantly, and optimize it relentlessly.

Your production systems and your users will thank you.

Press a little deeper, and quality stops being a universal truth. It becomes a moving target, a mirror reflecting priorities. For one, it has zero bugs and predictable behavior. For another, it is speed. For someone else, it is user experience.

I have cycled through all of these at different points. First, I thought quality was about clean code. Then, it meant the absence of bugs. Then, performance. Then, seamless user flows. At one point, it was developer experience; how easy it was to build and maintain. Eventually, it became about solving the actual user’s problem. Over time, I have realized quality is not one thing. It is a composite of many properties, and which ones matter most depends on the trade-offs you are willing to make.

The easiest way to talk about quality is through numbers: uptime, error rates, and latency.

Metrics are useful because they are tangible. You can track them, chart them, and set thresholds. But they can also be deceptive. A system can “pass” all the numbers and still feel low quality due to clunky flows or answers that are technically correct but miss the human tone.

But numbers alone cannot capture the messy, human side of quality; the trade-offs every team eventually faces.

One of the lessons I have learned: quality almost always competes with something else.

Ship fast or refine further. Add more features or keep fewer, more meaningful ones.
Innovate quickly or maintain stability.

When deadlines loom, quality quietly shifts from the ideal we would aim for in a perfect world to the minimum level that feels acceptable under the circumstances.

My real aha moment has come from reading user feedback and watching people use the product.

It was then that it hit me: quality is not just about correctness or speed. It is about whether the user walks away feeling their problem was solved.

No metric captured that. It is subjective and human, and yet it makes the difference between a feature that worked and one that failed.

I no longer see quality as a checklist. My working definition is

Quality is the degree to which a product consistently meets the right needs, for the right users, in the right context while balancing the trade-offs that actually make sense.

It is not static. It needs to evolve with the product and user expectations. The balance between speed, cost, and excellence shifts depending on where you are in the product lifecycle.

The danger is not having different definitions of quality; it is assuming everyone is aligned on the same one.

Engineering might push for reliability and scalability, design for usability, and product for market fit. None of them is wrong. But if you do not make those differences explicit, you end up with decisions that satisfy no one.

In practice, defining quality together forces the hard conversations. It makes trade-offs deliberate instead of accidental. It helps avoid the trap of saying we need to improve quality without agreeing on what that actually means. It might mean sitting down before a release and agreeing on the absolute necessary items for that particular release.

Every team’s definition of quality is really what it is willing to trade and what it refuses to compromise on.

I still do not have a universal definition that works everywhere. But I have learned that when a team aligns on quality for this release, this user group, this moment, the work sharpens, decisions get clearer, and outcomes feel intentional.

I am still learning and redefining quality. Maybe that is the real lesson: quality is not defined once. It is something you define, defend, and refine again and again as you and the product grow.

When I started my career, I initially worked on various products, none of which were successful. It was probably never the effort we put in that became the bottleneck, but a mix of market conditions, timing, and the hard truth of no product–market fit. Those years taught me the craft of building and the satisfaction of elegant solutions, but they also left me wondering what product market fit might actually feel like.

That is why this year stood out. We had two big releases, and on the surface they might just feel like product milestones. But for me, they became mirrors. In each, I met a different version of myself; versions I maybe did not even know existed.

In the first, my world shrunk to code, features, and problem-solving. I was heads down, deep in the craft, living in the details. I chased down bugs late into the night, obsessed over edge cases, and felt the quiet satisfaction when things finally clicked. Exhausting, yes, but clear. Seeing the product work, with users engaging with it, made up for everything. That version of me felt familiar.

The second launch refused to let me stay in that comfort zone. I was still building, still obsessing over details, but I could not stay heads down. Dependencies had to be untangled, conversations held, threads pulled together. My calendar filled with check-ins, my days with code reviews, and my evenings with last-minute fixes and coordination. It was no longer just about solving my own problems; it was about making sure everyone around me moved forward together.

It was noisier, more chaotic, and often draining. But it stretched me. In that version, I saw someone who had to zoom out, connect dots, and hold the bigger picture without losing the details; someone bridging deep technical work and the messy reality of building a product outside the code. Still, there is much to learn.

Looking back, both launches mattered for reasons beyond the products themselves. The first reminded me of the joy of building, the flow that comes from losing yourself in your work. The second showed me the weight and reward of orchestrating more than just my own contributions when something is at stake.

Together, they revealed two sides of me I had not fully recognized. One that finds deep satisfaction in craft and precision. The other that can hold complexity and help others move forward together.

The contrast with my early years is sharp. Back then, I was building in isolation, both technically and organizationally. This year, I learned what it means to build with purpose, with others, and for something that has real traction. Those failed products were not wasted time; they were the training ground that prepared me for when it finally mattered.

If so far this year has been about discovering these two sides of myself, then the next is about learning to bring them together; moving fluidly between deep building and thoughtful orchestration, knowing when each is needed.

At the end of the day, it is never about perfection. It is about patience, about learning as you go, staying curious, and enjoying the work along the way.

It Starts With Code, But It Doesn’t Stay There

Most of us learn debugging in code. You hit a 500, a failing test, and log everything, throw in some print statements, and stare at a stack trace until something clicks. Over time, you get better. You learn to isolate, to reproduce, and to reduce.

But then, as you work on more complex systems, debugging shifts. It shows up not just when you hit a bug in your codebase, but when a user drops off unexpectedly, when a system slows down without explanation, when a product decision doesn't work out the way you had hoped, and sometimes even when our assumptions are wrong. The skill evolves from just fixing bugs to a way of thinking, one that stays relevant no matter how much the tech changes.

What I have found is that I am not always trying to fix bugs anymore. I am trying to become someone who can debug anything that comes up.

Dealing with Gray Areas

Most real-world systems don't break with clear error messages. They degrade subtly. A feature feels sluggish. A model's responses seem "off." An integration half-fails without throwing.

In AI applications especially, you won't always get deterministic failures; you get symptoms: unexpected behavior, weird edge cases, and users reporting "it feels broken" without being able to say how.

When you are building AI systems, debugging takes on a whole new dimension. There is no single root cause, just a tangled mix of model behavior, data assumptions, retrieval failures, evaluation noise, and blurry feedback loops. The bugs aren't always obvious, and sometimes, they don't even look like bugs.

I have spent hours staring at outputs, asking myself: Is this bad behavior, or just expected ambiguity? Did the model hallucinate, or did we prompt it wrong? Debugging AI isn't about fixing lines of code. It's about tracing the problem across layers: product, infra, model, expectations, and even user perception. It's one of the clearest reminders I have gotten that debugging is not just technical; it's about the entire system.

I have found that it is not always about having the answers but being relentless about finding the questions. It has given me a way to chip away at ambiguity, one hypothesis at a time.

Navigating the Unknown

Systems are always going to break, and when something breaks, it's not about panicking but staying curious. We can't predict everything. But we can get better at navigating the unknown. That's what's made debugging feel like such an enduring skill.

Every system failure becomes an opportunity to understand something better. That broken feature reveals an edge case that was not considered. That misbehaving model shows where the data assumptions were wrong. That confusing UX flow highlights a gap between what users want and what we built.

You can read best practices all day. But real insight often comes from debugging the mess. That's where systems reveal their true shape.

The Muscle That Never Stops Growing

I have come to see that I never stop debugging, not because systems are bad, but because systems are alive. As engineers, we are constantly navigating change: shifting requirements, new constraints, evolving tools, and unexpected outcomes.

It's taught me to zoom in and zoom out, to trace problems across logs, layers, teams, and sometimes across months of decisions. It is how I have developed some sense: not just in code, but in architecture, product, and trade-offs.

The deeper I go, the more I realize: I never stop debugging because building is just debugging in motion. Building and debugging aren't separate phases. They blur. Every feature is full of tiny, invisible bugs in understanding and in decisions, and we only find them by being willing to look closely.

Some days, debugging is about fixing a crash. Other days, it's sitting with a vague discomfort that something's off and trying to trace where it began. Either way, it's the same muscle.

New frameworks will come. Languages will change. But I have found the ability to debug, to calmly untangle the mess, and to move toward clarity never stops being useful. It's the most transferable skill I have encountered. It works whether you are knee-deep in logs or navigating an issue faced by the user. It has become how one can make sense of chaos and still ship things that work.

I don’t think debugging will ever stop because things never stop changing. And the more you practice, the better you get at spotting what's going on underneath the surface.

This post is a reflection on what I have learned about the AI Engineer role, not from hype cycles, but from building and shipping LLM-powered systems. It’s about the day-to-day engineering muscle it takes to turn an idea into something resilient, useful, and observable. Along the way, I have come to see this as a new kind of engineering discipline; one that blends software fundamentals with a feel for probabilistic interfaces, product intuition, backend systems, and user trust.

How AI Engineering Differs from ML Engineering

Traditional ML follows a Data → Model → Product pipeline. AI engineering reverses this: Product → Data → Model. You start with the product experience and work backwards.

The focus shifts from model training to system composition. There are parts of assembling, prompting, evaluating, and routing models, not fine-tuning them. This is similar to how backend engineers compose microservices rather than building monoliths, but with the added complexity of probabilistic outputs.

It also involves tighter feedback loops. There are rapid iterations based on user interactions, not just batch metrics. This is more like web development's A/B testing culture than traditional ML's training cycles.

Unlike traditional software engineering, where a function either works or doesn't, AI engineering operates in a permanent gray area. Your system might work perfectly for 95% of cases, degrade gracefully for 4%, and catastrophically fail for 1%, and that 1% might be the cases your users care about most.

This creates three core challenges that define AI Engineering:

1. Reliability: Users expect deterministic experiences from probabilistic systems.

2. Economics: Every interaction costs real money, and costs scale unpredictably.

3. Observability: Traditional monitoring tells you almost nothing about why an LLM failed.

Prompt Engineering

At the start, a good prompt feels like a clever trick. You tweak a few words, get better output, and move on. But once your system hits production, with real users, edge cases, and expectations, prompts stop being experiments. They become product-critical interfaces and need to be treated the same way as code.

And like any code, prompts need to be versioned, tested, observed, and maintained.

Here is what that looks like in practice:

• Templating and Reusability: Writing prompts by hand doesn’t scale. Templating systems like Jinja2 can be used to compose reusable blocks of personality and rules into modular prompt templates. This allows for the separation of concerns, avoids duplication, and supports scalable experimentation. A small change in tone or context structure shouldn’t require rewriting 20 different variants. This keeps your prompts DRY and modular.

• Version Control and Change Tracking: Prompt templates can live in source control, often alongside application code. Every change is tracked with diffs, and changes undergo review like any other pull request. This helps in maintaining changelogs so we know when a prompt changed, why, and what its impact was. This makes it easier to trace regressions from user-reported bugs.

• Few-shot and Zero-shot Examples: Few-shot prompting often improves instruction following, but the examples need to be diverse and updated consistently. These examples also require maintenance, as they might go stale as product requirements change or as new failure modes emerge.

• Snapshot Testing: Every prompt version is snapshot-tested against a fixed set of inputs to ensure behavioral stability. This means generating outputs using the new prompt, comparing them against expected outputs (or embeddings of them), and flagging deviations beyond a defined threshold. This catches subtle drifts early.

• Prompt Versioning: Multiple versions of a prompt can be managed via feature flags or routing rules. This allows A/B testing, gradual rollout, or fast rollback if a new version causes regressions.

When I started building applications with LLMs, the prompts were duplicated in multiple places, and as the system scaled, maintenance became an overhead; that’s where the above practices helped.

Context Length

LLMs have a strict context window of 8k, 32k, 128k tokens, depending on the model. Theoretically, that sounds like a lot, but in practice, it runs out fast. Once we start adding system prompts, user history, retrieved documents, task-specific inputs, and metadata, we might be racing the token limit.

With limited tokens, what you include and exclude matters. Similar to how overfitting and underfitting work in machine learning (ML), understanding the right context for a given task is important, as more isn't always better.

Here is what that looks like in practice:

• Identifying what Context Matters: Not all tokens will be equally important. Understanding which parts of the prompt directly influence the model’s output and which parts are noise is important. This comes down to experimenting: removing one instruction might degrade the answer, while trimming three paragraphs of user history does nothing.

• Retrieval Scoring and Ranking: If we are using systems like RAG, we might end up retrieving more than the context can fit. To avoid this, retrieving information based on semantic relevance measured by cosine similarity can help in avoiding the token stuffing problem.

• Context Truncation Strategies: If the limit is hit, there are two approaches; either do a hard truncation or truncate based on the semantic structure to avoid cutting a paragraph in between.

RAG Systems

The core idea is to take a user query, retrieve relevant information from a custom knowledge base, and include that context in the prompt.

Here’s what typically goes into building a RAG setup:

• Choosing the Right Embedding Model: Embedding quality has a big impact on what documents get retrieved, which in turn affects the AI system’s performance; hence, choosing the right model is crucial to boost performance. One of the crucial steps is also to ensure the model used for generating embeddings is the same for the user query input and for the underlying information.

• Index Management: As the data changes, embeddings and indexes need periodic updates to give the right context to the LLM.

• Retrieval & Chunking: A simple vector search based on cosine similarity is a strong baseline, but results can be refined with re-ranking models or keyword scoring to boost precision. To select the right context, we can chunk the information based on the token or semantic chunking.

Evaluation

Evaluating LLM systems is less about asserting the right answers since the outputs are usually open-ended, but they are more about understanding behavior. Unlike traditional software, where a test case either passes or fails, LLMs operate in a wide spectrum where responses can be technically correct but unhelpful, or partially wrong yet directionally useful.

Because of this, evaluating an LLM usually is a combination of automated metrics, LLM as judges, and human feedback to get a clearer picture of quality and drift over time. Evals for an LLM are still an evolving space.

Here is what that looks like in practice:

• Golden Datasets: These are curated sets of inputs with high-quality expected outputs, often reviewed or written by humans or gathered from users using the product. They act as ground truth baselines for evaluating new models, prompt versions, or any config updates. Updating this set periodically ensures it stays representative of real-world usage.

• LLMs as Judges: In many cases, a secondary model can be used to evaluate primary model outputs by asking questions like: “Does this follow the instruction?”, “Is it factually accurate?”, or “How helpful is this response?” While not perfect, LLMs-as-judges offer fast and scalable assessments when paired with clear criteria and examples. While it sounds efficient, using one LLM to evaluate another can have systematic biases that can invalidate your entire evaluation pipeline. This includes bias in terms of length, style of responses, and confidence.

• Semantic Similarity: Metrics like cosine similarity can be used to compare outputs against the high-quality outputs from the golden dataset. These are useful in regression tests, where we want to confirm that behavior hasn’t unintentionally drifted, even if wording has changed.

While the above three sound efficient, they are still evolving, and automating LLM evaluation is still a challenging technical problem.

Below are the two approaches that can be practiced from the start:

• Human-in-the-loop: You will want humans reviewing model outputs regularly, especially on critical user flows or ambiguous completions. This helps catch nuanced issues like tone, missed edge cases, or unexpected model shifts. This also involves sometimes executing the prompt 10-20 times with the same inputs to make sure that we get good results each time.

• User-Tagged Feedback Loops: Collecting feedback directly from users, whether through thumbs up/down or feedback comments, is one of the best ways to understand real-world quality. This feedback can be used to enrich eval datasets and train internal judge models to align closer with user expectations.

Qualitative and Quantitative Metrics

Once LLM is deployed, measuring quality isn’t as simple as tracking accuracy or latency. Two answers might look very different, but both might be correct. Or a response could be factually accurate but completely miss the user’s intent.

This is why both quantitative and qualitative metrics matter.

Here’s how this typically looks in practice:

• Quantitative Metrics: These are the numbers you can graph and track over time. Common examples include:

  • Overall Latency: time per inference, useful for tracking user experience.

  • Time to first token (TTFT): the time a user has to wait before they see the first word/character/token.

  • Token usage: input/output tokens per call, helpful for understanding cost implications.

  • Instruction adherence: measured via LLM judge scores or simple heuristics.

  • User Feedback: a metric to track the sentiment of users

• Qualitative Metrics: This is where human insight comes in. These include understanding whether the model made something up, were the responses were helpful, and whether it matches users expectations.

• Feedback Integration: One of the ways I think that helps is to merge the above two loops, like tracking a drop in instruction adherence/user feedback (quant), might trigger a human review (qual), or even upvoted completions from users can be added to golden datasets or used to fine-tune LLM-as-judge prompts or repeated tags of too verbose can inform prompt compression or instruction tweaking.

Model Selection

Choosing the right model is like choosing the right tech stack. There is no one-size-fits-all.

Here’s how this looks in practice:

• Matching model to task: Different models have different strengths. Some are better at structured tasks like classification or code generation, others handle open-ended reasoning more effectively, and some are optimized for multimodal inputs like images. For one of the features I was building, GPT-4 was hallucinating with JSON responses, and moving to GPT-4o with structured outputs improved the accuracy. Understanding these strengths and doing small-scale evaluations can help you match the model to the specific job at hand.

• Token limits and cost trade-offs: Each model comes with constraints on input/output tokens, rate limits, and cost per token. These impact both latency and scalability. For example, a model with high accuracy but a 10x cost may not make sense for a high-throughput endpoint. Production choices often come down to balancing capability with cost per call, rate limits, and total monthly budget.

• Fallback and multi-provider strategies: Relying on a single model or provider can be risky. Network issues, API outages, or pricing changes can all affect uptime. Many robust systems are built with feature flags and routing layers that allow calls to be dynamically shifted between providers depending on availability.

Debugging Hallucinations

Traditional debugging assumes you can trace through your code to understand what went wrong. LLM debugging often feels like detective work.

Even when everything looks fine but LLMs sometimes say things that are just wrong. Debugging these cases is about understanding where the pipeline broke down.

Here are some of the things I tried:

• Tracing the full input stack: The first step is always to capture the full input context prompt, system input, and user input. Without this, debugging is just guesswork. Going through the invocation history in sequence has helped me a lot in debugging the issue.

• Investigating retrieval mismatches: Sometimes the model hallucinates simply because it wasn’t shown the right context. This can happen due to wrong queries, wrong retrieval of information. Debugging retrieval involves logging and validating whether the correct document was even eligible to be retrieved.

• Session Replay tooling: Unlike traditional bugs, LLM failures are often non-deterministic. The same input can produce different outputs, making reproduction and fixing extremely difficult. Having tooling that can replay a request with different prompt versions or see the user’s exact request, the model choices of the system, or context configurations is invaluable.

Bringing It All Together

Building AI systems that go beyond prototypes means thinking holistically from how prompts are designed to how systems are deployed, evaluated, and maintained in production. Working with LLMs is non-deterministic, and one of the biggest wins is getting comfortable with it.

• System Design Considerations: Just like with microservices, designing LLM systems involves making trade-offs between latency, context size, caching, and API dependencies. For example, whether to precompute embeddings or compute on the fly affects the response time.

• Being cost-aware: Cost is part of system design. Token-heavy prompts, unnecessarily large context windows, or routing all requests to the most expensive model can tank efficiency and need to be thought of before making the system live.

• Observability and Debuggability: Every user interaction with an LLM should be traceable to a specific prompt version, context snapshot, model, and input. Replay tooling, logs, debug hallucinations, or sudden behavior changes. While this sounds efficient, the observability stack for AI systems is still evolving.

• Feedback Loops: Qualitative user feedback, whether explicit thumbs down or implicit drop-offs, should inform prompt tweaks, retrieval tuning, and even product decisions.

• Canary Deployments: New prompts or model configurations are first rolled out to a small percentage of users. This helps detect regressions or unintended behaviors before full release.

Building LLM systems that go beyond prototypes requires the same engineering discipline as any production system, just adapted for probabilistic outputs. The difference between a clever demo and a reliable product isn't the underlying model; it's the engineering practices around it. Your 1% failure rate might be your users most important use cases. AI engineering is still a young discipline, but the principles are becoming clear: building AI systems that users trust and depend on, not just impressive demos that break in production. The tools will keep evolving, but the engineering mindset remains constant: build systems that work reliably, cost-effectively, and transparently, even when the underlying technology is inherently uncertain.

Picking the right trade-off is everything

Most decisions will not have a right answer; rather, they will consist of a set of good tradeoffs.

Some days you will be juggling between increasing a metric vs keeping the website up. Some days debating about tech debt vs moving fast. And some days on increasing reliability or building a new feature. In conversations like this is where you grow.

As engineers, we often find ourselves craving technical correctness. But in reality, it's all about judgement, knowing which corners you can and cannot cut.

Trade-offs are not a sign of failure. They are a core part of the work. The earlier you get comfortable making them, and owning them, the more effective you become.

Code is the easy part

When you are first starting out, coding can feel like the entire job. Commits and pull requests are the most visible features.

But, over time, I realised that the real challenges were elsewhere: talking to a frustrated user, owning a broken system during an outage, and unblocking a teammate who is waiting for you.

Writing code is simple (not always); the logic either works or doesn't. But engineering is far more than that. It's about how you deal with pressure, responsibility, and communication.

You don’t need to have an answer always

There is sometimes a hidden pressure we put on ourselves to always know what to do. To have a clean solution, a clear plan, and a quick fix.

However, sometimes the best solution is to say, "I'm not sure yet, but I will figure it out."

It's easy to think that expertise means having all the answers. However, in practice, it often involves asking better questions, remaining calm in a moment of uncertainty, asking help when needed, and trusting that you will learn your way through it.

Consistency is all that matters. Showing up daily, even when things are unclear or fuzzy is sometimes all that is needed.

The emotional side is real

No one tells you how emotional this journey can be. The uncertainty after a failed project. The happiness in a clean release. The joy of seeing someone use what you built. The challenge when things break. The gratitude when a teammate steps in to help. The little wins that mean everything to you.

What surprised me the most was how deeply personal this work is. Sure, it is code. It’s systems. But it’s also people. It’s identity. It’s the constant, quiet process of figuring out what kind of engineer and person you want to be.

It’s easy to think engineering is purely rational. But the truth is, most of us care a lot and that’s what makes the work meaningful.

Growth isn’t always obvious

As engineers, we are obsessed with numbers. Doing X increases it by Y. However, growth will not always feel like this.

Growth isn't always obvious. It occurs when no one is watching. It appears later, when you realise you handled a situation better than you would have a year ago. When you believe you handled a difficult situation better, approached a problem differently, or collaborated more effectively.

Not every day will be a drum roll, but if you show up and strive to be 1% better, that is progress.

Writing is a superpower

For years, I have made it a habit to write down my thoughts whenever I am faced with a difficult decision or problem. Not for anyone else, only for me.

At first, it was just a way to clear my head. But over time, I realized it was more than that. Writing helped me notice patterns. It helped me go back to the old decisions with a fresh perspective. Sometimes it even helped me avoid making the same mistake twice.

We often think clarity comes from more data. But clarity also comes from thinking slowly.

Product building is hard (And that’s the fun part)

No matter how good the architecture is or how clean the codebase looks, if you are building a product, you are stepping into a world of ambiguity. Requirements will change as the product matures. Priorities will shift as demand changes. Users will not always use it the way you expect.

Product building is an iterative process. You try something, ship it, observe, learn, collect feedback and iterate. Sometimes you're figuring out the right problem rather than the right solution.

What’s hard and also incredibly rewarding is becoming comfortable with the chaos. Learning to ask why before jumping to how. Developing the instinct to question: Is this the right thing to build? before thinking How fast can I ship this?

Over time, I have realized that engineering is not just a function for product, it is product. Every abstraction, every error message, every retry mechanism is part of the user experience.

At the end of the day, it’s not about being perfect. It’s about being patient, constantly learning, staying curious, and enjoying the work along the way.

What are something’s you have learned in your own journey?

AI applications have gone from prototypes to core elements of software engineering in the past few years, such as recommendations, identifying fraud, help with writing, and a lot more. However, with great power comes great unpredictability. Reliability is no longer just a feature; it is the foundation. Users may be impressed by intelligence, but they return for consistency. If your AI system fails unexpectedly, even once, users' trust in it will erode faster than it was built. And once that trust is lost, it's nearly impossible to regain.

What makes issues even harder is that AI does not always fail in obvious ways. There may be no error message or crash, just an off-putting response, a recommendation that falls short, or a moment when a user feels unseen. These are the times when reliability becomes obvious. They determine whether your product succeeds or fails.

So, reliability is no longer just a backend concern; it now extends to the way we build products. It doesn't always appear on launch day. However, it is what encourages a user to return tomorrow. And for the AI systems we're developing, I can't think of a more important metric.

What Makes Reliability in AI Different

Traditional software is largely deterministic: the same input produces the same result every time. AI is probabilistic. Even with identical inputs, we may receive different results depending on model randomness, inference parameters, hyperparameters, or changes in the upstream data. This makes bugs harder to reproduce and problems more difficult to detect.

Furthermore, AI systems degrade over time due to data changes rather than code changes. Models trained on yesterday's data may not be accurate tomorrow. This is known as data drift or model decay, and it is one of the most significant reliability risks in production machine learning.

LLM reliability extends beyond uptime and performance. It is about the predictability of behavior. It's about how the system handles uncertainty.

Common Ways LLM Fails

1. Hallucinated responses: The AI makes things up but expresses them with such confidence that users don’t realize anything went wrong.

2. Inconsistent behavior: Asking the same question twice or multiple users going through the same flow generates different results, creating confusion.

3. Lack of graceful handling for edge cases: The system doesn’t know how to say “I don’t know” and instead offers low-quality or misleading responses.

4. Data Drift: As the training data becomes outdated, models begin to fail in new, unanticipated ways.

Ways that have helped me improve Reliability

• Don’t just test functionality but also behavior: Traditional test cases validate “Does this work?” But with LLMs, I’ve learned to ask, “How will this feel?” “What if this output is wrong but sounds right?”

• Observe real users and not just metrics: Quantitative metrics are helpful, but they often miss the story, and hence qualitative metrics play an equally important role. Watching and reading the chat histories of users’ interactions with our LLM features has been far more enlightening. You start to notice patterns, confusion points, or expectations that weren’t met.

• Build with confidence in mind: For every LLM response, we ask, “How confident are we in this output?” If the answer is “not very,” then the UI should reflect that, and we iterate further on how to increase the confidence in the LLM response.

• Avoid frustrating loops: Users should never feel trapped inside an LLM-generated flow. There should always be a way to correct, back out, or ask for help. Even if there is a failure in response, there should be an action item/next step for the user to break the loop.

• Remember the worst cases: One thing that’s helped me is mentally walking through the “worst case” for any new AI feature. What happens if the model returns garbage? What does the user see if our inference endpoint is down?

• Make reliability part of the definition of done: We often think a feature is complete when it works — but it’s only truly done when it works reliably. That includes graceful failure, observability, and user clarity. I now treat those as first-class aspects of every AI feature, not just add-ons.

Reliability is ultimately about Respect. Respect for the user’s time. Respect for their mental model. And respect for the trust they place in what we build.

The Role of Humans in the Loop

No matter how advanced the model, it needs human context or a human in the loop to stay grounded. That’s why human-in-the-loop (HITL) systems aren’t just a safety net; they’re a reliability engine.

Human-in-the-loop ensures:

• Critical errors are detected before they reach the user.

• Ambiguities are resolved through human judgment.

• User feedback is fed back into the system to help it improve over time.

Even lightweight HITL systems, such as feedback buttons, add a layer of adaptability that code cannot provide.

We often talk about AI’s capabilities: its speed, scale, or intelligence. But what truly matters in the long run is its Reliability. That’s what users come back to. And that’s what determines whether an AI system quietly fades away or becomes a trusted part of a user’s journey.

Reliability doesn’t announce itself. It shows up in the quiet moments; when things just work, even when the stakes are high or the path is unclear. Reliability isn’t just a technical challenge. It’s a design principle. A product value. A reflection of how much we care.

And in the world of AI, it just might be the thing that makes or breaks everything else.

Empathy in Engineering begins with two simple things

1. The first and most important step is to understand who your users are, rather than just what features they require or how we will build them. When you can connect with the end user or put yourself in their shoes, everything becomes easier and decisions are made objectively.

2. The next step is to understand the entire context in which any feature, no matter how small, exists. Once you understand the why, it becomes easier to figure out how.

Why does empathy matter in engineering?

Empathy is often thought of as a soft skill, secondary to the hard logic of engineering. But, in reality, empathy combined with an engineer's technical depth and expertise can perform magic! It improves our problem-solving skills, reduces unnecessary complexity, fosters collaboration, and makes our work more fulfilling.

Problem Solving

When we truly understand our users' frustrations, workflows, and mental models, we can stop solving theoretical problems and start tackling real ones. Without empathy, it is simple to design for an abstract "user" who only exists in the theory. It enables us to create solutions that are practical in nature. This shift not only improves the impact of our work, but it also saves time and effort on features that no one needs.

Reducing Technical Debt

Engineering with empathy often results in cleaner, more maintainable systems—even though it might seem counterintuitive at first. When we truly understand how users interact with our software over time, we design for adaptability. We resist the temptation to over-engineer speculative features or prematurely optimise for scale that may never come. At the same time, we ensure that critical user flows are built with care, avoiding under-engineering that could lead to friction. The result? A system that evolves gracefully instead of accumulating unnecessary technical debt.

My most recent example was when I was integrating a payment gateway. We started with Razorpay, but we knew we wanted to support global payments in the near future. Instead of hardcoding logic specific to a single provider, we designed the entire payment system around state machines and reusable methods. This approach was not only intended to make future integrations easier for the business; it also ensured that any developer working on the system (including my future self) would not be burdened with unnecessary complexity. That's when I realized that empathy-driven engineering is about making life easier for everyone who interacts with the system, not just the users.

Finding Meaning

I believe that knowing who we are developing for and why we are doing what we are doing is key to finding fulfillment in our work. It is one thing to say "I write code," but completely different to say "I solve this problem for people by writing code." It's really exciting to know that what I create is actually helping people solve an important problem for them. It reminds us that behind every API call, performance optimization, and bug fix, there is a user trying to accomplish something important to them. That perspective provides more satisfaction than simply shipping features; it transforms our work into something meaningful.

How to Engineer with Empathy

After a few years of learning, here are practices that have helped me engineer with empathy:

Diving into User Feedback

This is something I strongly believe in: digging deep into user feedback. Don't delegate understanding of your users to the product team. Make it your business to understand who uses your software, including the smallest feature you've created, and how they feel about it. This includes actively engaging with feedback channels, reading support tickets personally, and looking beyond feature requests to detect emotional cues in user feedback.

One of my daily practices is to review user feedback to better understand how people use the product. It has become a consistent source of motivation. We track the Developer Love Index (DLI) for one of the products we're developing in HackerRank's Developer Community team on a regular basis—a measure of how much users value the product and whether they'd pay for it. It provides an instant energy boost. When the DLI rises, it indicates that we are on the right track. When it drops, it's a clear signal to investigate further and figure out what's wrong.

This viewpoint is powerful: even if only one user benefits from what we're creating, the right speed and direction can spread the impact to countless others facing similar challenges.

Get rid of preconceived notions

When you've spent weeks developing and testing a feature, it's natural to believe you know exactly how users will interact with it. After all, you've become the most experienced user. However, real users frequently navigate software in unexpected ways; what appears intuitive to you may be confusing or frustrating to someone else.

One of the most effective ways to combat this bias is to observe how actual users interact with the system. Sessions replays have changed the game for me. Actual user interactions can reveal pain points that specs, test cases, and even direct feedback cannot. It highlights the various applications of a feature and bases each decision on data rather than assumptions.

While the first version of any feature is always built with assumptions and hypotheses, this is where iterations come in handy.

Embrace "What If" Thinking

This involves identifying potential frustrations before they become major problems. A good habit is to pause and ask the right questions, for example:

• What if the bot gets stuck while generating a response?

• What happens if a payment fails? Will they understand what to do next?

Thinking about failure cases and edge scenarios helps to create a more seamless user experience. Proactively handling errors with clear, human-friendly messages can mean the difference between a frustrated and a delighted user.

One of the things I recently did for one of the AI features we are developing was to make the error message more human-friendly so that, in the worst-case scenario, the user sees the message; this may reduce the frustration of an already disappointed user.

Remember the problem you are trying to solve

One of the most important guiding principles in any discussion—whether it's naming a variable, structuring code, or deciding on the next feature—is to keep the core problem in mind. It's easy to get lost in discussions about implementation details, abstract optimizations, or even personal preferences.

However, at the end of the day, every decision should relate back to the fundamental question.

"What is the actual problem we are solving?"

Always begin with the "why" before diving into the "how" of implementing something.

Once we start understanding these fundamentals, the fuzziness starts becoming clear. For example, we need to worry about the variable name because it will become easy for you and other developers to come back to it in the future. We are building a feature X because it solves the problem Y.

I've gradually come to believe that empathy is more than just a nice-to-have soft skill for engineers; it's an essential value added along with technical depth and competency. It's what separates code that functions from code that serves, systems that run from systems that enable humans. So the next time you sit down to write code, remember: there's a human on the other side, and your empathy might be as important as your algorithms.

What practices have helped you engineer with empathy? I'd love to hear your experiences.

That’s where tools like brag documents, priority lists, and got-done lists come in. They’re not just about staying organized—they’re about giving yourself credit where it’s due, tracking your growth, and building a system that works with your busy life, not against it.

In this blog, I share some of the things that I have been trying to keep things organized. They are simple, flexible methods that have helped me stay on top of things and feel a little more in control.

Brag Document

What is a brag document?

It’s a simple list where you record your achievements, the impact you’ve made, and what you’re working toward. Think of it as your archive of wins, big and small, and the things that you need to improve at.

Why do we need a brag document?

1. Avoiding Recency Bias

   It is a fact that we tend to remember recent events better than events that occurred in the past, giving the recent event more significance. If someone asks, “What did you do six months ago?” we’d probably need a moment to recall. Without a brag document, we often have to dig through pull requests, tickets, slack conversations, launch emails, and design documents. I’ve been there—rediscovering small and sometimes major wins that I’d completely forgotten about.

2. Identifying patterns and preferences

   A brag document is a great tool not only for performance reviews but also for self-reflection. By reviewing it, you can spot patterns in the work you enjoy, the skills you’ve developed, areas where you’ve had the most impact, and areas where there is a scope for improvement. It’s an opportunity to understand your preferences and chart a more intentional career path.

On days when imposter syndrome hits hard, going back to the brag document can be like having a good cup of ice cream on a bad day!

When I first wrote a brag document I was kind of nervous about sharing it with anyone. It felt weird to be like I wrote a long document listing everything I did this year so I kept the name of the document as “Review Document”.

How do we start writing a brag document?

• Start by listing down everything
  Include everything—big wins, small contributions, and fuzzy work that’s hard to quantify like refactoring code, reducing tech debt, fixing critical bugs, and improving the on-call process.

• Explaining the big picture
  Organize your work by projects or themes. For example, if you have been working on multiple projects in parallel, you can add headings like “Project 1”, and “Project 2” and then list everything done under each project. This not only showcases the breadth of work but also highlights how your efforts align with broader team or organizational goals.

This is a simple template that I have been using and can help anyone get started.

How often to update the brag document?

Updating the document on a daily/weekly or sometimes even every month might feel like a lot. A trick that I have been following is to take notes that don't have to be flawlessly formatted; a few bullet points can do the trick and serve as a recall of the entire scenario when you go back to it. I record everything that comes to me and once every Quarter I update the document.

You might have to experiment with a few cadences to see which one suits you the best. Usually having one document for the year helps.

One important thing to keep in mind is to not pressure yourself with writing the perfect document in the first go. Keep it simple. It doesn’t need to be perfect or fancy—just real. You just have to explain your work the way it was done.

You can also check out https://jvns.ca/blog/brag-documents/

Kudoboard

There are going to be days when we tend to doubt ourselves. We overthink a message we sent or a conversation that happened, or we just need that quick pep talk before a big day or consolation after a not-so-good one.

That’s where our personal Kudoboard comes in—a treasure of positivity and encouragement.

It could be a screenshot of a Slack message from a teammate who appreciated your help, an email from your manager praising your work, or even a quick note from a friend saying, “I couldn’t have done this without you.” These small yet meaningful moments often fade into the background amidst the daily hustle, but collecting them in one place creates a powerful reservoir of confidence and motivation.

Your kudoboard becomes your personal cheerleader, whispering, “You’ve got this!” It’s a reminder that while setbacks happen, they don’t define you.

To start with, create a folder in your inbox, a private document, or use tools like Kudoboard or Google Docs or as simple as a Whatsapp group with just you in it.

To-Do, Priority & Got-Done List

A priority list outlines the key things you aim to achieve over the month/week, keeping you aligned with the bigger picture. Alongside this, a daily to-do list helps in breaking those objectives into manageable tasks for the day, ensuring steady progress. I usually use a notebook to maintain these lists but you can also experiment with Notion or tools like Trello. When you have a clear list of what needs to be done, your brain automatically follows and gets into a problem-solving state.

Some days, despite careful planning, unexpected tasks crop up, leaving you feeling like you’re constantly firefighting without making real progress. That’s when the got-done list becomes valuable.

To conclude, these documents aren’t about rigid systems—they’re about finding what works for you and following the rhythm to stay organized. What are some of the things that you have tried?

This love for shared reading experiences introduced me to two wonderful communities: CubbonReads and BombayBookies.

The Beginning of LabsReads

The idea for LabsReads emerged from a simple tradition between Hari and me. We often exchanged articles or blogs and discussed them informally. Over time, this casual practice sparked a thought: what if we turned this into a regular, organized activity with the team?

The timing couldn’t have been better. Around the same week, I attended a session of CubbonReads for the first time, and I loved how the community gathered and read together. Inspired, I decided to bring a similar concept, naming it LabsReads since the name of our team was Labs.

I was both nervous and excited. Starting something new always comes with a mix of excitement and uncertainty—like opening the first page of a new book. Would people join? Would they enjoy it? What if the article I chose is a stupid one? These questions ran through my mind, but the thought of creating a space where people could come together to read and share ideas kept me motivated. It was a big step into the unknown, but one that felt interesting.

What is LabsReads?

LabsReads is more than just a reading session; it’s a space for learning and collaboration. The format is simple:

• 15 minutes of reading: We read an article on any topic related to software engineering.

• 15-30 minutes of discussion: We share perspectives, insights, and questions about the article we read.

Our sessions happen every other Wednesday. Sometimes, we gather amidst artificial trees in the HackerRank Bangalore office; other times, we meet outdoors, surrounded by lakes or rivers. The goal remains constant: to create an environment where reading ignites conversations.

Lessons Learned

Over the past year of running LabsReads, here are some of the things that I have learned and tried

Deciding what we read

Choosing the right article is key to keeping the discussion engaging. I maintain a Google Doc of links that I come across, categorizing them by themes like code quality, testing, tech debt, or product building. I have realized that rotating themes keep discussions fresh, and relevant to engineers, product managers, and designers alike.

One of the important factors that decide the article we will be reading is also something that will directly help us in our day-to-day work while building our product. For example, when the team had to plan on what Technical Debt we would be picking up for the year, we read about the different ways in which technical debt could be solved and how we needed to attach a business value to it. When we were deciding on what Testing strategy that we would follow, we read about how QA was something everyone had to do.

Facilitating the Flow

Before the first session, I was worried about how the discussion would follow. Preparing a list of questions or discussion prompts helped structure the conversation and ensured active participation. This practice continues to be a cornerstone of how the sessions are conducted. Before every session now, I write a maximum of 5 questions around which we could have a discussion and then the room is open for any free discussion.

Keeping it exciting

To add a touch of fun, we introduced LabsReads bookmarks. Whoever attends the session gets a bookmark—a small touch to keep everyone attending it engaged.

Finding the Right Cadence

When LabsReads began, we held sessions weekly, eager to maintain momentum. However, we quickly learned that weekly sessions were a sprint we couldn’t sustain while balancing other work priorities—so we paced ourselves to biweekly to ensure we were not risking the freshness of each session.

A biweekly schedule has proven to be a better fit. Our goal now is simple: to host a minimum of 12 sessions each year (approx 1/month).

What makes LabsReads isn’t just the articles we read but the conversations that follow. Whether it’s someone sharing a story from their projects or a lively debate about an unexpected perspective.

Here’s to many more sessions of reading and sharing together.

Links we read so far

• 10 hard-to-swallow truths they won't tell you about software engineer job

• Tech Debt and the Pragmatic Middle Ground

• How to Make Good Code Reviews Better

• Writing Code? STOP! Think First...

• Clever code is probably the worst code you could write

• You are never taught how to build quality software

• What happens when the pager goes off?

• Principle of Chaos Engineering

• Random Thoughts 15 years into Software Engineering

• Using your own product is a superpower

• Paying down tech debt

• Rethinking the startup MVP: Building a competitive product

It was business as usual at the HackerRank office. As the HackerRank Community team gathered to plan the next quarter, one of the topics that came up was the Tech debt we will address this quarter.

To give some context, every quarter in our team we pick two Tech debt items – one on each Backend & Frontend alongside all the product items we have to build.

After some discussion, we finally decided to pick MySQL Upgrade to 8 as one of the items. It was a long due.

Motivation for Upgrade

Why did we need to upgrade to MySQL 8.0? We had been running AWS Relational Database Service (RDS) with extended support for MySQL 5.7, as it had reached its end of life. However, this extended support came at a steep cost—an additional $3,000 per month. It was time to upgrade to a version of MySQL that would provide the latest security updates, bug fixes, and performance enhancements but more importantly, time to save that money.

And so, the roller coaster began: upgrading the HackerRank Community Database from MySQL 5.7 to 8.

Honestly, I was the most excited about this upgrade. Reading the High-Performance MySQL book, I was eager to dive into the core infrastructure changes. It was finally time for some database fun!

A sneak peek into the HackerRank’s Infrastructure

Now, our AWS RDS setup looks like this:

• Master, where all the reads and writes are directed to

• Read Replica, where all the long-running reads are directed to

• Analytics Replica, which is used for our internal analytics

Now, as a next step, I had to devise a plan for the upgrade and what all prerequisites would be needed to be performed to get the upgrade done.

Approach to the Upgrade

As we delved deeper into upgrading our RDS instance, we explored using AWS's Blue/Green deployment. 

Amazon RDS Blue/Green deployment solution is simple, and most of the replication/sync effort between existing MySQL 5.7 (blue deployment) and MySQL 8 (green deployment) is managed by AWS using logical replication. You can make changes to the RDS DB instances in the green environment without affecting production workloads like upgrading the DB engine version or changing database parameters in the staging environment. 

When ready, you can switch over the environments to promote the green environment as the new production environment. The switchover typically takes under a minute with no data loss and no need for application changes.

One key prerequisite for this approach was ensuring that both the master and replicas had identical indexes, with no separate read indexes on the replicas. To verify this, we took an SQL dump of all three databases and compared the indexes. Thankfully, all three had matching indexes, so we decided to proceed with the Blue/Green deployment. One problem solved!

Preparing for the Upgrade

With the upgrade method confirmed to be Blue/Green (rather than the traditional method of creating a duplicate RDS and switching traffic), we outlined the necessary steps:

After running the upgrade-checker tool, we encountered one error due to an intermediate temporary table. This was resolved by dropping the table. The error message was:

[ERROR] Schema inconsistencies resulting from file removal or corruption.This issue likely stemmed from an ALTER command being interrupted prematurely.

Reference: https://serverfault.com/questions/1002704/cant-upgrade-rds-instance-from-mysql-5-7-x-to-8-0-x

Once this error was fixed, we successfully set up the Blue/Green environment in non-production.

We conducted thorough regression testing, which showed that our application was compatible with MySQL 8 and that no major code changes were necessary.

After completing the tests, we upgraded all non-production environments to MySQL 8. The next step was to implement Blue/Green deployment in production.

In the production environment, we resolved the same error and got MySQL 8 running in the green environment. 

However, since production had two replicas and also 4 TB worth of data, we experienced a significant replication lag for around 2 days. We tried multiple approaches like changing the parameter value like slave_parallel_workers to a higher value and recreating the Blue/Green setup but that did not seem to get the lag down. One of the other things we tried was, we initiated a mysqldump and increased the IOPS (Input/Output Operations Per Second).

Running a mysqldump operation on your read replica allows Amazon RDS to prioritize and download all of the backed-up table data from Amazon Simple Storage Service (Amazon S3) thus helping in getting the lag down.

This slowly brought the replication lag down to 0, just 15 hours before the scheduled time for the production maintenance window.

 Once the lag was back to 0, we started to change the parameter group from default MySQL 8 to the desired ones for our database and upgraded the instance to the graviton. As soon as we did this, we encountered an error

REPLICATION_ERROR/Replica failed to initialize applier metadata structure from the repository.

This was caused due to an unexpected server halt that was triggered by Multi-AZ failover on the green instance and failover was expected. While we were trying to resolve the above issue, there came another twist…

The Twist

As Murphy’s law states, anything that can go wrong will go wrong. Here's the twist: we encountered a warning in the logs that seemed innocuous at first, but in reality, it was a fatal error. 

2024-07-20T04:24:44.767492Z 13 [Warning] [MY-011825] [InnoDB] 

Cannot add field `codechecker_time` in table `hackerrank`.`h_submission_shards_co_1_ch_520` because after adding it, the row size is 8827, greater than the maximum allowed size (8126) for a record on the index leaf page.

This issue arose because the row_format for some tables was set to Compact, while MySQL 8 uses Dynamic as the default format.

Upon deeper investigation, revealed a serious issue that could lead to potential data corruption. This would render all tables with the earlier error inaccessible. Surprisingly, none of this was flagged in the PrePatchCompatibility logs during the upgrade process.

What was the Issue?

Before MySQL 5.7, the default row_format was Compact. Many of our tables had been created when we were using MySQL 5.6. When we upgraded from 5.6 to 5.7 back in 2018, all newly created tables adopted the Dynamic row format, which is also the default in MySQL 8 & 5.7. However, we never updated row_format of the older tables during the 5.6 to 5.7 upgrade. Now, during the upgrade to MySQL 8, these old tables triggered errors due to the large amount of data they contained.

This issue only surfaced in production because the database there holds significantly more data than in non-production environments. The larger data volume caused the database to start throwing errors.

References:

• MySQL Bug #99791

• Understanding & Fixing “Index Column Size Too Large”

• The row size is too large

The only way to resolve this issue was to update the row_format to Dynamic MySQL 5.7 before performing the upgrade. And so, like every great movie, for our database upgrade one rollout plan wasn’t enough— here comes Rollout 2.0, bigger and full of twists!

Rollout 2.0

The next step was to verify whether changing the row_format would be a blocking call. These are the set of tasks that we started with,

We tested the row_format changes in the lower environment, and it was confirmed that it was a non-blocking change. This means it allows DML/DDL operations on the table while it's being modified. MySQL documentation also confirmed that this change is non-blocking, which was a relief.

The next concern was whether this change on the master would be replicated in the read replicas.

To verify this, we created a temporary database from the existing master snapshot with one replica. There were 2 ways to modify the row_format

1. OPTIMIZE TABLE <table_name>

2. ALTER TABLE <table_name> ROW_FORMAT = DYNAMIC;

This is what the OPTIMIZE TABLE does; it essentially rebuilds the table for InnoDB. It performs a process that involves creating a temporary table with the optimized structure and data which means it will also set the row_format to default value, copying the data into it, and then replacing the original table with the optimized one. This process results in defragmentation and compaction, which can reclaim disk space and improve performance. 

We decided to go ahead with the OPTIMIZE TABLE command and not do ALTER because of the following reasons:

1. Performing the ALTER statement explicitly sets a value of row_format=DYNAMIC under create_options for the table, even though it matches the current default. Setting an explicit row format on the table is something we may need to be aware of if we ever choose to change row formats in the future.

2. OPTIMIZE TABLE command also changes the row format of the table to the current default, but does so without setting create_options;

We modified the row_format to Dynamic using the OPTIMIZE TABLE command and confirmed that the changes were successfully replicated to the read replica. This verification was crucial because we couldn't run any ALTER commands directly on the replica, as it has read_only=1.

After these tests, we realized the only way to modify the row_format to Dynamic in production was by running the OPTIMIZE TABLE command directly on the production master.

Yet again, there was an unexpected turn. The OPTIMIZE TABLE command works by creating a duplicate table, reading the current table's online DML, and then switching to the new table. For this to succeed, the database needs enough free storage equal to the size of the table being optimized. Once the optimize is done, the storage is recovered. One of our tables was a massive 824 GB, but our AWS RDS had only limited free storage. To proceed with the OPTIMIZE command, we first had to clean up this table to free up the necessary storage. And this is how one more task got added to our checklist,

Deleting & Optimizing

To minimize the impact of running the OPTIMIZE command on our database, we first tested it on a small production table and noticed increased replica lag during execution. To mitigate this, we temporarily redirected all read traffic to the overprovisioned master server, efficiently handling both reads and writes.

Our largest table was the 824 GB sessions table managed by Devise in Ruby on Rails, containing data since 2018. Since we only needed data from the past year, we wrote a deletion script to remove old records in batches of 10,000 with 1-second pauses between batches. Over a week, this reduced the table size to 124 GB, allowing us to run OPTIMIZE smoothly.

We also cleaned up unused archival tables, freeing 1.5 TB of storage. We proceeded to run OPTIMIZE on all tables with a row_format of Compact, starting with the smaller ones and monitoring CPU usage to keep it below 80%.

However, we faced duplicate entry errors due to unique composite indexes on tables with high concurrent writes during the OPTIMIZE process. This occurred because OPTIMIZE rebuilds the table, and concurrent writes could cause duplicates in the new table. This is a bug in MySQL itself.

Reference: MySQL Bug #98600.

| Table                          | Op       | Msg_type | Msg_text                                                                                             |
+--------------------------------+----------+----------+------------------------------------------------------------------------------------------------------+
| hackerrank.h_hacker_broadcasts | optimize | note     | Table does not support optimize, doing recreate + analyze instead                                    |
| hackerrank.h_hacker_broadcasts | optimize | error    | Duplicate entry '26450629-2629173' for key 'index_h_hacker_broadcasts_on_hacker_id_and_broadcast_id' |
| hackerrank.h_hacker_broadcasts | optimize | status   | Operation failed

To resolve this, we monitored traffic patterns to find low-traffic windows for running OPTIMIZE, which worked for some tables. For one problematic table, we temporarily dropped the unique composite index and implemented stricter code checks to prevent duplicates, allowing us to complete OPTIMIZE and re-add the index afterward.

For a particularly write-heavy table, we introduced a Redis flag with random sleep intervals between reads to prevent duplicate entries, enabling us to re-add the unique index successfully. In hindsight, implementing this Redis-based approach earlier could have avoided the need to drop indexes.

Ultimately, we updated all tables to have a row_format of DYNAMIC, solving a major issue. However, due to the extensive changes made to the master, the replicas were now experiencing significant lag.

Testing the Blue/Green Setup

The next step was setting up a Blue/Green deployment in production, updating the Green environment's parameter group, and upgrading to Graviton. We began this process but also wanted to ensure we had clear metrics on how long the switchover would take in production. While AWS suggests a switchover time of 30 seconds to 2 minutes, we decided to test it cautiously due to the size of our RDS.

We created a temporary database with production changes, added a replica, and set up Blue/Green to benchmark the switchover time. During testing, we simulated the production load on the database and performed the switchover.

The switchover took 1 minute, with 2-3 seconds of downtime, giving us confidence that the upgrade would have minimal user impact.

Production Blue/Green Setup

Since we had been running the OPTIMIZE command in production, the read replica lag had become significantly high. To avoid any potential issues during the real switchover, we wanted to ensure that the lag was reduced to zero. To address this, we increased the IOPS on the active replica, which helped bring the lag down gradually.

Additionally, the Blue/Green setup on the production master was also experiencing lag. Based on our previous experience, we decided to disable Multi-AZ on the Green setup, start a mysqldump, and increase the IOPS. As a result, we observed the lag steadily decreasing.

One other warning we have been seeing in the logs was 

DB Instance has a large number of tables and has the parameter innodb_file_per_table set to 1, which can increase database recovery time significantly. Consider setting this parameter to 0 in the parameter group associated with this DB instance to minimize database downtime during reboots and failovers.

We decided to keep it as 1 itself because It being turned on helped us reclaim storage space in AWS when we were running the OPTIMIZE command since each of the tables had its idb files. If it is off and we do the optimize, it cleans up the space but we can't see the storage increase on the AWS level it will be maintained in the shared tablespace within MySQL itself, and MySQL will utilize it as new records are added.

The Endgame

Our upgrade was scheduled for August 31, 2024, at 10:30 AM IST. By this point, we were fully prepared, having thoroughly tested the application and the Blue/Green switchover.

The final switchover took less than a minute to complete with zero downtime. Even our monitoring tools, such as Uptime, didn’t detect any issues. Most importantly, no users or candidates were affected by the process.

And this is how the MySQL Upgrade for HackerRank Community finally happened.

Repost from https://engineering.hackerrank.com/behind-the-scenes-at-hackerranks-upgrade-to-mysql-8-0/

The Value of Code Reviews

Engineers gain significant advantages from thorough code reviews:

• Learning best practices of a language

• Catching bugs early

• Understanding tradeoffs among different solutions to the same problem

For example, I learned how to write effective RSpec tests and Ruby best practices after receiving suggestions on improving my code. I also started adding detailed descriptions to PRs when I saw how an old description in a PR helped me solve an issue.

In today's fast-changing world, where Large Language Models (LLMs) are used to generate code and tools like Copilot help us write boilerplate code, code reviews are even more important. While AI can be a powerful tool, it is not flawless. AI can sometimes hallucinate, producing code that appears to be correct but contains subtle or significant errors that do not fully meet all of the criteria for our use case. In such cases, it is especially important to review one's code.

Key Aspects of Effective Code Reviews

1. Understanding the Big Picture When reviewing code, our first instinct should not be to immediately suggest removing changes, but rather to ensure that we understand the context and purpose of the existing code. This principle, inspired by Chesterton's Fence, emphasizes the importance of understanding why a piece of code exists before changing or removing it. For example, in some cases, removing a piece of code after receiving review comments breaks the functionality. To avoid situations like this, one solution is to include comments or details about the unusual changes in the PR description.

   

2. Knowledge Sharing When we see multiple similar conversations/comments about coding style across different PRs, it indicates the need for proper linters and standardization. These linters and automated tools can handle a wide range of code quality issues, while human reviewers can provide insights into best practices, architectural decisions, and domain knowledge. This improves the learning experience and encourages a culture of continuous improvement.

3. Having a Review Buddy It's a good idea to have a review buddy with whom you can discuss design and architecture from the beginning of a code-related feature. This helps to avoid significant rewrites during the review stage and ensures that design and architectural considerations are aligned from the start. A review buddy can provide early feedback and suggestions, making the development process smoother and more efficient, which in turn speeds up the code review process.

4. Understand Testing & Deployment Plans As important as code reviews are for reviewing a piece of code, I believe it is also valuable to understand and review the feature's testing and deployment plans. One of the practices we've implemented in our team is to write a Minimal Effective Dose (MED) Testing Plan for each of our code changes, in which the code author mentions the critical pieces of the code that can be affected by the code change as well as test plans, which are then verified by the reviewer to make sure no edge cases were missed.

5. Suggest, Don’t Instruct Use a constructive and supportive tone when adding review comments as suggestions rather than instructions. Instead of instructing with phrases like "You should stop doing," use more collaborative language such as "I think it would be better to." Avoid using blameful or aggravating language.

6. Maintain a Priority List I usually keep a list of PRs and the expected date when the feature should go live. This allows me to organize the reviews by priority. Next, when reviewing the code, determine which parts are most critical and can cause production issues. Those hot paths should be reviewed first. This approach makes sure the most critical problems are addressed first.

7. Avoid Context Switch: Try to review code in a single sitting without unnecessary interruptions. Context switching can lead to oversight and reduce the quality of your review. Dedicate specific time slots for code reviews to maintain focus and provide thorough feedback.

Effective code reviews are a must for engineers. As technology advances with Copilot and LLMs, the engineer's role in ensuring the code is bug-free and meets standards through rigorous review practices becomes increasingly important.

Code reviews are more than just reviewing code; they also involve sharing knowledge, reviewing architecture design, and maintaining clear testing and deployment plans, all of which are beneficial to the development process.

Since the time I started working as an Engineer at HackerRank, I have had the chance to work on multiple 0-1 products, some of which we stopped due to lack of Product Market Fit (PMF) and some we continue iterating on.

Share The Engineer's Notebook

Here’s a glimpse into the life of a developer in a 0-1 product environment.

1. Understanding user feedback and iterating

The heart of any successful product is its users. When building from scratch, understanding the user feedback directly can help in interpreting what the users truly need. One of the habits that I have developed is to regularly go through the support tickets and the user feedback via in-product channels to get a better sense of how the product needs to be tweaked.

As engineers, this loop of user feedback and iteration often helps in refining vague ideas into features that can help solve problems with clarity and purpose.

2. Balancing Quality with Velocity

There’s always a tug-of-war between perfecting a feature and moving fast to release new ones. Finding that balance—knowing when to prioritize what—is crucial for momentum.

3. High Learning Curve

In the world of building new products from scratch, you learn a lot, and fast. One big plus is that you get to take care of whole features by yourself. For example, if you learn about a new way to organize your code or a new technology or design pattern, you can try it out in one of your features.

As this feature grows—by getting updates, connecting with other parts of the product, or being used by other developers—you get feedback that's more useful than reading lots of blogs about it. This real-world experience helps you learn better and keeps things interesting.

4. Becoming a Product Engineer

As you develop in this role, you begin to see beyond the code. You start to understand how each line of code you write impacts user experience and business goals. There needs to be a shift in perception—from a pure coder to a product thinker. Keeping the users in mind during development can get us very far.

5. Scalable Tech Design

In the early stages of product development, it’s important to design architectures that can scale. As a developer, you can closely with the product manager and designer to discuss the long-term vision for each feature. This collaboration helps in grasping not just what the feature needs to do now, but how it might need to evolve in the future. One of the things to keep in mind is to aim to understand the right user experience and iterate backward from there for technical feasibility.

Start with a simple technological framework—it keeps the initial development swift and manageable. However, this simplicity must be balanced with scalability. By designing systems that are easy to modify and expand, you ensure that the product can quickly adapt to new requirements or user feedback without extensive rework.

This foresight in tech design not only speeds up iteration cycles but also supports robust growth as the product scales.

6. Wearing Multiple Hats

In a small team, you’re not just coding. You might also be handling customer support, and doing data analysis for product metrics. You get to be a part of the product discussions to first hand understand the idea and fundamentals behind the feature you will be building.

Each hat you wear provides a unique perspective on what the product needs and how it’s being received, making you a more versatile and empathetic engineer.

7. Embracing the chaos and ambiguity

Not everything will be structured, and that’s okay. Embracing the chaos, finding joy in unpredictability, and adapting quickly are part of the excitement. In the world of 0-1 products, the "end state" is a moving target. This can be disorienting but also liberating, as it fosters continuous innovation and improvement.

Finally, comfort with ambiguity is perhaps your greatest ally. Not knowing the exact path or the outcome encourages creativity and resilience.

8. Dealing with Tech Debt

As Engineers, Tech Debt is usually where all of us are excited to go deep into the problem and convert the debt to wealth. One of the things that I have learned is when building a product that is still in the early stages of finding a Product Market Fit (PMF), tech debt only becomes a real problem when it starts affecting our users or slows down our velocity to iterate fast and achieve PMF.

9. High Feature Churn

In the early stages, what you build today might be scrapped tomorrow. High feature churn is common as the team pivots and adapts to the market. It’s all about staying flexible and not getting too attached to any single element of the product.

The code that you write today might become a tech debt tomorrow.

To conclude, the journey of building something new is as rewarding as the destination itself, filled with continuous learning and adaptation.

Welcome to "The Engineer's Notebook," where I share my journey. From debugging nightmares to eureka moments, join me as I delve into exciting, real engineering experiences. This space is for candid reflections, practical tips, and a bit of introspection.

Like many of you, I've endured those late nights trying to fix a stubborn bug and have felt the rush of solving a problem that seemed impossible. The satisfaction and joy of overcoming these challenges is what drives us forward. With "The Engineer's Notebook," I aim to peel back the curtain on these universal experiences.

I look forward to having you join me on this journey from inception to execution.