|
The all-in-one intelligent cloud (Sponsored)
 |
Writing code is fast, shipping it is still hard. Railway’s push-button compute, storage, and networking is built for both small and hyperscale software.
Humans and agents alike operate on one vertically integrated system on our own hardware. Get superior speed, better economics, and a lot more calm.
Sign up with the link below for free cloud credits and let your agents cook.
What is an LLM-powered application?
It’s just like any software application. But there is one major difference. Apart from normal processing, the application also sends data to a large language model (LLM). It then uses the model’s response to carry out a task. For example, a customer support chatbot might depend on an LLM to answer user queries. A document-processing system can use an LLM to extract names, dates, and invoice amounts from the uploaded documents. A coding assistant might ask an LLM to write a piece of code and integrate it into the logical flow.
On face value, we might feel that building such an application is quite simple:
The user sends a request.
The application sends a prompt to an LLM.
The LLM returns an answer.
The application displays that answer or uses the same for some other processing.
However, there are chances that any of these steps can fail. The network might be unavailable. The LLM provider might reject the user request for various reasons. The request might go through, but the model might return invalid JSON. The LLM can misunderstand the instructions and invent false information due to hallucinations. They might also take a long time to respond.
The techniques for resiliency and error handling help prepare the application to handle these situations in the best possible manner. Here’s what we will cover in the article:
What is the meaning of error handling and resiliency?
Why do LLM applications need special treatment for error handling?
How does a request flow in an LLM-powered application?
Main types of errors and failures
How to classify failures to take appropriate action?
How to retry a request correctly?
How to manage timeouts and deadlines?
How to deal with business rules in an LLM-powered application?
How to manage fallbacks and graceful degradation?
How do circuit breakers work in this setup?
Rate limiting, queues, and concurrency control
How to make LLM tool calls safe with idempotency?
How to stream the responses from the LLM?
 |
Meaning of Error Handling and Resiliency
Before we get into the details, let us understand what error handling and resiliency exactly mean.
You can think of error handling as a part of the program responsible for deciding the appropriate action when something goes wrong. For example, consider an application that calls an LLM API. If the request is successful, the program processes the response in a normal way. But if it fails, the program might have to choose from various options, such as retrying the request, showing an explanatory message, using a backup model, or recording the failure for further investigation.
Any of these approaches is much better than allowing the entire application to crash. But we also cannot treat every problem in the same manner. A real production application should be able to distinguish between different types of failures so that we can take the right action. Therefore, good error handling needs an understanding of what failed and why.
In contrast, resiliency is an application’s ability to continue doing its job even when some parts of the system are failing. We don’t need an application to operate perfectly to be resilient. Failures are fine. But they should happen in a controlled manner. This is also known as graceful degradation.
For example, consider a travel assistant that uses an LLM to build personalized itineraries for travellers. If the primary LLM it depends on is unavailable, the application can use a smaller backup model for creating the itineraries. If that model also becomes unavailable, the application can display previously created destination guides for the same locations. Of course, these answers won’t be personalized, but the user won’t just see a useless error message.
While error handling deals with an individual failure, resiliency is more concerned about the behavior of the entire system even when things are failing.
Why do LLM Applications Need Special Treatment for Error Handling?
Traditional software apps work according to well-defined rules. If we write a function that adds 5 and 7, it will always return 12. If the function returns successfully, the result is deemed valid.
With LLMs, we don’t have this luxury. This is because the output from LLMs is probabilistic. In other words, the same prompt can produce different answers every single time. Also, a successful API request to an LLM doesn’t guarantee that the response we receive is correct or even usable. For all we know, the response might be utterly gibberish. This means that an LLM API call might appear successful based on status code, but it might be a failure in logical terms. There are several things that could be wrong with the response: