Control Your API Calls
my-chat-router
.
‘my-chat-router’ is defined as a language
router which accesses the /chat
endpoints for supported providers. Additionally, we’ve specified this router to
have a priority
strategy which implements failover logic.
This router will first try Openai (default model: gpt-3.5-turbo
) once the router token budget has been drawn down (more info below) the router will failover to
an Azure OpenAI deployment.
Simple as that! Below we will cover each router in detail.
200
response is returned. This differs from priority as there is no token bucket to track api health across the instance.
config.yaml
.
In the below example 80% of traffic will be served to Model A with 10% sent each to Model B and Model C.