Routing

The Federation AIService handles request routing and failover across model providers and Services. This ensures Agents stay available even when individual backends are down.

Model routing

When an Agent sends a model request:

1. The Federation AIService looks up the model ID
2. It selects the provider with the best health and lowest latency
3. It forwards the request
4. If the provider fails, it retries with the next available provider

Service routing

When an Agent invokes an Action:

1. City resolves the Action to the Service that owns it
2. It checks the Service's health and load
3. It routes the request to the best instance
4. If the Service is unavailable, it returns an error or queues the request

Failover

• Provider health is monitored continuously
• Failed requests are retried with exponential backoff
• Circuit breakers prevent cascading failures
• Load balancing distributes traffic across healthy instances

Continue with:

• Model Provider
• Service