Salesforce Uses Amazon SageMaker AI Cross-Region Inference Routing for Global Model Serving
Salesforce integrates Amazon SageMaker AI’s cross-Region inference router to consolidate distributed model-serving capacity, provide automatic failover, and maintain data residency compliance. Developers can now use a single invocation interface to route inference traffic across Regions and AWS accounts, while gaining detailed telemetry for each request.
What changed?
Salesforce replaced its custom model serving logic with Amazon SageMaker AI's cross-Region inference routing. Instead of directing inference requests to single endpoints, the Salesforce Internal Gateway now calls a SageMaker inference router, passing a router name and routing target name. The router determines which already-deployed endpoint across multiple AWS Regions and accounts receives the request, based on current endpoint health and capacity signals. Each response includes telemetry fields indicating which endpoint served the request and the reason for the routing decision.

Why does it matter to an everyday developer?
This update removes the need for developers to manually manage failover logic, health checks, or regional routing code for model inference. It enables developers to consolidate fragmented capacity across global Regions for improved reliability and utilization, while also supporting secure cross-account access. The router respects data residency requirements, guaranteeing that region-specific requests do not cross boundaries. Enhanced telemetry lets developers track exactly where requests were served and understand routing decisions for debugging, auditing, or compliance.
What can the developer do now?
- Route model inference requests safely across Regions and AWS accounts using a single router invocation, simplifying global application scaling.
- Rely on the router's automatic failover to handle local capacity shortages or endpoint failures, subject to preconfigured data residency requirements.
