Scenarios
The Model API is the unified interface exposed by the AI gateway. Clients use large model capabilities by calling specific Model APIs. Its core working principle is: you create a Model API and associate a backend model service with it. Based on the configuration, the gateway automatically generates the corresponding access route. Client requests enter the gateway by matching this route and are then forwarded by the gateway to the associated model service for processing.
You need to create and manage Model APIs here to define how clients access and route requests to specific model services. A Model API can be bound to a single model service or to multiple model services with routing policies configured for flexible traffic distribution. It also supports configuring Header matching conditions, allowing you to create multiple Model APIs under the same Base Path to implement scenarios such as multi-tenant isolation and canary release. Additionally, you can configure QPM and Token-based rate limiting for fine-grained traffic control over requests. This document describes how to add a Model API to the AI gateway and manage its associated model services and automatically generated routes.
Operation Steps
Adding a Model API
2. On the instance list page, click the ID of the gateway instance you want to configure to go to its basic information page.
3. In the left sidebar, click Model Management. Then, click the Model API tab. On the API list page, click New.
Note:
The Model API list displays the following fields: ID/Name, Use Case, Request Protocol, Service Type, Description, Creation/Modification Time, and Operations (Edit/Delete). In the console, the "Service Type" field is displayed as Single Model Service or Multiple Model Services.
4. In the Create Model API window, complete the configuration for the first step, Basic Information.
|
API Name | Yes | Enter a name for this API for identification. It can be up to 60 characters long, supports uppercase and lowercase letters in Chinese and English, digits, and separators ("-" and "_"). It cannot start with a digit or separator, and cannot end with a separator. |
Scenario | Yes | Select the purpose of this API. It supports "Text Generation". The system will preconfigure relevant default routes based on the selected scenario. |
Request protocol | Yes | Select the protocol used by clients to call this API, for example, "OpenAI". This selection will affect how the preconfigured routes and gateway process the request/response format. |
Routing | Yes | Default routes are preconfigured automatically based on the selected "Purpose" and "Request Protocol". Select the routes you want to enable for this API. Each selected route will be combined with the Base Path to generate an independent access path. |
Base Path | No | Set a unified route prefix for this API. The complete path for a client request is /{Base Path}/{route path}. For example, if the Base Path is set to /qwen and the route /v1/chat/completions is selected, the complete access path is /qwen/v1/chat/completions. |
Header | No | Configure API-level entry filtering conditions for this model API. After configuration, only requests carrying headers that meet all conditions can be routed to this model API. If left blank, there is no restriction, and all requests can go to the API. Multiple conditions have an AND relationship. Each rule contains: parameter source (Header), parameter key (for example, X-User-Level), matching method (equals, starts with, contains / exists), and parameter value (optional when the matching method is "exists"). Under the same Base Path, an API with Header conditions takes precedence over an API without conditions. |
Description | No | The description of this API, which facilitates subsequent management. |
In this step, the final access path for the API is formed by combining the Base Path you define with the selected route. Based on the Use Case and Request Protocol you select, the system automatically pre-configures one or more default routes. For example, if you select the "Text Generation" use case and the "OpenAI" protocol, the system pre-configures the /v1/chat/completions route. After creation, the gateway automatically generates a routing rule based on this complete path.
5. After completing the basic information configuration, click Next to go to the "Select Model Service" step. In this step, you need to bind this API to a specific model service, which has been configured with policies such as vendor, key, and model Fallback.
|
Service type | Yes | Supports selection of the following two types: 1. Single-model service: This API is statically routed to a backend model service. 2. Multi-model service: This API distributes traffic to multiple backend model services according to a routing policy. It is applicable to scenarios such as canary release, load balancing, and multi-vendor aggregation. |
Routing policy (displayed for multi-model service) | Yes | When you select the multi-model service, configure the relevant content according to the selected policy. |
Service list (displayed for multi-model service) | Yes | Configure multiple backend model services and their routing rules. Each row contains: target service, routing rules, and so on. |
Select service (displayed for single-model service). | Yes | Select a created model service. You can also click the "New Service" link to go to the model service page and quickly create a model service. Select an Anthropic protocol model service (protocol conversion): When you select the OpenAI request protocol in the "Basic Information" step and select single-model service in this step, the list displays both OpenAI protocol and Anthropic protocol model services. When you select a model that uses the Anthropic protocol (such as claude-sonnet-4-5), a "Protocol Conversion Reminder" dialog box appears, indicating: The selected model service uses the Anthropic protocol, which is inconsistent with the OpenAI request protocol. The gateway automatically converts OpenAI requests to Anthropic format, and backend responses are automatically converted back to OpenAI format before being returned to the client. Some parameters may be removed or populated with default values. For details, see the field mapping preview. No changes are required in client code. You can continue using the OpenAI SDK. Click I Understand, Continue Selection to confirm; click Cancel to abandon the selection, and you can select an OpenAI protocol model again. Protocol conversion currently supports only OpenAI > Anthropic one-way conversion and only the single-model service scenario. |
6. Click OK to complete the Model API creation. At this point, the gateway automatically generates a corresponding routing rule for each route you selected in the Basic Information step.
Viewing and Editing a Model API
2. On the instance list page, click the ID of the gateway instance you want to configure to go to its basic information page.
3. In the left sidebar, click Model Management, and then click the Model API tab.
4. Click the ID/Name of the API to go to its details page.
5. On the Basic Information tab, you can view the complete configuration information for the API.
6. In the upper-right corner of the Basic Information tab on the details page, click Edit to modify its Basic Information configuration. After making changes, click OK to save.
Managing Routes
A route is a rule by which the gateway distributes client requests to the corresponding model API. When a model API is created, the system automatically generates routes based on the configuration.
2. On the instance list page, click the ID of the gateway instance you want to configure to go to its basic information page.
3. In the left sidebar, click Model Management, and then click the Model API tab.
4. Click the ID/Name of the API to go to its details page.
5. On the API details page, click the Route Management tab to view details of all routing rules that the system automatically generated for this API. This page displays the route ID, name, type, and the complete matching path. The gateway determines which model API to forward incoming requests to by matching these routing rules.
Managing Associated Model Services
A Model API must be associated with a model service to function. Changes to the associated model service, service type (single-model service/multi-model service), and routing policy are made by clicking Edit Model API, not by directly managing them on the Basic Information tab of the details page. A Model API of the single-model service type can be bound to a maximum of one model service. A Model API of the multi-model service type can be bound to multiple model services and uses routing policies (weight-based routing, parameter-based routing, or model-name-based routing) to determine how traffic is distributed. To change the service type or routing policy, modify the settings via the Edit API entry.
2. On the instance list page, click the ID of the gateway instance you want to configure to go to its basic information page.
3. In the left sidebar, click Model Management, and then click the Model API tab.
4. Locate the target model API and click Edit in its operation column. Alternatively, go to the API details page and click Edit in the upper-right corner.
5. In the edit wizard, go to the Select Model Service and Routing Policy step to make the following changes:
Change the Associated Model Service: In the service list, reselect the target model service.
Change the Service Type: Switch between Single-Model Service and Multi-Model Service.
Change the Routing Policy: When the service type is multi-model service, you can adjust the routing policy (weighted routing, parameter-based routing, or model-name-based routing) and the routing rules for each service.
6. After completing the modifications, click OK to save. Subsequently, requests made through this Model API will be processed according to the new model service and routing policy.
Deleting a Model API
2. On the instance list page, click the ID of the gateway instance you want to configure to go to its basic information page.
3. In the left sidebar, click Model Management, and then click the Model API tab.
4. On the Model API list page, locate the target API and click Delete in its operation column. The system will then perform a dependency check before deletion.
5. The system will display a pop-up window asking you to confirm the deletion and automatically check whether the API is bound to any other resources, such as an authorized consumer group.
If no dependencies exist, the pop-up window will directly display the API information. Click OK to delete it. When the API is deleted, all its auto-generated routing rules are also deleted.
If dependencies exist, the pop-up window will display the message "Resource Deletion Dependency Check Result", prompt "Unresolved dependencies exist", and list the specific dependency items.
6. If dependencies exist, you must first remove all listed dependencies. After the dependencies are removed, click the Recheck link in the pop-up window. The system will then perform the check again.
7. After the check passes and the dependency prompt disappears, click OK to finally delete the API. To cancel the deletion, click Cancel.